No title

NOTE

Communicated by Heiga Zen

How to Pretend That Correlated Variables Are Independent by Using Difference Observations Christopher K. I. Williams [email protected] School of Informatics, University of Edinburgh, Edinburgh EH1 2QL, U.K.

In many areas of data modeling, observations at different locations (e.g., time frames or pixel locations) are augmented by differences of nearby observations (e.g., δ features in speech recognition, Gabor jets in image analysis). These augmented observations are then often modeled as being independent. How can this make sense? We provide two interpretations, showing (1) that the likelihood of data generated from an autoregressive process can be computed in terms of “independent” augmented observations and (2) that the augmented observations can be given a coherent treatment in terms of the products of experts model (Hinton, 1999).

1 Introduction In automatic speech recognition, it is often the case that hidden Markov models (HMMs) are used on observation vectors that are augmented by difference observations (so-called δ features; see Furui, 1986). Under the HMM, each observation vector is modeled as being conditionally independent given the hidden state. How can this make sense, as close-by differences are clearly not independent? A similar difficulty arises in image analysis tasks such as texture segmentation (see, e.g., Dunn & Higgins, 1995). Here derivative features obtained from Gabor filters or wavelet analysis, for example, are modeled as being independent at different locations, despite the fact that these features will have been computed sharing some pixels in common. In this article, we present two solutions to this problem. In section 2, we show that if the data are generated from a vector autoregressive (AR) model, then the likelihood can be expressed in terms of “independent” difference observations. In section 3, we show that the local models at each location can be combined using a product of experts model (Hinton, 1999) to provide a well-defined joint model for the data and that this can be related to AR models. Section 4 discusses how these interpretations are affected if the local models are conditional on a hidden state variable, as is the case for HMMs. Neural Computation 17, 1–6 (2005)

c 2004 Massachusetts Institute of Technology


2

C. Williams

2 An AR Model Consider a temporal vector autoregressive model, Xt =

p


Ai Xt−i + Nt ,

(2.1)

i=1

where the Ai ’s are square matrices and Nt is independent and identically distributed gaussian noise ∼ N(0, N ). Xt and Nt have dimension D for all t. To avoid complicated end effects, we will use periodic (wraparound) boundary conditions, so that the subscript t − i should be read mod(t − i, N). Thus, there are N random variables X0 , . . . , XN−1 , which collectively we denote as X, and similarly for N. Then X and N are related by N = TX for an appropriate matrix T. Thus,   1 −1 exp − NTt N Nt 2 t=0   p T  p  N−1  1
 
−1 exp − Ai Xt−i N Ai Xt−i , =  2 i=0  t=0 i=0

P(X) ∝

N−1 

(2.2)

(2.3)

p where we have set A0 = −I so that Nt = − i=0 Ai Xt−i . p Now let Y0t , . . . , Yt be linearly independent linear combinations of Xt , . . . , Xt−p . For example, we could choose Y0t = Xt , Y1t = Xt − Xt−1 and so on. As the Yit ’s are simple linear combinations of Xt , . . . , Xt−p , we have p


Ai Xt−i =

i=0

p


Bi Yit ,

(2.4)

i=0

for some set of matrices Bi . We can now write  T  p   p N−1  1
 
−1 exp − Bi Yit N Bi Yit , P(X) ∝  2 i=0  t=0 i=0

(2.5)

showing that the likelihood of the underlying X process can be expressed in terms of a product of terms involving the difference observations up to p order p at each time. Stacking Y0t , Y1t , . . . , Yt as the vector Yt we have P(X) ∝

N−1  t=0

  1 exp − YTt MYt , 2

(2.6) j

where the (i, j) block of the matrix M (between Yit and Yt ) has the form −1 Bj . Equation 2.6 almost looks like a product of independent gaussians, BTi N

How to Pretend Correlated Variables Are Independent

3

but note that M is singular (it has rank D as it arises from Nt ) so the correct normalization factor of the gaussian cannot be obtained from it. As a simple example, consider the scalar AR(1) process Xt = αXt−1 + Nt and set Yt0 = Xt , Yt1 = Xt − Xt−1 . Thus, Xt − αXt−1 = (1 − α)Xt + α(Xt − Xt−1 ) = (1 −

α)Yt0

+

αYt1 .

(2.7) (2.8)

To obtain the likelihood for the sequence X, the matrix M will have the form   1 (1 − α)2 α(1 − α) , (2.9) M= 2 α2 σn α(1 − α) where σn2 = var(Nt ). As expected, M has rank 1 (it is an outer product). Interestingly, the matrix M is not equal to the inverse covariance of the Yt ’s derived from the distribution for X. To show this, we first use the result that for the scalar AR(1) process on the circle, the covariance C[j] = Xt Xt−j  is given by C[j] =

σn2 (α |j| + α |N−j| ) . (1 − α 2 )(1 − α N )

(2.10)

Thus, cov(Yt ) =

 0 0 Yt Yt  Yt0 Yt1 

  Yt0 Yt1  C[0] = (C[0] − C[1]) Yt1 Yt1 

 (C[0] − C[1]) . (2.11) 2(C[0] − C[1])

Inversion of cov(Yt ) shows that it is not equal to M as given in equation 2.9. Notice that the joint distribution of Y0 , . . . , YN−1 is singular. If we take an AR process on the X variables, then one can choose linear combinations of the Xt s that are truly independent by carrying out an eigenanalysis. (For the periodic boundary conditions described above and time-invariant coefficients, the eigenbasis would be the Fourier basis.) However, if we allow ourselves an overcomplete basis set, then we have shown that the likelihood of X under the AR process can readily be computed using “independent” densities at each location. Although we have given the derivation above using gaussian noise, in fact the conclusion concerning expressing the likelihood of the X sequence in terms of a product of terms involving Yt ’s is independent of the form of the noise driving the AR process. It is also possible to extend the AR model described above beyond the temporal one-dimensional chain. For example, Abend, Harley, and Kanal (1965) describe Markov mesh models in two dimensions. A simple example of such a model is a “third-order” Markov mesh, where Xi,j depends autoregressively on Xi,j−1 , Xi−1,j−1 , and Xi−1,j . The same construction in terms of Y variables can be used in this case.

4

C. Williams

3 Product of Experts Interpretation At an individual location, we have a model Pt (Yt ) for the augmented vector Yt . To define a joint distribution on X, we set P(X) =

1 Pt (Yt ), Z t

(3.1)

where Z is a normalization constant (known in statistical physics as the partition function). This is the product of experts construction (Hinton, 1999). One can also think of this as a Markov random field construction where

P(X) ∝ exp −E(X) and E(X) = − t log Pt (Yt ). If each Pt (Yt ) is gaussian, then P(X) will also be gaussian, and Z = (2π)N/2 |C|1/2 where C is the covariance matrix of X. Again we consider a simple example relating to a scalar AR(1) process, so Yt = (Xt , Xt − Xt−1 )T . Let 1 Pt (Yt ) ∝ exp − {a0 Xt2 + a1 (Xt − Xt−1 )2 }, 2

(3.2)

with a0 , a1 > 0. Then we obtain the joint distribution,  

1 2 2 P(X) ∝ − Xt + a1 (Xt − Xt−1 ) . a0 2 t t

(3.3)

C−1 , the inverse covariance matrix of X, is circulant with entries a0 + 2a1 on the diagonal and −a1 in the bands above and below the diagonal and in the northeast and southwest corners. For the AR(1) process, Xt = αXt−1 + Nt with Nt ∼ N(0, β −1 ), we obtain corresponding entries of β(1 + α 2 ) on the diagonal and −βα off the diagonal. The overall scale of a0 and a1 has the def

same effect as β in setting the variance of the process but r = aa01 = (1−α) α , so for any given α value, there is a corresponding value of r1 . For the gaussian case with expert t involving interactions between Xt and Xt−p , we obtain a quadratic form with the same pattern of banding as in the inverse covariance matrix of an AR(p) process, but as above, for some choices of parameters there may not be a corresponding AR process. Again this construction can be extended to two (or more) dimensions. For example, in 2D we might consider the variable Xi,j and the differences to its four neighbors to the north, south, east, and west to obtain a fivedimensional Y vector. Equation 3.1, with each expert being gaussian, then defines a gaussian Markov random field over the lattice of X variables. 2

1 Interestingly for r ∈ (−4, 0), there are no corresponding values of α. Note that α = 0 ⇒ a1 = 0.

How to Pretend Correlated Variables Are Independent

5

4 Incorporating Hidden State In speech recognition using HMMs, the Yt s are modeled as conditionally independent given the discrete hidden variable st . We now consider how this affects the interpretations given above. For interpretation 1, we consider a switching AR(p) process or AR-HMM (see, e.g., Woodland, 1992), so that Xt depends on Xt−1 , . . . , Xt−p and also st . For example, using gaussian noise and setting st = k, we have Xt ∼ p N( i=1 Aki Xt−i ,  k ). Notice that the AR model parameters now depend on

p the switching variable. However, we can still write the prediction i=1 Aki Xt−i as a linear combination of the Yit s, so the likelihood can be written in the form of “independent” contributions from the Yt s. Note that the usual forward and backward HMM recursions can be carried out for the ARHMM. For interpretation 2, we have the individual component densities Pt (Yt | st ), and the joint distribution P(X | s) =

1  Pt (Yt | st ), Z(s) t

(4.1)

where s = (s0 , . . . , sN−1 ). Notice that the normalization constant in general depends on s, and thus when given X, the computation of P(X | s) depends no only on the component densities but also on Z(s). However, if Pt (Yt | st ) is gaussian and has the same covariance structure but different means depending on st for all t, then Z would turn out to be independent of s. While writing this article, I became aware of the work of Tokuda, Zen, and Kitamura (2003), who correctly derive the product of gaussian experts construction conditional on s and note the general dependence of Z(s) on s. They also observe that use  of the Viterbi algorithm to find the state sequence s that maximizes P(s) t Pt (Yt | st ) (which is easily done with standard dynamic programming techniques) will not, in general, yield the sequence that maximizes P(s | X), because of the Z(s) term. Most practical HMM-based speech recognition systems use mixtures of gaussians to model the Yt s at each frame. The product of experts interpretation readily handles this situation. For an AR model interpretation, the use of a mixture distribution for the Yt s already suggests a switching AR process with the switching variable hidden. 5 Discussion We have described both conditionally specified models (AR processes) and simultaneously specified models (products of experts) to define the joint density2 P(X) and relate it to the augmented feature vectors {Yt }. 2

This terminology is derived from Cressie (1993, sec. 6.3).

6

C. Williams

While this article describes a theoretical framework for understanding why using difference observations make sense, it would be interesting to examine empirically the question of how well AR and products of experts models characterize the dependencies between time frames or pixel locations. Acknowledgments This note was inspired by questions raised by Joe Frankel’s Ph.D. thesis. Thanks to John Bridle and Joe Frankel for helpful conversations and comments on earlier drafts, to Joe Frankel for drawing my attention to Tokuda et al. (2003), and to the anonymous referees for their comments, which helped to improve the article. References Abend, K., Harley, T. J., & Kanal, L. N. (1965). Classification of binary random patterns. IEEE Transactions on Information Theory, 11(4), 538–544. Cressie, N. A. C. (1993). Statistics for spatial data. New York: Wiley. Dunn, D., & Higgins, W. E. (1995). Optimal Gabor filters for texture segmentation. IEEE Transactions on Image Processing, 4(7), 947–964. Furui, S. (1986). Speaker-independent isolated word recognition using dynamic features of speech spectrum. IEEE Transactions on Acoustics, Speech and Signal Processing, 34, 52–59. Hinton, G. E. (1999). Products of experts. In Proceedings of the Ninth International Conference on Artificial Neural Networks (Vol. 1, pp. 1–6). London: IEE. Tokuda, K., Zen, H., & Kitamura, T. (2003). Trajectory modelling based on HMMs with the explicit relationship between static and dynamic features. In Proceedings of the 8th European Conference on Speech Communication and Technology (Eurospeech 2003). N.P.: International Speech Communication Association. Woodland, P. C. (1992). Hidden Markov models using vector linear prediction and discriminative output distributions. In Proceedings of 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing (Vol. I, pp. 509–512). Piscataway, NJ: IEEE. Received February 3, 2004; accepted May 28, 2004.

NOTE

Communicated by Thorsten Joachims

Convergence of the IRWLS Procedure to the Support Vector Machine Solution Fernando P´erez-Cruz [email protected] Gatsby Computational Neuroscience Unit, 17 Queen Sqaure, London WC1N 3AR, U.K., and Department of Signal Theory and Communications, University Carlos III in Madrid, Avda Universidad 30, 28911 Leganes (Madrid), Spain

Carlos Bousono-Calz ˜ on ´ [email protected]

Antonio Art´es-Rodr´ıguez [email protected] Department of Signal Theory and Communications, University Carlos III in Madrid, Avda Universidad 30, 28911 Leganes (Madrid), Spain

An iterative reweighted least squares (IRWLS) procedure recently proposed is shown to converge to the support vector machine solution. The convergence to a stationary point is ensured by modifying the original IRWLS procedure. 1 Introduction Support vector machines (SVMs) are state-of-the-art tools for linear and nonlinear input-output knowledge discovery (Vapnik, 1998; Scholkopf ¨ & Smola, 2001). The SVM relies on the minimization of a quadratic problem, which is frequently solved using quadratic programming (QP) (Burges, 1998). The iterative reweighted least square (IRWLS) procedure for solving SVM for clas´ sification was introduced in P´erez-Cruz, Navia-V´azquez, Rojo-Alvarez, and Art´es-Rodr´ıguez (1999) and P´erez-Cruz, Navia-V´azquez, Alarcon-Diana, ´ and Art´es-Rodr´ıguez (2001) and it was used in P´erez-Cruz, Alarcon-Diana, ´ Navia-V´azquez, and Art´es-Rodr´ıguez (2000) to construct the fastest SVM solver of the time. It solves a sequence of weighted least-square problems that, unlike other least-square procedures such as Lagrangian SVMs (Mangasarian & Musicant, 2000) or least square SVMs (Suykens & Vandewalle, 1999; Van Gestel et al., 2004), leads to the true SVM solution, as we will show here. However, to prove its convergence to the SVM solution, the IRWLS procedure has to be modified with respect to the formulation that appears in P´erez-Cruz et al. (1999, 2001). The IRWLS has been also proposed for solving regression problems (P´erez-Cruz, Navia-V´azquez, Alarcon-Diana, ´ & Art´es-Rodr´ıguez, 2000). AlNeural Computation 17, 7–18 (2005)

c 2004 Massachusetts Institute of Technology


8

F. P´erez-Cruz, C. Bousono-Calz ˜ on, ´ and A. Art´es-Rodr´ıguez

though we will deal only with the IRWLS for classification, the extension of this proof to regression is straightforward. The article is organized as follows. We prove the convergence of the IRWLS procedure to the SVM solution in section 2 and summarize the algorithmic implementation of it in section 3. We conclude with some comments in section 4. 2 Proof of Convergence of the IRWLS Algorithm to the SVC Solution The support vector classifier (SVC) seeks to compute the dependency between a set of patterns xi ∈ Rd (i = 1, . . . , n) and its corresponding labels

φ(.)

yi ∈ {±1}, given a transformation to a feature space φ(·) (Rd −→ RH and d ≤ H). The SVC solves
min

w,ξi ,b

n  1 ξi w2 + C 2 i=1



subject to yi (φT (xi )w + b) ≥ 1 − ξi ξi ≥ 0, where w and b define the linear classifier in the feature space (nonlinear in the input space, unless φ(x) = x) and C is the penalty applied over training errors. This problem is equivalent to the following unconstrained problem, in which we need to minimize LP (w, b) =

n  1 L(ui ) w2 + C 2 i=1

(2.1)

with respect to w and b, where ui = 1−yi (φT (xi )w+b) and L(u) = max(u, 0). To prove the convergence of the algorithm, we need LP (w, b) to be both continuous and differentiable; therefore, we replace L(u) by a smooth approximation,  0, L(u) = Ku2 /2,  u − 1/(2K),

u<0 0 ≤ u < 1/K u ≥ 1/K,

which tends to max(u, 0) as K approaches infinity (limK−→∞ L(u) = max(u, 0)).

IRWLS Convergence to the SVM Solution

9

Being a convex problem, the SVM solution is achieved at w∗ and b∗ , which makes the gradient vanish,  ∇w LP (w∗ , b∗ ) ∇b LP (w∗ , b∗ )   n  dL(u) ∗ φ(xi )yi   w − C du u∗i  0   i=1 , = =  n  0 dL(u)   −C yi du u∗i i=1

∇LP (w∗ , b∗ ) =



(2.2)

where u∗i = 1 − yi (φT (xi )w∗ + b∗ ). Optimization problems are solved using iterative procedures that rely in each iteration on the previous solution (wk and bk , in our case) to obtain the following one, until the optimal solution has been reached. To construct the IRWLS procedure, we modify equation 2.1 using a first-order Taylor expansion of L(u) over the previous solution, as is common in other optimization procedures (Nocedal & Wright, 1999). This leads to   n  1 dL(u)  2 k k LP (w, b) = w + C L(ui ) + [ui − ui ] , 2 du uki i=1 



where uki = 1 − yi (φT (xi )wk + bk ), LP (wk , bk ) = LP (wk , bk ) and ∇LP (wk , bk ) = ∇LP (wk , bk ). Now, we construct a quadratic approximation imposing that   LP (wk , bk ) = LP (wk , bk ) and ∇LP (wk , bk ) = ∇LP (wk , bk ), leading to   n  1 dL(u) (ui )2 − (uki )2 2 k LP (w, b) = w + C L(ui ) + 2 du uki 2uki i=1 

=

n 1 1 ai (1 − yi (φT (xi )w + b))2 + CT w2 + 2 2 i=1

(2.3)

where

 0, C dL(u) ai = k = KC,  ui du uki C/uki ,

uki < 0 0 ≤ uki < 1/K uki ≥ 1/K

and CT are constant terms that do not depend on w nor b. The IRWLS procedure consists in minimizing equation 2.3 and then recomputing ai with the obtained solution, and continuing until the solution has been reached. We will focus on the algorithmic implementation in the following section; meanwhile, we will demonstrate the following items to prove that the IRWLS procedure converges to the SVM solution:

10

F. P´erez-Cruz, C. Bousono-Calz ˜ on, ´ and A. Art´es-Rodr´ıguez

• The sequence (w0 , b0 ), . . . , (wk , bk ), . . . converges to (wop , bop ). • wop = w∗ and bop = b∗ . First, we need to prove that the sequence of solutions converges to a limiting point in solution space (wop , bop ). Then we need to assess that this limit point corresponds with the SVM solution in equation 2.2. Line search algorithms, for advancing toward the optimum, look at the minimizing functional for a descending direction, pk , and modifies the previous solution, zk , and amount ηk to obtain the following one, zk+1 = zk + ηk pk . Wolfe conditions (Nocedal & Wright, 1999) ensure that line search methods make sufficient progress in each iteration, so the limit point is reached with required precision, being LP (zk + ηk pk ) ≤ LP (zk ) + c1 ∇LP (zk )T pk

(2.4)

∇LP (zk + ηk pk )T pk ≥ c2 ∇LP (zk )T pk

(2.5)

for 0 < c1 < c2 < 1. Wolfe conditions can be applied to the IRWLS procedure because we can describe it as a line search method, where zk = [(wk )T bk ]T , pk = [(ws − wk )T (bs − bk )]T , where ws and bs represent the minimum of the weighted least-square problem in equation 2.3. To prove the first Wolfe condition, also known as the strictly decreasing property, we will first show that LP (zk ) > LP (zk + ηk pk ) = LP (zk+1 ). We  know that LP (wk , bk ) = LP (wk , bk ) and, being ws and bs the minimum of   k k equation 2.3, LP (w , b ) ≥ LP (ws , bs ), equality will hold only if ws = wk and bs = bk , due to the fact that we are solving a least-square problem.   Consequently, LP (wk , bk ) ≥ LP (wk+1 , bk+1 ) ∀ηk ∈ (0, 1], because (wk+1 , bk+1 )  are a convex combination of (wk , bk ) and (ws , bs ) and LP (w, b) is a convex functional, and equality will hold only if wk+1 = wk = ws and bk+1 = bk =  bs . Now we will set ηk to enforce that LP (wk+1 , bk+1 ) ≥ LP (wk+1 , bk+1 ), to   guarantee that LP (wk , bk ) = LP (wk , bk ) > LP (wk+1 , bk+1 ) ≥ LP (wk+1 , bk+1 ).  To show that LP (wk+1 , bk+1 ) ≥ LP (wk+1 , bk+1 ), it is sufficient to prove that L(uki ) +

(uk+1 )2 −(uki )2 dL(u) i du |uki 2uki

≥ L(uk+1 ) ∀i = 1, . . . , n. i (u )2 −(uk )2

i i is tangent to L(ui ) at ui = uki , its For uki ≥ 0, L(uki ) + dL(u) du |uki 2uki minimum is attained at ui = 0, and its minimal value is greater than or

equal to zero. Therefore, in this case, L(uki ) + dL(u) du |uki

(ui )2 −(uki )2 2uki

≥ L(ui ) for any

ui ∈ R. We show an example for = 1 in Figure 1. For uki < 0, we need to ensure that L(uk+1 ) ≤ 0, which can be obtained i k+1 , bk+1 ) are a convex combination of (wk , bk ) and only for uk+1 ≤ 0. As (w i (ws , bs ), uk+1 can be greater than zero only if usi > 0. For the samples, whose i uki

uki < 0 and usi > 0, we will need to set ηik ≤

uki uki −usi

to ensure that uk+1 ≤ 0, i

IRWLS Convergence to the SVM Solution

11

L(u) L(1)+(u2−12)/2 L(−1)

2.5 2 1.5 1 0.5 0 −2

−1

0

1

2

u Figure 1: The dash-dotted line represents the actual SVM loss function L(u). The dashed line represents the approximation to the loss function used in equation 2.3 when uki = 1, and the solid line represents this approximation when uki < 0.

and it can be easily checked that 0 < ηk < 1. Then if we set ηk = min S

uki

uki , − usi

(2.6) 

where S = {i| uki < 0 & usi > 0}, we will ensure that LP (wk+1 , bk+1 ) ≥ LP (wk+1 , bk+1 ). In the case S = ∅, we set wk+1 = ws and bk+1 = bs (i.e., ηk = 1), which proves that LP (zk + ηk pk ) < LP (zk ). Now we can set c1 ∈ (0, c∗1 ] to fulfill equation 2.4, where c∗1 =

LP (zk +ηk pk )−LP (zk ) ∇LP (zk )T pk

is greater than zero because

∇LP < 0. Otherwise, would not be a descending direction. Before proving the second Wolfe condition for the IRWLS, let us rewrite  LP (w, b) as follows: (zk )T pk



LP (w, b) =

pk

n  1 L(uki ) w2 + C 2 i=1 dL(u) yi [φT (xi )(wk − w) + (bk − b)], + du uki

and let us define 

LP (w, b) =

n  1 L(uk+1 ) w2 + C i 2 i=1 dL(u) yi [φT (xi )(wk+1 − w) + (bk+1 − b)], + du uk+1 i

12

F. P´erez-Cruz, C. Bousono-Calz ˜ on, ´ and A. Art´es-Rodr´ıguez 

which is equivalent to LP (w, b) but defined over the actual solution instead.  Being LP (w, b) convex, it can be readily seen that LP (w, b) ≥ LP (w, b) and  LP (w, b) ≥ LP (w, b) ∀w ∈ RH and ∀b ∈ R. As (wk+1 , bk+1 ) are a convex combination of (wk , bk ) and (ws , bs ), we can rewrite pk = [(ws − wk )T (bs − bk )]T = [(wk+1 − wk )T (bk+1 − bk )]T /ηk , leading in the left-hand side of equation 2.5 to: ηk (∇LP (zk+1 )T pk )  k+1 T

) −C

= (w  ×

n 

i=1  k w

 n  dL(u) dL(u) φ (xi )yi −C yi du uk+1 du uk+1 i=1 i i T

− bk+1 − bk

wk+1

= wk+1 2 − (wk+1 )T wk − C

n  dL(u) du i=1 k+1

uk+1 i

× yi [φT (xi )(wk+1 − wk ) + (b − bk )] n  dL(u) = wk+1 2 − (wk+1 )T wk − C du k+1 T

× yi [φ (xi )(w

k+1

1 − wk 2 − C 2

k

i=1 k+1

− w ) + (b

n  i=1

ui

k

− b )]

n  1 k 2 L(uk+1 ) +  + C L(uk+1 ) w i i 2 i=1

1 1 1 = wk+1 2 − (wk+1 )T wk + wk 2 + wk+1 2 2 2 2 n   +C L(uk+1 ) − LP (wk , bk ) i i=1

1  = wk+1 − wk 2 + LP (wk+1 , bk+1 ) − LP (wk , bk ). 2 We now repeat the same algebraic transformations over the right-hand side, of equation 2.5, leading to: ηk (∇LP (zk )T pk ) 

 n  dL(u) dL(u) φ (xi )yi −C yi = (w ) − C du uki du uki i=1 i=1   k+1 w − wk × bk+1 − bk n  dL(u) = (wk+1 )T wk − wk 2 + C du k k T

n 

T

i=1

ui

× yi [φT (xi )(wk − wk+1 ) + (bk − bk+1 )]

IRWLS Convergence to the SVM Solution

13

n n   1 1 + wk+1 2 + C L(uki ) − wk+1 2 − C L(uki ) 2 2 i=1 i=1

1  = − wk+1 − wk 2 − LP (wk , bk ) + LP (wk+1 , bk+1 ). 2 We now show that 

wk+1 − wk 2 /2 + LP (wk+1 , bk+1 ) − LP (wk , bk ) ηk 

>

−wk+1 − wk 2 /2 − LP (wk , bk ) + LP (wk+1 , bk+1 ) , ηk 

which is equivalent to wk+1 − wk 2 + [LP (wk+1 , bk+1 ) − LP (wk+1 , bk+1 )] +  [LP (wk , bk ) − LP (wk , bk )] > 0, because ηk ∈ (0, 1]. The terms L(wk+1 , bk+1 ) −   L (wk+1 , bk+1 ) and L(wk , bk ) − L (wk , bk ) are equal to or greater than zero because the loss function is convex. Moreover, wk+1 − wk 2 ≥ 0, and it is zero only if wk+1 = wk . Therefore, we are not at the solution ∇LP (zk+1 )T pk > ∇LP (zk )T pk . Now, we can set c2 ∈ [c∗2 , 1)1 to fulfill equation 2.5, where c∗2 = ∇LP (zk+1 )T pk is less than one because ∇LP (zk )T pk < 0; otherwise, pk would ∇LP (zk )T pk not be a descending direction. We now need to prove that the proposed algorithm stops when the gradient of LP (w, b) vanishes. The Zoutendijk condition (Nocedal & Wright, 1999) tell us that if LP (w, b) is bounded below and it is Lipschitz continuous,2 and the optimization procedure fulfills the Wolfe conditions, then ∇LP (wk , bk )2 ∇L (wk ,bk )T pk

cos2 θk → 0 as k → ∞, where cos2 θk = ∇L P(wk ,bk )pk  . If we prove that θk P does not tend to π/2 as k → ∞, we would have proven that the gradient of LP (w, b) vanishes and that the proposed algorithm converges to a minimum. Finally, we would need to prove that the achieved solution corresponds to the SVM solution, which we will first prove. The minimum of equation 2.3 is obtained by solving the following linear system:   n  T φ(xi )yi ai (1 − yi (φ (xi )w + b))   w − 0   i=1 .  = n  0   T − yi ai (1 − yi (φ (xi )w + b)

(2.7)

i=1

If c∗2 < 0, the minimum value c2 can take is c1 , and if c∗1 > 1, the highest value c1 can take is c2 , but this does not affect the given proof. 2 L (w, b) is equal to or greater than zero, and it is Lipschitz continuous, because we P made it differentiable. 1

14

F. P´erez-Cruz, C. Bousono-Calz ˜ on, ´ and A. Art´es-Rodr´ıguez

The IRWLS procedure stops when ws = wk and bs = bk ; if we replace them in equation 2.7, we are led to C dL(u) T s s φ(xi )yi k k (1 − yi (φ (xi )w + b ))  w − du u  u i i=1 i  n  C dL(u)  − yi k (1 − yi (φT (xi )ws + bs ) u du k 

s

n 

i=1

i

    

ui

  n  dL(u) s φ(xi )yi   w − C du usi  0   i=1 , = =  n  0 dL(u)   −C yi du usi i=1

(2.8)

which is equal to equation 2.2. Consequently the IRWLS algorithm stops when it has reached the SVM solution. To prove the sufficient condition, we need to show that if wk = w∗ and k b = b∗ , the IRWLS has stopped. Suppose it has not; then we can find   ws = wk and bs = bk such that LP (wk , bk ) > LP (ws , bs ), and the strictly decreasing property will lead to LP (w∗ , b∗ ) > LP (ws , bs ), which is a contradiction because w∗ and b∗ give the minimum of LP (w, b). We have just proven that if the IRWLS has stopped, we will be at the SVM solution, and if we are at the SVM solution, the IRWLS has stopped. Finally, we do not need to prove that θk does not tend to π/2, because we have just shown that the algorithm stops iff we are at the SVM solution and, consequently, this is the point at which the gradient of LP (w, b) vanishes. That was what we still needed to prove, ending the proof of convergence. 3 Iterative Reweighted Least Squares for Support Vector Classifiers The IRWLS procedure, when introduced in P´erez-Cruz et al. (1999), did not consider the modification to ensure convergence presented in the previous section (i.e., ηk < 1 in some iterations). We will now describe the algorithmic implementation of the procedure. But before presenting the algorithm, let us rewrite equation 2.7 in matrix form,  T Da + I aT

T a aT 1

   T  w Da y = , b aT y

(3.1)

where = [φ(x1 ), φ(x2 ), . . . , φ(xn )]T , y = [y1 , . . . , yn ]T , a = [a1 , . . . , an ]T , (Da )ij = ai δij (∀i, j = 1, . . . , n), I is the identity matrix, and 1 is a column vector of n ones. This system can be  solved using kernels,  as well as the regular SVM, by imposing that w = i φ(xi )yi αi and i αi yi = 0. These conditions can be obtained from the regular SVM solution (KKT conditions; see Scholkopf ¨

IRWLS Convergence to the SVM Solution

15

& Smola, 2001, for further details). Also, they can be derived from equation 2.2 in which the αi have replaced the derivative of L(ui ). The system in equation 3.1 becomes  H + Da −1 yT

    1 y α , = 0 0 b

(3.2)

where (H)ij = k(xi , xj ) = φT (xi )φ(xj ) and k(·, ·) is the kernel of the nonlinear transformation φ(·) (Scholkopf ¨ & Smola, 2001). The steps to derive equation 3.2 from equation 3.1 can be found in P´erez-Cruz et al. (2001). The IRWLS can be summarized in the following steps: 1. Initialization: set k = 0, α0 = 0, b0 = 0 and u0i = 1 2. Solve equation 3.2 to obtain 3.

αs

and

Compute usi . Construct S = {i|uki αk+1 = αs and bk+1 = bs and go to

∀i = 1, . . . , n.

bs .

< 0 & step 5.

usi > 0}. If S = ∅, set

4. Compute ηk , using equation 2.6, and αk+1 and bk+1 . If L(αk+1 , bk+1 ) > L(αs , bs ), set αk+1 = αs and bk+1 = bs . 5. Set k = k + 1 and go to step 2 until convergence. The modification in the third step helps to further decrease the SVM functional. The value of ηk in equation 2.6 is a sufficient condition but not a necessary one, and in some cases, αs and bs can produce a further decrease in L(α, b) than using αk+1 and bk+1 . It is worth pointing out that the solution achieved in the first step coincides with the least-square support vector machine solution (Suykens, Van Gestel, De Brabanter, De Moor, & Vandewalle, 2003), as Da is the identity matrix multiplied by C. In a way, we can say that the starting point of the IRWLS procedure is the LS-SVM solution. 4 Comments on the IRWLS Procedure In this article, we have proven the convergence of the IRWLS procedure to the SVM solution. This algorithm was devised for solving SVMs and presents several properties that make it desirable. First, the IRWLS algorithm, as its name indicates, needs to solve only a simple least-square problem in each iteration. Moreover, the linear system is formed only by the samples whose uki > 0, while those samples whose uki < 0 will not affect  the functional LP (w, b) in equation 2.3. This property allows working with only part of the kernel matrix, significantly reducing the run-time complexity. Second, during the first iterations, which are the most computationally costly, many samples change the value of ui from positive to negative. Moreover, if ηk < 1, a sample whose uki < 0 ends with uk+1 = 0, and in the next i iteration its ai = KC. Therefore if any sample whose u∗i ≥ 0, but in some

16

F. P´erez-Cruz, C. Bousono-Calz ˜ on, ´ and A. Art´es-Rodr´ıguez

intermediate iteration uki < 0, the algorithm will recover it. The IRWLS changes several constraints from active to inactive in one iteration, while most QP algorithms stop when a constraint changes. We illustrate this property with a simple example in Figure 2. Third, we point out that the value of ηk is 1 in most iterations; only seldom does a sample change from uki < 0 to usi > 0 and L(wk+1 , bk+1 ) ≤ L(ws , bs ), which explains why the IRWLS was working correctly without this modification. The role of K can be analyzed from the proof and implementation perspectives. A finite value of K allows demonstrating that the IRWLS procedure converges to the SVM solution. If K was infinite, the functional would not be differentiable, and although equation 2.2 and 2.8 will be equal, that would not mean that wop (bop ) is equal to w∗ (b∗ ). From the implementation viewpoint, we are adding at least 1/K to the diagonal of H; therefore, if H is nearly singular (as it is for most problems of interest—even for infinite VC dimension kernels), a finite value of K would avoid numerical instabilities. We usually fix K between 104 and 1010 , depending on the machine precision,

(a)

(b)

(d)

(e)

(c)

(f)

120

LP in (2)

100 80 60 40 0

# Iter

10

20

Figure 2: (a–e) The intermediate solution of the IRWLS procedure for a simple problem respectively, for iterations 1, 2, 3, 8, and 18 (final). (f) The value of LP for every iteration. The squares represent the negative-class samples and the circles the positive-class samples. The solid circles and squares are those samples whose uk+1 > 0. The solid line is the classification boundary, and the dashed i lines represent the ±1 margins. It can be seen that in the first step, almost half of the samples change from a u0i > 0 to u1i < 0, significantly advancing toward the optimum and reducing the complexity of subsequent iterations. The solution in iteration 8 is almost equal to the solution in iteration 18; in these intermediate iterations, the algorithm is only fine-tuning the values of αi .

IRWLS Convergence to the SVM Solution

17

and do not modify it during training. For this range of K, the solution does not vary significantly, meaning that we are close to or at the SVM solution. A software package in MATLAB can be downloaded from our web page (http://www.gatsby.ucl.ac.uk/∼fernando). 4.1 Extending the IRWLS Procedure. We have proven the convergence of the IRWLS procedure to the standard SVM solution. This procedure is sufficiently general to be applied to other loss functions. It can be directly applied to any convex, continuous, and differentiable loss function. As we rely on the derivative of the loss, we demonstrate that the limiting point of the IRWLS procedure is the solution to the actual functional; we need the first-order Taylor expansion to be a lower bound on the loss function to show the sufficient decreasing property. To complete the IRWLS procedure, one needs to come up with a quadratic approximation, which has to be at least locally an upper bound to the loss function to ensure the strictly decreasing property. This upper bound has to take the same value and derivative that the loss function does at the actual solution to ensure that the sequence of solutions converges to the solution of our functional. The question that readily arises is if it can be employed for nonconvex loss functions. First, if it is used at all, it would lead only to a local minimum; another method would be needed to assess the quality of the obtained solution. If the loss function is not convex, the sufficient decreasing property would not hold for every possible solution found by the second step of the IRWLS procedure and further analysis would be necessary, taking into account the shape of the nonconvex function, to demonstrate the convergence of the algorithm. Acknowledgments We kindly thank Chih-Jen Lin for his useful comments and pointing out the weakness of our previous proofs. References Burges, C. J. C. (1998). A tutorial on support vector machines for pattern recognition. Knowledge Discovery and Data Mining, 2(2), 121–167. Mangasarian, O. L., & Musicant, D. R. (2000). Lagrangian support vector machines. Journal of Machine Learning Research, 1, 161–177. Nocedal, J., & Wright, S. J. (1999). Numerical optimization. New York: Springer. P´erez-Cruz, F., Alarcon-Diana, ´ P. L., Navia-V´azquez, A., & Art´es-Rodr´ıguez, A. (2000). Fast training of support vector classifiers. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems, 13, Cambridge, MA: MIT Press.

18

F. P´erez-Cruz, C. Bousono-Calz ˜ on, ´ and A. Art´es-Rodr´ıguez

P´erez-Cruz, F., Navia-V´azquez, A., Alarcon-Diana, ´ P. L., & Art´es-Rodr´ıguez, A. (2000). An IRWLS procedure for SVR. In Proceedings of the EUSIPCO’00. Tampere, Finland. P´erez-Cruz, F., Navia-V´azquez, A., Alarcon-Diana, ´ P. L., & Art´es-Rodr´ıguez, A. (2001). SVC-based equalizer for burst TDMA transmissions. Signal Processing, 81(8), 1681–1693. ´ P´erez-Cruz, F., Navia-V´azquez, A., Rojo-Alvarez, J. L., & Art´es-Rodr´ıguez, A. (1999). A new training algorithm for support vector machines. In Proceedings of the Fifth Bayona Workshop on Emerging Technologies in Telecommunications (pp. 116–120). Baiona, Spain. Scholkopf, ¨ B., & Smola, A. (2001). Learning with kernels. Cambridge, MA: MIT Press. Suykens, J. A. K., Van Gestel, T., De Brabanter, J., De Moor, B., & Vandewalle, J. (2003). Least squares support vector machines. Singapore: World Scientific. Suykens, J. A. K., & Vandewalle, J. (1999). Least squares support vector machine classifiers. Neural Processing Letters, 9(3), 293–300. Van Gestel, T., Suykens, J. A. K., Baesens, B., Vanthienen, S., Dedene, G., De Moor, B., & Vandewalle, J. (2004). Benchmarking least squares support vector machines classifiers. Machine Learning, 54(1), 5–32. Vapnik, V. N. (1998). Statistical learning theory. New York: Wiley. Received January 7, 2004; accepted May 25, 2004.

LETTER

Communicated by Bruno Olshausen

Efficient Coding of Time-Relative Structure Using Spikes Evan Smith [email protected] Department of Psychology, Center for the Neural Basis of Cognition, Carnegie Mellon University, Pittsburgh, PA 15213, U.S.A.

Michael S. Lewicki [email protected] Department of Computer Science, Center for the Neural Basis of Cognition, Carnegie Mellon University, Pittsburgh, PA 15213, U.S.A.

Nonstationary acoustic features provide essential cues for many auditory tasks, including sound localization, auditory stream analysis, and speech recognition. These features can best be characterized relative to a precise point in time, such as the onset of a sound or the beginning of a harmonic periodicity. Extracting these types of features is a difficult problem. Part of the difficulty is that with standard block-based signal analysis methods, the representation is sensitive to the arbitrary alignment of the blocks with respect to the signal. Convolutional techniques such as shift-invariant transformations can reduce this sensitivity, but these do not yield a code that is efficient, that is, one that forms a nonredundant representation of the underlying structure. Here, we develop a non-block-based method for signal representation that is both time relative and efficient. Signals are represented using a linear superposition of time-shiftable kernel functions, each with an associated magnitude and temporal position. Signal decomposition in this method is a non-linear process that consists of optimizing the kernel function scaling coefficients and temporal positions to form an efficient, shift-invariant representation. We demonstrate the properties of this representation for the purpose of characterizing structure in various types of nonstationary acoustic signals. The computational problem investigated here has direct relevance to the neural coding at the auditory nerve and the more general issue of how to encode complex, time-varying signals with a population of spiking neurons. 1 Introduction Nonstationary and time-relative acoustic structures such as transients, timing relations among acoustic events, and harmonic periodicities provide essential cues for many types of auditory processing. In sound localization, Neural Computation 17, 19–45 (2005)

c 2004 Massachusetts Institute of Technology


20

E. Smith and M. Lewicki

human subjects can reliably detect interaural time differences as small as 10 µs, which corresponds to a binaural sound source shift of about 1 degree (Blauert, 1997). In comparison, the sampling interval for an audio CD sampled at 44.1 kHz is 22.7 microseconds. Auditory grouping cues, such as common onset and offset, harmonic comodulation, and sound source location, all rely on accurate representation of timing and periodicity (Slaney & Lyon, 1993). Time-relative structure is also crucial for the recognition of consonants and many types of transient, nonstationary sounds. Neurophysiological research in the auditory brainstem of mammals has found cells capable of conveying precise phase information up to 4 kHz or of tracking the quickly varying envelope of a high-frequency sound (Oertel, 1999). The importance of these acoustic cues has long been recognized, but extracting them from natural signals still poses many challenges because the problem is fundamentally ill posed. In natural acoustic environments, with multiple sound sources and background noises, acoustic events are not directly observable and must be inferred using numerous ambiguous cues. Another reason for the difficulty in obtaining these cues is that most approaches to signal representation are block based; the signal is processed piecewise in a series of discrete blocks. Transients and nonstationary periodicities in the signal can be temporally smeared across blocks. Large changes in the representation of an acoustic event can occur depending on the arbitrary alignment of the processing blocks with events in the signal. Signal analysis techniques such as windowing or the choice of the transform can reduce these effects, but it would be preferable if the representation was insensitive to signal shifts. Shift invariance alone, however, is not a sufficient constraint on designing a general sound processing algorithm. Another important constraint is coding efficiency or, equivalently, the ability of the representation to capture underlying structure in the signal. A desirable code should reduce the information rate from the raw signal so that the underlying structures are more directly observable. Signal processing algorithms can be viewed as a method for progressively reducing the information rate until one is left with only the information of interest. We can make a distinction between the observable information rate, or the rate of the observable variables, and the intrinsic information rate, or the rate of the underlying structure of interest. In speech, the observable information rate of the waveform samples is about 50,000 bits per second, but the intrinsic rate of the underlying words is only around 200 bits per second (Rabiner & Levinson, 1981). Information reduction can be achieved by either selecting only the desired information (and discarding everything else) or removing redundancy, such as the temporal correlations between samples. This reduces the observable information rate while preserving the intrinsic information. In this letter, we investigate algorithms for fitting an efficient, shiftinvariant representation to natural sound signals. The outline of the letter is as follows. The next section describes the motivations behind this approach

Efficient Coding of Time-Relative Structure Using Spikes

21

and illustrates some of the shortcomings of current methods. After defining the model for signal representation, we present different algorithms for signal decomposition and contrast their complexity. Next, we illustrate the properties of the representation on various types of speech sounds. We then present a measure of coding efficiency and compare these algorithms to traditional methods for signal representation. Finally, we discuss the relevance of the computational issues discussed here to spike coding and signal representation at the auditory nerve. 2 Representing Nonstationary Acoustic Structure Encoding the acoustic signal is the first step in any algorithm for performing an auditory task. There are numerous approaches to this problem, which differ in both their computational complexity and in what aspects of signal structure are extracted. Ultimately, the choice about what the representation encodes depends on the tasks that need to be performed. In the ideal case, the encoding process extracts only that information necessary to perform the task and suppresses noise or unrelated information. A generalist approach, like that taken by most mammalian auditory systems, requires a representation that is efficient for a wide range of signals. As natural sounds contain both relatively stationary harmonic structure (e.g., animal vocalizations) as well as nonstationary transient structure (e.g., crunching leaves and twigs), this generalist approach requires a code capable of efficiently representing these disparate sound classes (Lewicki, 2002a). Here we seek an auditory representation that is useful for a variety of different tasks. 2.1 Block-Based Representations. Most approaches to signal representation are block based, in which signal processing takes place on a series of overlapping, discrete blocks. This not only obscures transients and periodicities in the signal, but can also have the effect that for nonstationary signals, small time shifts can produce large changes in the representation, depending on whether and where a particular acoustic event falls within the block. Figure 1 illustrates the sensitivity of block-based representation with small shifts in speech signals. The upper panel shows a short speech waveform sectioned into blocks using two sequences of Hamming windows (solid and dashed curves). Each window spans approximately 30 msecs (512 samples) and successive blocks (A1, A2, and so on) are shifted by 10 msecs. The B blocks offset from the A blocks by an amount indicated by the dot-dash vertical lines (∼ 5 msecs), representing the arbitrary alignment of the signal with respect to the two block sequences. The lower panel shows spectral representations for the three corresponding blocks (solid for the A blocks, dashed for the B blocks). The jagged upper curves show the power spectra for each windowed waveform. The smooth lower curves (offset by −20 dB) show the spectrum of the optimal filter derived by linear predictive coding.

22

E. Smith and M. Lewicki !

!

" 

3IGNAL,EVELD"



 ! "









!

" 

 ! "

" 





 MS ! "

´ ´ ´ ´ 







  &REQUENCYK(Z















Figure 1: Block-based representations are sensitive to temporal shifts. (Top panel) A speech waveform with two sets of overlaid Hamming windows, A1–3 (continuous lines above waveform) and B1–3 (dashed lines below the waveform). (Lower panels) The power spectrum (jagged) and LPC spectrum (smooth) of hamming windows offset by less than 5 ms are overlaid (A, continuous; B, dashed). In either of these, small shifts (e.g., from A2 to B2) can lead to large changes in the representation.

The sound used in Figure 1 is /et/ in the context of the word Vietnamese. The three-block sequence contains an abrupt, transient signal feature, a relatively high-frequency, and high-amplitude /t/ sound occurring at about the 38th msec. The windows preceding the /t/, A1 and B1, contain only the /ee/ vowel waveform. The spectra of these windows nearly overlap, although differences resulting from the slow change in the vowel can be seen. The spectra for windows A2 and B2 show a dramatic difference in the range of 3 to 5 kHz. This results entirely from the arbitrary alignment of each window and the /t/. Because window B2 contains a significant portion of the /t/ waveform, it shows a pronounced increase in powering the higher range. The spectra for the following windows, A3 and B3, are again nearly overlapping, as the /t/ is well represented in both windows. Notice that the increase in the power for window B2 is not as great as that for the final windows. This implies that the alignment of the B sequence will cause a temporal smearing of the constant onset, spreading the energy of the 2 msec transient over a 10 msec window. Discrimination of the phonemes, such as /ba/ and /pa/, is based on differences as small as 5 to 10 msecs in voice-onset time (Liberman, Delattre, & Cooper, 1958). The temporal smear-

Efficient Coding of Time-Relative Structure Using Spikes

23

! m #

ƒ "

k k k

Figure 2: A continuous filter bank produces a shift-invariant representation but does not reduce the information rate. An input signal (A) is convolved with a filter bank (B). The output of the convolution (C) has increased the dimensionality of the input signal.

ing illustrated in Figure 1 could create an ambiguity in the onset of voicing and lead to an alteration in the phoneme perception of a listener. 2.2 Convolutional Representations. One way to minimize the shift sensitivity problem is to increase the block rate. This reduces the variability of the observed spectra but results in a very inefficient code, because there are then several, slowly changing representations of the same underlying acoustic events. In the limit, increasing the block rate simply produces a filter bank in which windowed sinusoids are convolved with the signal. Although this yields a representation that is invariant to shifts, a major drawback is that a filter bank does not reduce the information rate because the dimensionality of the each output is identical to the input; furthermore, there is one output for each filter. This problem is illustrated in Figure 2. The speech waveform in the top row of Figure 2A (the /et/ in Vietnamese and identical to that used in figure 1) is convolved with each of the three (time domain) filters shown in the right column (see Figure 2B). The filters are Gabor functions with peak resonance frequencies at the first and second formants (360 and 2750 Hz) and 4000 Hz. The filter outputs (see Figure 2C) show that the formant energy is roughly constant throughout the sound, while energy in the /t/ is relatively localized. Clearly, it would be preferable to have an efficient representation that was insensitive to signal shift, preserving transients and harmonic shifts, but encoded structure in an event-based fashion. 3 A Sparse, Shiftable Kernel Representation Here we employ a sparse, shiftable kernel method of signal representation (Lewicki & Sejnowski, 1999; Lewicki, 2002b). In this model, the signal x(t) is encoded with a set of kernel functions, φ1 , . . . , φM , that can be positioned arbitrarily and independently in time. The mathematical form of the repre-

24

E. Smith and M. Lewicki

sentation with additive noise is x(t) =

nm M



m sm i φm (t − τi ) + (t),

(3.1)

m=1 i=1

where τim and sm i are the temporal position and coefficient of the ith instance of kernel φm , respectively. The notation nm indicates the number of instances of φm , which need not be the same across kernels. In addition, the kernels are not restricted in form or length. A more general way to express equation 3.1 is to assume that the kernel functions exist at all time points during the signal and let the nonzero coefficients determine the positions of the kernel functions. In this case, the model can be expressed in convolutional form,
 x(t) = sm (τ )φm (t − τ )dτ + (t), (3.2) m

where sm (τ ) is the coefficient at time τ for φm . By using a sparse coefficient signal sm (t) composed only of delta functions, equation 3.2 reduces to equation 3.1. A similar approach assuming only sparse coefficients has been used for coding of natural movies (Olshausen, 2002). The key theoretical abstraction of the model is that the signal is decomposed in terms of discrete acoustic events, represented by the kernel functions, each of which has a precise amplitude and temporal position. Here we assume the kernels are gammatone functions (gamma-modulated sinusoids) whose center frequency and width are set according to an equivalent rectangular band (ERB) filter bank cochlear model using Slaney’s auditory toolbox for Matlab (Slaney, 1998). Except where noted, we used a set of 64 kernel functions for the results below. The use of gammatone functions is well motivated by both biology and natural sound statistics (Lewicki, 2002a). In principle, we could also adapt the set of kernel functions to maximize the efficiency of the code. Figure 3 illustrates the generative model. A signal is represented in terms of a sparse set of discrete temporal events, a spike code. For example, the waveform in Figure 3A consists of three aperiodic “chirps,” each composed of discrete acoustic events with differing amplitudes but identical relative temporal alignments. This signal can be represented by nine spikes, each with a precise time and amplitude. We can plot this representation in terms of a spikegram, Figure 3B, where the nine spikes are shown as ovals of varying size, intensity, and position. Each oval indicates the temporal and spectral position (center of mass and center frequency, respectively) of one gammatone kernel function, with oval size and intensity indicating the amplitude of the kernel coefficient. Representing a kernel’s temporal position based on its center of mass causes them all to align precisely given a delta function as input. We adopt this convention to help illustrate the temporal precision of the spike code.

Efficient Coding of Time-Relative Structure Using Spikes

25

!

+ERNEL#&(Z

"

      



















MS

Figure 3: An illustration of generative model and its spikegram representation. The signal (A) is represented in the spikegram (B) as a set of ovals whose size and intensity indicate the amplitude of the spike. The position of the oval indicates the kernel center frequency (CF, y-axis) and timing (x-axis). The gammatone functions corresponding to the spikes (represented by each oval) are overlaid in gray.

3.1 Encoding Algorithms. Equation 3.1 specifies the generative form of the model but does not provide an encoding algorithm, that is, how to compute the optimal values of τim and sm i for a given signal. The computational objective is to minimize the error (t) while maximizing coding efficiency. As is the case with most coding algorithms, there is a trade-off between the error of the representation and the computational complexity of the algorithm. For the results here, we used three different encoding algorithms to select values for τim and sm i . These show a clear trade-off between complexity and accuracy, but we can gain some flexibility along these dimensions by hybridizing, using the simpler algorithms to initialize the most complex. 3.1.1 Filter Threshold. One approach to efficient audio coding has been to use filter banks based on the human cochlea (Baumgarte, 2002; Lyon, 1982; Shamma, 1985; Gitza, 1988; Patterson, Holdsworth, Nimo-Smith, & Rice, 1988). The filter-threshold algorithm is a computationally simple approximation of cochlear processing. This is a causal approach, and it begins by convolving the signal with the full set of kernel functions from the gammatone ERB filter bank. (Note that for all of the algorithms described here, the kernels are restricted to have unit norm.) The encoded coefficients and m times, sm i and τi , are chosen based on the values and positions of all convolution peaks that exceed a preset threshold. This greatly reduces the size of the observable information rate compared to the convolutional representation, but some degree of (threshold-dependant) temporal and spectral redundancy remains. Filter banks with more than 16 gammatones kernel functions are highly overcomplete, but the filter threshold algorithm does not take the correlations between kernel functions into account during cod-

26

E. Smith and M. Lewicki

ing. As a result, it tends to be a poor estimate of the signal given our linear superposition model. We compensate for this to some degree by adding a single parameter to scale the coefficients. Despite its shortcomings under our model, filter threshold is relatively fast and resilient to noise due to its inherent redundancy. These could be desirable properties depending on the task the system must perform. 3.1.2 Matching Pursuit. An obvious improvement on filter threshold would be to account explicitly for the correlations between kernels, iteratively regressing the signal onto the kernels. This is a noncausal approach, but our goal here is to determine the optimal signal representation. One well-studied formalization of this approach is the matching pursuit algorithm (Mallat & Zhang, 1993). We employ it here to produce a more efficient estimate of the τim and sm i values for a given signal. Our goal is to decompose the signal, x(t), over a set of kernels selected from the gammatone filter bank so as to best capture the structure of the signal. Matching pursuit’s approach to this problem is to iteratively approximate the input signal with successive orthogonal projections onto some basis (in this case the unit-normed gammatone kernels). The signal can be decomposed into x(t) = x(t)φm φm + Rx (t),

(3.3)

where x(t)φm  is the inner product between the signal and the kernel and is equivalent to sm in equation 3.1. The final term in equation 3.3, Rx (t), is the residual signal after approximating x(t) in the direction of φm . The projection with the largest inner product will minimize the power of Rx (t), thereby capturing the most structure possible given a single kernel. Equation 3.2 can be rewritten more generally as Rnx (t) = Rnx (t)φm φm + Rn+1 x (t),

(3.4)

with R0x (t) = x(t) at the start of the algorithm. With each iteration, the current residual is projected onto the gammatones. A single kernel is selected such that φm = arg maxRnx (t)φm . m

(3.5)

This best-fitting projection is subtracted out, and its coefficient and time are recorded. This projection and subtraction leaves Rnx (t)φm φm orthogonal to the residual signal, Rn+1 x (t). It is relatively straightforward to see that each projection is orthogonal to all previous and future projections (Mallat & Zhang, 1993). As a result, matching pursuit codes are composed of mutually orthogonal signal structures.

Efficient Coding of Time-Relative Structure Using Spikes

27

Assuming the kernels span the signal space, the power of the residual, Rnx (t), is guaranteed to decrease on each iteration of the algorithm (Mallat & Zhang, 1993; Goodwin & Vetterli, 1999), and so, in the limit, matching pursuit codes will have arbitrarily small error. For most practical purposes, however, some halting criteria should be defined. The simplest is a lower bound on the inner product between the signal and the kernels. We can also track the signal-to-noise ratio of the code over time and stop at a desired fidelity, or halt when some number of spikes has been recorded. More sophisticated criteria are also possible. We reduce some of the computational overhead of the algorithm by defining local neighborhoods among the kernels via cross-correlation. If the maximal inner product between two kernels across all time shifts was greater than some value θ, then they were included in each other’s neighborhood. Typically, θ was set to 0.001 (all kernels were normalized to have an L2 norm of 1). These neighborhoods are used for reconvolution with the residual signal (i.e., if the last spike involved kernel φn , then a new residual was calculated only for the neighborhood around n). This can introduce very low magnitude distortion in the code, but the computational cost is significantly reduced as most of the kernels in the filter bank are orthogonal to one another at all time shifts. 3.1.3 MAP Optimization. A probabilistic method for inferring spike amplitudes and times was described in Lewicki and Sejnowski (1999) and Lewicki (2002b). This approach makes no heuristic assumptions about where spikes should occur, for example, selecting convolution maxima as in the previous two algorithms. Instead, the problem is recast in a Bayesian probabilistic framework in which we attempt to maximize the a posteriori distribution of coefficients. To describe this approach, we begin by expressing the model in matrix form using a discrete sampling of the continuous time series: x = As + .

(3.6)

The rows of the basis matrix, A, contain each gammatone kernel replicated at each sample position making the basis highly overcomplete. The optimal set of τim and sm i for a signal is found by maximizing the posterior distribution of coefficients given the signal and the gammatones, sˆ = arg max P(s|x, A) = arg max P(x|A, s)P(s). s

s

(3.7)

We make two assumptions in modeling the distributions in equation 3.7. First, the noise, , is gaussian and so the data likelihood, P(x|A, s), is also gaussian. Second, the prior, P(s), a function of the spike times and amplitudes, is very sparse. Given these assumptions, the gradient of equation 3.7

28

E. Smith and M. Lewicki

is given by ∂ log P(s|A, x) ∝ AT (x − As) + z(s), ∂s

(3.8)

where z(s) = (log P(s)) . P(s) was assumed to follow a Laplacian, but other distributions are possible. The assumption of sparseness of the kernel coefficients means that optimizing equation 3.7 essentially selects out the minimal set of gammatones that best accounts for the structure of the sound signal for a given noise level. Although optimally efficient codes are possible in theory, in practice only the briefest sounds can be encoded in this manner. For example, using 64 kernels to encode a signal sampled at 44.1 kHz requires approximately 2.8 million coefficients to be optimized per second of signal. We can reduce most of the computational overhead by using filter threshold or matching pursuit to initialize the maximum a posteriori (MAP) optimization. Instead of optimizing over the entire parameter space, these hybrid algorithms search for optimal amplitude values, s, over a set of spike times τ selected by one of the two approximative algorithms. The departure from optimality is a function of the number and “quality” of the spike times selected by the initializing algorithm. In the results that follow, these hybrid algorithms are evaluated alongside the other algorithms as approximations of the true optimally efficient code. 4 Spike Code Signal Representation The sparse, shiftable kernel representation and a set of decomposition algorithms have now been formalized. To evaluate the model, we will present both examples of the codes it generates and an objective comparison between those and other codes. The following section contains specific examples of spike codes to illustrate its qualities as a method for signal representation and some benefits of time-relative coding. 4.1 Comparison of Encoding Algorithms. There are five possible encoding algorithms described in the previous section: filter threshold, matching pursuit, MAP optimization, optimized filter threshold, and optimized matching pursuit. Figure 4 shows the spike code for a short section of speech (three glottal pulses from the vowel /a/ sampled at 16 kHz) using four of these different encoding algorithms. Even at timescales of 480 to 800 samples, the optimization problem is prohibitive, and an example is not presented here. These spikegrams are formatted identically to Figure 3, with ovals representing the time, center frequency, and magnitude of a spike; only the kernel function overlays are removed. For each, we measure the quality of its representations in terms of signal-to-noise ratio (SNR). To compute this, a reconstruction of the input is generated from the code, and a

Efficient Coding of Time-Relative Structure Using Spikes

29

residual error is computed between the original and reconstruction. The SNR in decibels (dB) is then SNR = 10 log10 (Po /Pe ), where Po is the power of the original and Pe is the power of the residual error. The spikegram in Figure 4A is generated using filter threshold. A high degree of redundancy in both time and frequency is quite evident in the correlated waves of spikes that code each glottal pulse. This redundancy may serve to enhance structural similarity between sound events (e.g., the glottal pulses) and increase the representation’s resistance to noise, but it lacks a succinct description of the temporal and spectral characteristics of the sound. Filter threshold encodes the sound to 18 dB SNR (using the scaling parameter mentioned earlier). Perceptually, the input sound is noticeably distorted in the reconstruction, though the speech content is quite clear. Optimizing the filter threshold code has a dramatic effect on the quality of the encoding, pushing the SNR to 90.1 dB, well beyond the point where the original and reconstructed signals are perceptually discriminable. Given that the original .wav file had 16 bits of precision and assuming coding noise on the order of 1 bit, the estimated SNR of the original signal is about 90 dB. In the example shown in Figure 4B, we assumed a very low level of noise in the model. This results in the majority of spike amplitudes being shifted up or down but few pushed to zero; all of the available information is used to encode the signal accurately. Although few spikes are pruned given an assumption of very low noise, the distribution of spike amplitudes does become sparser as a result of optimization. As progressively high noise levels are assumed, the resulting codes become increasingly sparse, sacrificing SNR in order to prune spikes. Figure 4C shows an example of the spike code produced by matching pursuit. It is obviously vastly less redundant than the filter threshold code in Figure 4A. There is relatively little obvious structure within the representation of each glottal pulse, implying that primarily independent events are being represented; however, the similarity between pulses is evident. Despite the much more compact representation, the signal is encoded to 30.4 dB SNR, with only very subtle distortions perceivable. The code generated by a matching pursuit–MAP optimization hybrid (see Figure 4D) is nearly identical to that produced by matching pursuit alone. It is likely for a 30 dB SNR code that the optimization simply corrects some of the error introduced by our use of kernel neighborhoods when computing residuals on each iteration. One possible reason for the limited effect of optimizing is that matching pursuit codes represent a deep local minimum in the parameter space and the gradient method fails to find a global optimum. Another factor concerns the nature of signal decomposition using matching pursuit. This will be discussed further in a later section. 4.2 Convergence of Fidelity. When encoding a signal with matching pursuit, MAP optimization or any hybrid, the SNR of the code increases monotonically with the number of spikes (this is not necessarily true of

30

E. Smith and M. Lewicki

!

"

&ILTER#&(Z

      









MS

#









MS









MS

$

&ILTER#&(Z

      









MS

Figure 4: Spikegrams created from an input signal (top) using each of the four algorithms: (A) filter threshold encoded to 18.7 dB SNR (see text), (B) optimized filter threshold encoded to 90.1 dB SNR, (C) matching pursuit encoded to 30.4 dB SNR, and (D) optimized matching pursuit encoded to 33.0 dB SNR.

filter threshold). For the optimized codes, the amount of noise assumed in the model defines the trade-off between sparseness and accuracy. Because these codes are globally optimal, their specific form (the precise location of spikes) may be altered given different noise levels. For matching pursuit, the trade-off is much clearer: lowering the threshold for accepting a spike (or otherwise varying the halting criterion) simply adds additional spikes that code further residual structure. Figure 5 shows the effect of varying the number of spikes in a matching pursuit code. The input signal is a segment of speech (the word can sampled at 16 kHz). The spikegram in Figure 5A reflects a very high threshold, producing only 92 actual spikes (about 400 spikes/sec) and a relatively poor representation (10 dB SNR). Above the spikegram is the residual signal from the final iteration of the algorithm. It is apparent that a great deal of structure remains to be coded, although the onset of the consonant, /k/, and the periodicity of the /a/ and /n/ are already revealed. Perceptually, the sound is strongly distorted from the original, but the speech content is quite clear. Figures 5B through 5D show spikegrams and residuals for signals encoded to 20 dB (1600 spikes/sec), 30 dB (3100 spikes/sec), and 40 dB SNR (5500 spikes/sec). By 30 dB, Figure 5C, the distribution of residual amplitudes is not significantly different from a gaussian (based on Lilliefors statistical test to reject gaussian assumption, p-value > 0.2).

Efficient Coding of Time-Relative Structure Using Spikes

31

&ILTER#&(Z

      







MS









MS







MS









MS

&ILTER#&(Z

      

Figure 5: As the spike rate (spikes/sec) increases, the fidelity of the representation increases. The spikegrams above show the improvement of an optimized matching pursuit code with increasing spike rate: (A) 10 dB SNR at 400 spikes/sec, (B) 20 dB SNR at 1600 spikes/sec, (C) 30 dB SNR at 3100 spikes/sec, and (D) 40 dB SNR at 5500 spikes/sec. The residual error is plotted above each spikegram.

4.3 Effect of Kernel Number. Another parameter to be selected for any encoding algorithm is the number of kernel functions. Relatively few gammatones are needed to form a complete basis (i.e., a basis that spans the frequency space of the sounds used), but increasing the number allows greater spectral precision. Figure 6 shows the effect of using matching pursuit with 8, 16, 32, or 64 kernel functions (see Figures 6A–6D, respectively). To be certain that the four sets spanned the frequency space, they were generated independently using Slaney’s Matlab toolbox (Slaney, 1998). In each case, the signal is encoded to approximately 40 dB SNR, but the form of the code changes drastically. With relatively few gammatones (see Figures 6A and 6B), the code lacks both spectral and temporal precision. The time-relative coding is largely lost, and the representation becomes nearly convolutional. Using 32 or 64 kernels more clearly segments the acoustic events and begins to show invariant signal structure. Very similar findings were made testing MAP optimization and the hybrid algorithms. In contrast, increasing the number of kernels with filter threshold only shows enhanced spectral precision as spike times are selected independent of one another.

32

E. Smith and M. Lewicki

!

"

&ILTER#&(Z

      









MS









MS









MS

$

# &ILTER#&(Z

      









MS

Figure 6: The number of kernel functions affects both the spectral resolution and the temporal sparseness of the spike codes. The input signal (top) was encoded using matching pursuit with 8, 16, 32, or 64 kernel functions (A–D, respectively). The total number of spikes in each is (A) 12011, (B) 1167, (C) 497, and (D) 479.

4.4 Comparison to Spectrograms. Having described some of the details of the spike code, we now look more broadly at its representation. A comparison of a spectrogram and spikegram illustrates many properties of the model. In Figure 7, the upper plot shows the waveform of pizzerias spoken by an adult female and sampled at 16 kHz. The spikegram and spectrogram of this signal are shown in the middle and lower plots, respectively. The spikegram was constructed using an optimized filter threshold spike code with 128 ERB-spaced gammatone kernels. Both show the formant and harmonic structure of the vowels (e.g., 320–700 msec). Both also reveal the broad spectral and temporal characteristics of the signal, such as the diffuse energy of the /s/ from 700 to 800 msec. However, while the spectrogram is composed of 10 msec shifted windows (as illustrated in Figure 1), the spikegram possesses precise timing information to the sampling rate of the original signal and retains phase information. This allows it to reveal finely grained synchronous activity across bands. It also possesses a nonlinear frequency axis based on the cochlea. This axis emphasizes the range important to human hearing and is used in many auditory models and speech “front-ends.” 4.5 Sparse Representation of Transients. Though the “pizzerias” example demonstrates the large-scale features of the spike code, the fine structure

Efficient Coding of Time-Relative Structure Using Spikes

33

Figure 7: Three representations of the spoken word pizzerias. (A) Time-varying waveform. (B) Spikegram. (C) Spectrogram. They are presented on the same timescale (indicated at the bottom). Note that the spikegram and spectrogram use different frequency axes.

is more clearly revealed in a shorter speech segment. The waveform and spikegram of first half of the word wealth appear in Figure 8. Here we can see the time-relative coding of nonstationary structure. One hundred msec into the word (about 45 msec from the start of the spikegram), the period between glottal pulses begins to elongate. The spike code maintains a consistent representation of the individual pulses during this period, despite the time dilation. Although there is some slight variability in the representation of each pulse (appropriately reflecting the changes in the underlying signal), the spikes essentially align with the peak of each glottal pulse. Figure 8 also shows the efficiency of the code in representing harmonic structures. The spikegram shown can reconstruct the original signal to 30 db SNR, but it requires only a small number of spikes per pulse. Perceptually, the code contains only very subtle distortions of the original signal. Although the two are distinguishable, it is difficult to judge whether the original or reconstructed sound is the “true” signal. This demonstrates the efficiency of the spikegram with respect to nonstationary harmonic structures. One of the particularly desirable properties of the model is the efficient coding of transients, where precise temporal coding is most important. Dis-

34

E. Smith and M. Lewicki



+ ERNEL #& (Z



      









MS

Figure 8: A spikegram shows time-relative coding in the syllable /el/.

tinguishing consonants in continuous speech, for example, requires the detection of rapid, broadband transients. Figure 9 presents an example of a transient, /t/ sound from the word Vietnamese. The input signal in Figure 9A consists of an extended vowel with an embedded transient. The entire signal was encoded using matching pursuit and then optimized. The small set of spikes corresponding to the transient /t/ sound is easily distinguishable from the other spikes. We were able to segment them by hand from the rest of the representation. In Figure 9B, a spike code of only four events (magnified in the inset) is sufficient to encode the transient (see Figure 9A, Reconstruction), leaving only the vowel component (see Figure 9A, Residual). These two events are precise in time to within 0.06 msec (the sampling rate of the signal.) In a spectrogram (see Figure 9C), the same transient is smeared over 10 msec of time and a large region of frequency space. Note that the timescale (x-axis) is the same in Figures 9A, 9B, and 9C. Although this particular consonant is unusually short in duration, this example still illustrates the precise timing and localization achievable with a spike code. 5 Coding Efficiency The previous section shows that spike coding allows time-relative representation of sound structure, but it is not yet demonstrated that this produces an efficient representation of the signal. A complete evaluation of the spike code model requires some objective measure by which to compare the various algorithms and to compare the model against other representational techniques. Shannon’s rate distortion theory offers an objective measure of coding efficiency that is widely used in signal coding research (Shannon, 1948). The idea is to vary the rate of a code (typically in terms of bits per second) while measuring the effect that has on some measure distortion, such as mean squared error. For the comparisons between spike coding algorithms, we can start simply, varying the rate in terms of spikes per second and measuring the fidelity of the code (the inverse of distortion) in terms

Efficient Coding of Time-Relative Structure Using Spikes

35

Figure 9: Efficient representation of a speech consonant. An input signal (A, input) is represented as both a spikegram (B) and spectrogram (C). We can reconstruct the signal based on only the four spikes shown in the inset (B) to segment the /t/ sound from the vowel (A, reconstruction and residual).

of dB SNR. We will then address the issue of quantifying coding efficiency more precisely in terms of bits. 5.1 Coding Efficiency in Terms of Spikes. The within-model comparison of encoding algorithms will focus on filter threshold, matching pursuit, and the hybrids, allowing four different algorithm combinations. To generate a measure of coding efficiency, each algorithm was used to encode a large corpus of short (50–200 msec) segments of speech (Garofolo et al., 1990), music, and other “natural” sounds (e.g., birdsong, music and environmental sounds) at various spike rates. All of the stimuli were in .wav format, sampled at 16 kHz, and band-limited to 80 to 6000 Hz. The leading and trailing portions of the stimuli were multiplied by half-Hanning windows to prevent edge artifacts. The left panel in Figure 10 shows a simplified rate fidelity curve for each algorithm across the entire database. The x-axis indicates the spike rate on a log scale. The y-axis indicates the fidelity in terms of the mean SNR. The computationally simple filter threshold produces a highly redundant, relatively low-fidelity code (less than 11 dB SNR 10,000 spikes/second.) Its decomposition overrepresents large-amplitude components of signals while devoting relatively few spikes to lower-amplitude components, which may represent distinct sound structure. At large spike rates (more than

36

E. Smith and M. Lewicki 

/PTIM -ATCHING 0URSUIT /PTIM &ILTER´4HRESHOLD -ATCHING 0URSUIT &ILTER´4HRESHOLD

-ATCHING 0URSUIT  &ILTERS 3.2 D"

3.2 D"







 &ILTERS

 

 #OST SPIKESSEC



 

 #OST SPIKESSEC



Figure 10: Spike coding efficiency curves. Plotted on the left is the increase in fidelity with increasing spike rate for the four spike code algorithms. Plotted on the right is the increase in fidelity with increasing spike rate for matching pursuit using different numbers of gammatones in the filter bank (8, 16, 32, 64, 128, 256).

50,000 spikes/second) a mean reconstruction fidelity of about 20 dB SNR is possible. Codes produced by matching pursuit have much greater fidelity at all rates than those from filter threshold. By decomposing signals into sets of orthogonal components, it eliminates all spectral and temporal redundancies in its representation. At low spike rates, where the code tends to represent nonoverlapping signal structures, matching pursuit appears to generate a near-optimal code. However, at higher rates, the fidelity of the greedy algorithm tends to reach a ceiling, with a mean SNR of less than 60 dB. Although the filter threshold produces a relatively inefficient code, MAP optimization of its spike amplitudes can result in an extremely high-fidelity representation. To achieve this, the filter threshold must generate a set of spike times sufficient to span the signal space. There are two factors responsible for the increased efficiency of optimized codes. First, the gradientdescent optimization eliminates redundant spikes by driving spike amplitudes to zero in accordance with the sparse prior. Second, in minimizing the expected error, the optimization makes use of correlations between kernels, subtly adjusting spike amplitudes rather than eliminating them. The relative contribution of each factor is largely dependent on the amount of noise, ε(t), assumed in the model. Low-noise models preserve spikes and rely on precise signal fitting; high-noise models eliminate most spikes (pushing their amplitudes to zero). While optimization of the filter threshold codes greatly increases their efficiency, further optimization of matching-pursuit spike amplitudes leads to relatively small increases in efficiency at lower bit rates. Equation 3.4

Efficient Coding of Time-Relative Structure Using Spikes

37

shows that the algorithm decomposes signals into orthogonal components. As such, increases in efficiency cannot result from redundancy reduction thorough spike elimination. Instead, spike amplitudes are adjusted to make use of correlations between kernels. Using these correlations can prevent the ceilings in coding efficiency found in the raw matching pursuit codes at higher spike rates. The right panel in Figure 10 plots the rate fidelity curves for matching pursuit using different numbers of gammatones. ERB filter banks of 8, 16, 32, 64, 128, and 256 gammatones were produced using Slaney’s Matlab toolbox (Slaney, 1998). Generating each filter bank separately rather than using subsets of some fixed large filter bank allows each component filter’s bandwidth to vary and better tile the frequency space. The sound ensemble (the same used in the between-algorithms comparison) was encoded with each kernel set while varying the spike rate. The resulting curves show a clear relation between coding efficiency and filter bank size. The progression of lowest to highest curves on the plot exactly follows the number of kernel functions used. Although efficiency increases monotonically with the number of size of the filter bank (i.e., number of kernels), the relative gain beyond 64 is extremely small. Additionally, as shown by example earlier in Figure 6, codes produced by 32 or fewer kernels lack a sparse temporal structure in addition to their relatively course spectral representation. Figure 10 shows that matching pursuit is highly efficient at low spike rates but is surpassed by the hybrid optimized filter threshold beyond about 25 dB SNR. The reason for this inefficiency at high rates is that matching pursuit often fails to accurately describe true signal structure (Gribonval, Depalle, Rodet, Bacryf, & Mallatt, 1994; Goodwin & Vetterli, 1999). Because each component of its code is constrained to be orthogonal (by equation 3.4); see also Mallat & Zhang, 1993), it cannot capture independent signal structure, which closely overlaps in time-frequency space. To test matching pursuit’s ability to separate overlapping signal structure, a test signal was created by summing pairs of gammatone kernels separated systematically in time (first third of the signal) and in frequency (latter two-thirds of the signal). The spikegrams in Figure 11 show the potential for time-frequency separability using both matching pursuit and MAP optimization. The ideal sparse representation would consist of pairs of spikes at each of the signal “events” except in the two instances where the kernels perfectly overlap. The MAP optimization algorithm generates just such an encoding (top panel). Looking closely at the representation of one pair of clicks separated 20 msec (top panel, inset), it is clear that two independent events have been coded (allowing perfect reconstruction). In contrast, matching pursuit cannot separate kernels that closely overlap in time and frequency (bottom panel, around 400 and 3600 msec). Matching pursuit’s representation of the same 20 msec separated click pair (bottom panel, inset) is clearly very different from the optimal. On the first iteration of the algorithm, it selects a kernel that is lower frequency than the gam-

38

E. Smith and M. Lewicki

+ERNEL #& (Z



  









 MS

+ERNEL #& (Z



   



 

 

 

 

 MS 





  MS

Figure 11: Spikegrams for a signal made of overlapping gammatones. (Top) MAP optimization finds the underlying structure. An example from one pair of clicks (circled) is shown in the inset. The thick gray curve shows the signal, two approximately 2.8 kHz gammatones separated by 20 msec. The thin dark lines are the kernels found by MAP optimization. (Bottom) Matching pursuit cannot separate kernels that closely overlap in time frequency. Given the same two clicks, it generates a single high-amplitude event centered between the chirps and numerous low amplitude events.

matones used to make the signal and centers it between them in time. This means that the representation underestimates the frequency and describes an event when none actually took place. To compensate for this inaccuracy, a large number of additional, low-amplitude kernels are selected on subsequent iterations. With six spikes, it still produces only a 17 dB SNR representation. Nonetheless, the first spike is the single best choice to reduce the residual power. As such, matching pursuit is extremely efficient at low rates. The signal structure initially encoded is typically well separated in time. For example, Figure 5 showed that the initial encoding involves structure that was largely well separated in time and frequency. Tracking the decomposition spike by spike, the kernel that satisfies equation 3.5 on each iteration tends to occur once at each glottal pulse, capturing the largest amount of signal structure possible with a single spike, before returning to encode residual local structure around the same time point. This efficient, low-rate representation can also be generated by MAP optimization by assuming an appropriate degree of noise in the model, but this parameter can be difficult to determine a priori, and the algorithm is much slower.

Efficient Coding of Time-Relative Structure Using Spikes

39

5.2 Coding Efficiency in Terms of Bits. The sparse, shiftable kernel model and a set of algorithms for spike coding have been described in some detail. We now want to quantify the coding efficiency in bits so as to evaluate the model objectively and compare it quantitatively to other signal representations. Rate fidelity again provides a useful objective measure for comparison. Computing the rate-fidelity curves begins with the associated m pairs of coefficients and time values, {sm i , τi }, which are initially stored as double-precision variables. Storing the original time values referenced to the start of the signal is costly because their range can be arbitrarily large and the distribution of time points is essentially uniform. Storing only the time since the last spike, δτim , greatly restricts the range and produces a variable that approximately follows a gamma distribution. Rate fidelity curves are generated by varying the precision of the code, m {sm i , δτi }, and computing the resulting fidelity through reconstruction. A uniform quantizer is used to vary the precision of the code between 1 and 16 bits. At all levels of precision, the bin widths for quantization are selected m so that equal numbers of values fall in each bin. All sm i or δτi that fall within a bin are recoded to have the same value. We use the mean of the unquantized m values that fell within the bin. sm i and δτi are quantized independently. m We found that δτi for gammatones with low center frequencies required much less precision than for higher-frequency gammatones. Accordingly, temporal precision for the kernel functions was normalized with respect to its wavelength so that the same error during quantization would produce the same relative displacement with respect to a kernel’s wavelength. Treating the quantized values as samples from a random variable, we estimate a code’s entropy (bits/coefficient) from histograms of the values. Rate is then the product of the estimated entropy of the quantized variables and the number of coefficients per second for a given signal. At each level of precision, the signal is reconstructed based on the quantized values, and an SNR for the code is computed. This process was repeated across a set of signals, and the results were averaged to produce rate fidelity curves. m Matching pursuit was used to estimate the {sm i , δτi } pairs for these rate fidelity curves. Coding efficiency can be measured in nearly identical fashion for other signal representation. In addition to spike codes, rate fidelity curves were generated for four other signal representation methods using the same set of sounds. The two most common methods for signal processing are Fourier and wavelet transform. Fourier coefficients were obtained for each signal via fast Fourier transform. The real and imaginary parts were quantized independently, and the rate was based on the estimated entropy of the quantized coefficients. Reconstruction was simply an inverse Fourier transform on the quantized coefficients. Similarly, coding efficiency using eighth-order Daubechies wavelets was estimated using Matlab’s discrete wavelet transform and inverse wavelet transform functions. As a baseline for comparison, rate fidelity curves were produced for the waveform of time-varying am-

40

E. Smith and M. Lewicki 3PEECH

  

  3.2 D"

3.2 D"

2AW #OMPRESSED &OURIER 7AVELET 3PIKECODE



 

















-USIC



2AW #OMPRESSED &OURIER 7AVELET 3PIKECODE





 2ATE +BPS











 2ATE +BPS





Figure 12: Rate fidelity curves for the raw (uncompressed) time-varying signals, compressed signals, Fourier transform, discrete wavelet transform, and spike coding using matching pursuit for speech (left) and music (right).

plitude values. The fidelity was determined by quantizing the amplitude values and computing the SNR directly from the resulting signal. Two different methods were used to determine the rate. In the compressed case, the rate was based on the estimated entropy of the quantized values, just as above. In the raw case, the cost of each coefficient was equal to the quantization level rather than the estimated entropy. Figure 12 shows the rate fidelity curves calculated for two classes of sounds: speech (from the TIMIT database; Garofolo et al., 1990) and music (a wide variety of instrumental pieces collected from the Internet.) Compressed, Fourier, wavelet, and spike representations are all far more efficient than the raw signal over the range of rate fidelity parameters we analyzed. At rates below 40 Kbps, spike codes produce the most efficient representations of both speech and music. For example, between 10 and 20 Kbps, the fidelity of the spike representation of speech is approximately twice that of either Fourier of wavelet transformations. At higher bit rates (above 60 Kbps), the Fourier and wavelet representations produce much higher rate fidelity curves than either spike codes or compressed signals. At high rates (off the scale shown), the wavelet representation is most efficient followed by Fourier. In Figure 10, the filter threshold hybrid code also exceeded the matching pursuit–generated code, in this case beyond 25 dB SNR. This offers some evidence that optimized spike codes might exceed the efficiency of Fourier and wavelets throughout the range of perceptually discriminable fidelities.

Efficient Coding of Time-Relative Structure Using Spikes

41

6 Discussion We have presented a theoretically motivated model for sound coding in which the computational goal is to form an efficient, time-relative representation of the time-varying-amplitude signal. Signals are represented by decomposing them into a minimal number of gammatone acoustic events, each with an associated time and amplitude. We have shown that this yields efficient representations of transient signals, allowing highly precise representations of sound onset. We have also shown that over a large range of fidelities, this representation is more efficient than Fourier or wavelet representations. The research presented here adds to the literature on efficient sound representation in several ways. It expands on the original spike coding model (Lewicki & Sejnowski, 1999; Lewicki, 2002b) by presenting a new encoding algorithm, objectively measuring efficiency of spike codes, and comparing them against Fourier and wavelets, and providing a detailed analysis of the representation with specific examples of its strengths and weaknesses. This work also addresses a number of issues relating to matching pursuit. Goodwin and Vetterling (1999) proposed using damped sinusoids with matching pursuit to avoid problems arising from using symmetric wavelets to represent sound. We demonstrated the potential of using a physiologically derived set of gammatone kernels for sound representation rather than abstract sinusoids. The success of this approach lends support to the idea that these functions are adapted to the statistics of natural sounds (Lewicki, 2002a). Additionally, we presented an analysis of the relationship between matching pursuit codes and those derived via MAP optimization. Finally, we used rate distortion to compare matching pursuit against other approximative spike coding algorithms. Another view of these results is that we have shown a method for achieving significantly greater coding efficiency using what is in effect an overcomplete basis. A motivating goal for so-called overcomplete representations or atomic decompositions with overcomplete dictionaries (Mallat & Zhang, 1993; Chen, Donoho, & Saunders, 1996; Goodwin & Vetterli, 1999; Lewicki & Sejnowski, 2000) is to draw from a rich vocabulary of signal descriptors to find compact signal representations. The method we have described here can be viewed in these terms, because the kernel functions φm (t) can be arbitrarily numerous in their center frequencies and can be placed at arbitrary points in time (Lewicki & Sejnowski, 1999; Lewicki, 2002b). In previous studies, it has been difficult to show that the sparse representations made possible with overcomplete representations could actually yield demonstrably more efficient codes. The general problem is that the added cost of coding an overcomplete set of coefficients often outweighs any gain achieved in representation sparseness (Lewicki & Olshausen, 1999; Lewicki & Sejnowski, 2000). The advantage offered by the approach here is that it is not necessary for the code to describe each implicit coefficient (i.e., at every

42

E. Smith and M. Lewicki

sample position). Instead, it is sufficient to describe the time intervals between the spikes. This yielded a much more efficient code for two reasons. The first is that the use of gammatones is well matched to the underlying structure of speech and music. The second is that the matching-pursuit algorithm achieves highly sparse representations. This is crucial, because optimization with filter threshold yields a highly redundant code that does not show increased coding efficiency. It is possible that improvements in either the kernel functions or the encoding algorithms could yield spike codes with even greater efficiencies, and thus provide improved methods for representing the underlying signal structure. We will address a few assumptions made in our model. The first is that we have assumed an explicit generative model and assessed performance by computing the fidelity of the reconstructed signal. It is possible, for example, that the simple filter threshold algorithm does achieve a high-fidelity encoding of the signal, but we do not know how to reconstruct it. It was our motivation, however, to develop efficient codes that can reveal the intrinsic information of a signal. Our assumption of a linear superposition of kernel functions is a simple means of evaluating the degree of redundancy in the representation. Another assumption concerns our choice of distortion measure for evaluating coding efficiency. The reported signal-to-noise values are based on the sum-squared error between the original signal and the reconstruction. It is well known that this measure of distortion does not agree closely with human perception. For example, gaussian white noise signals would tend to appear very dissimilar using this measure, yet they all sound the same to a listener. Although a perceptual distortion measure will offer greater insight into the relationship between the algorithms described here and human perception, the sum-squared error is adequate to show their relative coding efficiency as general signal representations. Beyond the questions of general signal processing, we are particularly interested in the use of spike coding algorithms as models of neural auditory processing. The analog amplitude values in the model can also be interpreted as representing a local population of auditory nerve spikes. As a theoretic model of auditory coding, this posits that the purpose of the (binary) spikes at the auditory nerve is to encode as accurately as possible the temporal position and amplitude of underlying acoustic features, which are compactly described by gammatones. Analog spikes are a useful theoretical abstraction, and it is simple to convert these individual spikes into a population of probabilistically firing, binary units that can carry the same information. Some possible neural network architectures along these lines are described in Lewicki (2002b). One potential concern for describing the cochlear processing with the model presented here is that this model lacks the explicit representation of the nonlinearities found in detailed cochlear models. Our goal, however, was to construct a coding algorithm motivated by higher-level principles.

Efficient Coding of Time-Relative Structure Using Spikes

43

We have not yet investigated whether any cochlear nonlinearities can be interpreted in terms of the assumed computational objective. For example, matching pursuit and gradient optimization can be viewed as means of selecting out a subset of the spikes that removes interspike redundancy and yields an efficient representation. This could offer a novel theoretical interpretation of two-tone inhibition. The addition of nonlinearities may still not help describe the intrinsic coding processes of the system. For example, the model by Yang, Wang, and Shamma (1992) is spike based and contains numerous biophysically motivated nonlinearities, but the fidelity of their reconstructions falls in the same range as filter threshold and is much lower than that of matching pursuit or the hybrids. They reported SNR ranging from 11 to 25 dB, depending on the extent of processing. This underscores the complexity of the signal encoding problem solved by the peripheral auditory system. We have sought to develop an algorithm that extracts the intrinsic information in a signal from the stream of observable information. As was stated earlier, information reduction can be achieved by either selecting only the desired information or removing redundancy. The choice of kernel functions biases our model to “select” certain types of signal structure over others. Our choice of a gammatone filter bank, a highly efficient basis for natural sounds, biases the model to select the underlying structures of natural sounds. The algorithms for generating spike codes take the approach of redundancy reduction. Reducing the temporal correlations shows promise for yielding better methods to extract intrinsic structure from raw acoustic signals.

Acknowledgments E.S. was supported by NIH training grant MH19983. This material is based on work supported by the National Science Foundation under grant 0238351 to M.L.

References Baumgarte, F. (2002). Improved audio coding using a psychoacoustic model based on a cochlear filter bank. IEEE Transactions on Speech and Audio Processing, 10(7), 495–503. Blauert, J. (1997). Spatial hearing (rev. ed.). Cambridge, MA: MIT Press. Chen, S. S., Donoho, D. L., & Saunders, M. A. (1996). A composite model of the auditory periphery for the processing of speech. SIAM Review, 43(1), 129–159. Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., Dahlgren, N. L., & Zue, V. (1990). TIMIT acoustic-phonetic continuous speech corpus. Audio CD.

44

E. Smith and M. Lewicki

Gitza, O. (1988). Temporal non-place information in the auditory-nerve firing patterns as a front-end for speech recognition in noisy environments. J. Phonetics, 16, 109–124. Goodwin, M. M., & Vetterli, M. (1999). Matching pursuit and atomic signal models based on recursive filter banks. IEEE Transactions on Signal Processing, 47(7), 1890–1902. Gribonval, R., Depalle, P., Rodet, X., Bacryf, E., & Mallatt, S. (1994). Sound signals decomposition using a high resolution matching pursuit. In Proceedings of International Computer Music Conference (pp. 293–296). Lewicki, M. S. (2002a). Efficient coding of natural sounds. Nature Neuroscience, 5(4), 356–363. Lewicki, M. S. (2002b). Efficient coding of time-varying patterns using a spiking population code. In R. P. N. Rao, B. A. Olshausen, & M. S. Lewicki (Eds.), Probabilistic models of the brain: Perception and neural function (pp. 241–255). Cambridge, MA: MIT Press. Lewicki, M. S., & Olshausen, B. A. (1999). A probabilistic framework for the adaptation and comparison of image codes. Journal of the Optical Society of America, 16(7), 1587–1601. Lewicki, M. S., & Sejnowski, T. J. (1999). Coding time-varying signals using sparse, shift-invariant representations. In M. S. Kearns, S. Solla, & D. Cohn (Eds.), Advances in neural information processing systems, 11 (pp. 730–736). Cambridge, MA: MIT Press. Lewicki, M. S., & Sejnowski, T. J. (2000). Learning overcomplete representations. Neural Computation, 12, 337–365. Liberman, A., Delattre, P., & Cooper, F. (1958). Some cues for the distinction between voiced and voiceless stops in initial position. Language and Speech, 1, 905–917. Lyon, R. F. (1982). A computational model of filtering, detection and compression in the cochlea. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (pp. 1282–1285). Mallat, S. G., & Zhang, Z. (1993). Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing, 41(12), 3397–3415. Oertel, D. (1999). The role of timing in the brain stem auditory nuclei of vertebrates. Annual Review of Physiology, 61, 497–519. Olshausen, B. A. (2002). Sparse codes and spikes. In R. P. N. Rao, B. A. Olshausen, & M. S. Lewicki (Eds.), Probabilistic models of the brain: Perception and neural function (pp. 257–272). Cambridge, MA: MIT Press. Patterson, R., Holdsworth, J., Nimo-Smith, I., & Rice, P. (1988). Implementing a gammatone filterbank (Tech. Rep. No. 2341). Cambridge, UK: MRC Applied Psychology Unit. Rabiner, L. R., & Levinson, S. E. (1981). Isolated and connected word recognition: Theory and selected applications. IEEE Trans. Communications, COM-25(5), 621–659. Shamma, S. (1985). Speech processing in the auditory system: II. Lateral inhibition and the central processing of speech-evoked activity in the auditory nerve. Journal of the Acoustical Society of America, 78, 1622–1632.

Efficient Coding of Time-Relative Structure Using Spikes

45

Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423, 623–656. Slaney, M. (1998). Auditory toolbox (Tech. Rep. No. 1998-010). Palo Alto, CA: Interval Research Corporation. Slaney, M., & Lyon, R. F. (1993). On the importance of time—A temporal representaton of sound. In M. Cooke, S. Beet, & M. Crawford (Eds.), Visual representations of speech signals (pp. 95–116). New York: Wiley. Yang, X., Wang, K., & Shamma, S. (1992). Auditory representations of acoustic signals. IEEE Trans. Inf. Theory, 38, 824–839. Received March 11, 2004; accepted June 4, 2004.

LETTER

Communicated by Bruno Olshausen

Bilinear Sparse Coding for Invariant Vision David B. Grimes [email protected]

Rajesh P. N. Rao [email protected] Department of Computer Science and Engineering, University of Washington, Seattle, WA 98195-2350, U.S.A

Recent algorithms for sparse coding and independent component analysis (ICA) have demonstrated how localized features can be learned from natural images. However, these approaches do not take image transformations into account. We describe an unsupervised algorithm for learning both localized features and their transformations directly from images using a sparse bilinear generative model. We show that from an arbitrary set of natural images, the algorithm produces oriented basis filters that can simultaneously represent features in an image and their transformations. The learned generative model can be used to translate features to different locations, thereby reducing the need to learn the same feature at multiple locations, a limitation of previous approaches to sparse coding and ICA. Our results suggest that by explicitly modeling the interaction between local image features and their transformations, the sparse bilinear approach can provide a basis for achieving transformation-invariant vision. 1 Introduction Algorithms for redundancy reduction and efficient coding have been the subject of considerable attention (Olshausen & Field, 1996, 1997; Bell & Sejnowski, 1997; Hinton & Ghahramani, 1997; Rao & Ballard, 1999; Lewicki & Sejnowski, 2000; Schwartz & Simoncelli, 2001). Although the basic ideas can be traced to earlier work (Attneave, 1954; Barlow, 1961), recent techniques such as independent component analysis (ICA) and sparse coding have helped formalize these ideas and have demonstrated the feasibility of efficient coding through redundancy reduction. These techniques produce an efficient code by using appropriate constraints to minimize the dependencies between elements of the code. One of the most successful applications of ICA and sparse coding has been in the area of image coding. Olshausen and Field (1996, 1997) showed that sparse coding of natural images produces localized, oriented basis filters that resemble the receptive fields of simple cells in primary visual cortex. Neural Computation 17, 47–73 (2005)

c 2004 Massachusetts Institute of Technology


48

D. Grimes and R. Rao

Bell and Sejnowski (1997) obtained similar results using their algorithm for ICA. However, these approaches do not take image transformations such as translation into account. Thus, the model cannot take advantage of the fact that certain basis features model the same image features but under different transformations. As a result, for each oriented feature, a number of independent units must code for the same feature at different locations, making it difficult to scale the approach to large image patches and hierarchical networks. In this letter, we propose an approach to sparse coding that explicitly models the interaction between image features and their transformations. A bilinear generative model is used to learn both the independent features in an image as well as their transformations. Our approach extends Tenenbaum and Freeman’s (2000) work on bilinear models for learning content and style by casting the problem within a probabilistic sparse coding framework. Thus, whereas prior work on bilinear models used global decomposition methods such as singular value decomposition (SVD), the approach presented here emphasizes the extraction of local features by removing higher-order redundancies through sparseness constraints. We show that for natural images, this approach produces localized, oriented filters that can be translated by different amounts to account for image features at arbitrary locations. Our results demonstrate how an image can be factored into a set of basic local features and their transformations, providing a basis for transformation-invariant vision. In particular, we focus on the problem of invariance to transformations caused by moving objects or smooth selfmotion. We assume that for a given object, the goal is to estimate the motion of the object and learn bilinear features for both the object and its transformation. We conclude by discussing related work and suggest an extension of the approach to parts-based object recognition, wherein an object is modeled as a collection of local features (or “parts”) and their relative transformations. 2 Bilinear Generative Models We begin by considering the standard linear generative model used in algorithms for ICA and sparse coding (Bell & Sejnowski, 1997; Olshausen & Field, 1997; Rao & Ballard, 1999): z=

m


wi xi = Wx,

(2.1)

i=1

where z is a k-dimensional input vector (for instance, an image), wi is a k-dimensional basis vector, and xi is its scalar coefficient. Given the linear generative model above, the goal of ICA is to learn the basis vectors wi (i.e., the matrix W) such that the xi are as independent as possible, while the goal in sparse coding is to make the distribution of xi highly kurtotic given equation 2.1.

Bilinear Sparse Coding for Invariant Vision

49

Now consider adding a transformation parameterized by λ to the generative process of an image z, so that z = Tλ (Wx). Given the linear model as described above, as λ varies, the x will need to change accordingly. Transformation-invariance methods seek to model the image formation process in such a way that x is independent (or at least uncorrelated) with λ. A simple method for achieving invariance is to introduce another variable y, which accounts for the changes in the image due to the transformation Tλ . Invariance is achieved once y is known because x and λ are conditionally independent given y. Thus, the key requirement of any such model is that y can easily be inferred. Using a probabilistic approach, we specify the form of the image likelihood function P(z|x, y). To model this likelihood, we introduce an interaction function f (x, y) that models the interactions between x and y in the image formation process. For an additive gaussian noise model, the likelihood becomes P(z|x, y) = G (z; f (x, y), σ 2 ). The function f (x, y) should not only be able to represent the transformations of interest, but must also be invertible in the sense that x and/or y can be inferred given z and possibly one of x or y. Perhaps the simplest function f is the linear function: f (x, y) = Wx + W  y. Unfortunately, this model is too impoverished to represent most common classes of transformations such as affine transformations in the image plane. A logical next step is to consider multiplicative interactions between x and y. In this work, we explore the use of the bilinear function, which is the simplest form of f allowing multiplicative interactions. The linear generative model in equation 2.1 can be extended to the bilinear case by using two sets of coefficients xi and yj (or equivalently, two vectors x and y) (Tenenbaum & Freeman, 2000): z = f (x, y) =

n m



wij xi yj .

(2.2)

i=1 j=1

The coefficients xi and yj jointly modulate a set of basis vectors wij to produce an input vector z. For this study, the coefficient xi can be regarded as encoding the presence of object feature i in the image while the yj values determine the transformation present in the image. In the terminology of Tenenbaum and Freeman (2000), x describes the content of the image, while y encodes its style. Equation 2.2 can also be expressed as a linear equation in x for a fixed y:   m n m


y  z = f (x)|y = wij yj  xi = wi xi . (2.3) i=1 y

j=1

i=1

The notation wi signifies a transformed feature computed by the weighted sum shown above of the bilinear features wi,∗ by the values in a given y

50

D. Grimes and R. Rao

Figure 1: Examples of linear and bilinear features. A comparison of learned features between a standard linear model and a bilinear model, both trained using sparseness constraints to obtain localized, independent features. The two y rows in the bilinear case depict the translated object features wi (see equation 2.3) for different y vectors corresponding to translations of −3, . . . , 3 pixels.

vector. Likewise, for a fixed x, one obtains a linear equation in y. Indeed, this is the definition of bilinear: given one fixed factor, the model is linear with respect to the other factor. The power of bilinear models stems from the rich nonlinear interactions that can be represented by varying both x and y simultaneously. Note that the standard linear generative model (see equation 2.1) can be seen as a special case of the bilinear model when n = 1 and y = 1. A comparison between examples of features used in the linear generative model and the bilinear model is given in Figure 1. The features in the linear model represent a single instance within the range of features that can be learned by the bilinear model. 3 Learning Sparse Bilinear Models 3.1 Learning Bilinear Models. Our goal is to learn from image data an appropriate set of basis vectors wij that effectively describe the interactions between the feature vector x and the transformation vector y. A commonly used approach in unsupervised learning is to minimize the sum of squared pixel-wise errors over all images:  2   m
n
   E1 ({wij }, x, y) = z − wij xi yj     i=1 j=1  T   n n m
m


= z − wij xi yj  z − wij xi yj  , i=1 j=1

i=1 j=1

(3.1)

(3.2)

Bilinear Sparse Coding for Invariant Vision

51

where  ·  denotes the L2 norm of a vector. A standard approach to minimizing such a function is to use gradient descent and alternate between minimization with respect to {x, y} and minimization with respect to wij . Unfortunately, the optimization problem as stated is underconstrained. The function E1 has many local minima, and results from our simulations indicate that convergence is not obtainable with data drawn from natural images. There are many different ways to represent an image, making it difficult for the method to converge to a basis set that can effectively represent images that were not in the training set. A related approach is presented by Tenenbaum and Freeman (2000) in their article dealing with style and content separation. Rather than using gradient descent, their method estimates the parameters directly by computing the SVD of a matrix A containing input data corresponding to each content class in every style. Their approach can be regarded as an extension of methods based on principal component analysis (PCA) applied to the bilinear case. The SVD approach avoids the difficulties of convergence that plague the gradient-descent method and is much faster in practice. Unfortunately, the learned features tend to be global and nonlocalized, similar to those obtained from PCA-based methods based on second-order statistics. As a result, the method is unsuitable for the problem of learning local features of objects and their transformations. The underconstrained nature of the problem can be remedied by imposing constraints on x and y. In particular, we cast the problem within a probabilistic framework and impose specific prior distributions on x and y with higher probabilities for values that achieve certain desirable properties. We focus here on the class of sparse prior distributions for several reasons: (1) by forcing most of the coefficients to be zero for any given input, sparse priors minimize redundancy and encourage statistical independence between the various xi and between the various yj (Olshausen & Field, 1997); (2) there is some evidence for sparse representations in the brain (Foldi´ ¨ ak & Young, 1995): the distribution of neural responses in visual cortical areas is typically highly kurtotic, that is, cells exhibit little activity for most inputs but respond vigorously for a few inputs, causing a distribution with a high peak near zero and long tails; (3) previous approaches based on sparseness constraints have obtained encouraging results (Olshausen & Field, 1997); and (4) enforcing sparseness on the xi encourages the parts and local features shared across objects to be learned while imposing sparseness on the yj allows object transformations to be explained in terms of a small set of basis vectors.

3.2 Probabilistic Bilinear Sparse Coding. Our probabilistic model for bilinear sparse coding follows a standard Bayesian MAP (maximum a posteriori) approach. Thus, we begin by factoring the posterior probability of

52

D. Grimes and R. Rao

the parameters given the data as P(x, y, {wij }|z) = P(z|x, y, wij )

m 

P(xi )

i=1

∝ P(z|x, y, wij )

m 

n 

P(yj )P({wij })

(3.3)

P(yj ).

(3.4)

j=1

P(xi )

i=1

n  j=1

Equation 3.3 assumes independence between x, y, and {wij } as well as independence within the individual dimensions of x and y. Equation 3.4 assumes a uniform prior for P({wij }), which is thus ignored. We assume the following priors for xi and yj : 1 −αS(xi ) e Qα 1 −βS(yj ) P(yj ) = e , Qβ

P(xi ) =

(3.5) (3.6)

where Qα and Qβ are normalization constants, α and β are parameters that control the degree of sparseness, and S is a “sparseness function.” For this study, we used S(a) = log(1 + a2 ). As shown in Figure 2, our choice of S(a) corresponds to a Cauchy prior distribution, which exhibits a useful nonlinearity in the derivative S (a). The squared error function E1 in equation 3.2 can be interpreted as representing the negative log likelihood (− log P(z|x, y, wij )) under the assumption of gaussian noise with unit variance (see, e.g., Olshausen & Field, 1997). (a)

(b)

(c)

−6

x 10 4

6

2 1 0 −1

0

a

1

4

S’(a)

S(a)

P(a)

3

1

2 0 −1

0

a

1

0

−1 −1

0

1

a

Figure 2: A probabilistic sparse coding prior. (a) The probability distribution function for the Cauchy sparse coding prior. Although the distribution appears similar to a gaussian distribution, the Cauchy is supergaussian (highly kurtotic). (b) The derived sparseness error function. (c) The nonlinearity introduced in the derivative of the sparseness function. Note that the function differentially forces small coefficients toward zero, and only at some threshold are large coefficients made larger.

Bilinear Sparse Coding for Invariant Vision

53

Maximizing the posterior in equation 3.3 is thus equivalent to minimizing the following log posterior function over all input images:  2   m
m n n


   E({wij }, x, y) = z − wij xi yj  + α S(x ) + β S(yj ). (3.7) i    i=1 j=1 i=1 j=1 The gradient of E can be used to derive update rules at time t for the components xa and yb of the feature vector x and transformation vector y, respectively, for any image z, assuming a fixed basis wij :   n n m


1 ∂E dxa = wTaq z − wij xi yj  yq + =− dt 2 ∂xa q=1 i=1 j=1   m m
n

1 ∂E dyb T  = wqb z − wij xi yj  xq + =− dt 2 ∂yb q=1 i=1 j=1

α  S (xa ) 2

(3.8)

β  S (yb ). 2

(3.9)

Given a training set of inputs zl , the values for x and y for each image after convergence can be used to update the basis set wij in batch mode according to   q m
n

dwab 1 ∂E zl − = wij xi yj  xa yb . (3.10) =− dt 2 ∂wab i=1 j=1 l=1 One difficulty in the sparse coding formulation of equation 3.7 is that the algorithm can trivially minimize the sparseness function by making x or y very small and compensate by increasing the wij basis vector norms to maintain the desired output range. Therefore, as previously suggested (Olshausen & Field, 1997), in order to keep the basis vectors from growing without bound, we adapt the L2 norm of each basis vector in such a way that the variance of xi (and yj , as discussed below) were maintained at a fixed desired level (σg2 ). Simply forcing the basis vectors to have a certain norm can lead to instabilities; therefore, a “soft” variance normalization method was employed. The element-wise variance of the x vectors inferred during a single batch iteration was tracked in the vector xvar and adapted at a rate given by the parameter ε (see algorithm 1, line 18). A gain term gx is computed (see lines 19–20 in algorithm 1), which determines the multiplicative factor for adapting the norm of a particular basis vector: ˆ ij = gx,i w

wij . wij 2

(3.11)

An additional complication in the bilinear case is that wij 2 is related to the variance of both xi and yj . One possible solution is to compute a joint

54

D. Grimes and R. Rao

gain matrix G (which specifies a gain Gi,j for each wij basis vector) as the geometric mean of the elements in the gain vectors gx and gy : G=

 gx gy T .

(3.12)

However, in the case where sparseness is desired for x but not y (i.e., β = 0.0), the variance of y will rapidly increase, as the variance of x rapidly decreases, and no perturbations to the norm of the basis vectors wij will solve this problem. To avoid this problem, the algorithm performs soft variance normalization directly on the evolving y vectors, and scales the basis vectors wij based only on the variance of xi (see algorithm 1, lines 12–14). 3.3 Algorithm for Learning Bilinear Models of Translating Image Patches. This section describes our unsupervised learning algorithm that uses the update rules (see equations 3.8–3.10) to learn localized bilinear features in natural images for two-dimensional translations. Figure 3 presents

Figure 3: Example training data from natural scene images. Training data are formed by randomly selected patch locations from a set of natural images (a). (b) The patch is then transformed to form the training set of patches zij . In this case, the patch is shifted using horizontal translations of ±2 pixels. To learn a model of style/content separation, a single x vector is used to represent each image in a column, and a single y vector represents each image in a row.

Bilinear Sparse Coding for Invariant Vision

55

a high-level view of the training paradigm in which patches are randomly selected from larger images and subsequently transformed. We initially tested the application of the gradient-descent rules simultaneously to estimate {wij }, x, and y. Unfortunately, obtaining convergence reliably was rather difficult in this situation, due to a degeneracy in the model in the form of an unconstrained degree of freedom. Given a constant c, there is ambiguity in the model since P(z|cx, 1c y) = P(z|x, y). Our use of the priors P(x), P(y) largely mitigates problems stemming from this degeneracy, yet oscillations are still possible when both x and y are adapted simultaneously. Fortunately, we found that minimizing E({wij }, x, y) with respect to a single variable until near convergence yields good results, particularly when combined with a batch derivative approach. This approach of iteratively performing MAP estimation with respect to a single variable at a time is known within the statistics community as iterated conditional modes (ICM) (Besag, 1986). ICM is a deterministic method shown to generally converge quickly, albeit to a local minimum. In our implementation, we use a conjugate gradient method to speed up convergence, minimizing E with respect to x and y. The algorithm we have developed for learning the model parameters {wij } is essentially an incremental expectation-maximization (EM) algorithm applied to randomly selected subsets (“batches”) of training image patches. The algorithm is incremental in the sense that we use a single step in the parameter spaces, increasing the log likelihood of the model but not fully maximizing it. We have observed that an incremental M-step often produces better results than a full M-step (equivalent to fully minimizing equation 3.7 with respect to {wij } for fixed x and y)). We believe this is because performing a full M-step on each batch of data can potentially lead to many shallow local minima, which may be avoided by taking the incremental M-steps. The algorithm can be summarized as follows (see also the pseudocode labeled algorithm 1). First, randomly initialize the bilinear basis W, and the matrix Y containing a set of vectors describing each style (ys ). For each batch of training image patches, estimate the per patch (indexed by i) vectors xi and yi,s for a set of transformations (indexed by s). This corresponds to the E-step in the EM algorithm. In the M-step, take a single gradient step with respect to W using the estimated xi and yi,s values. In order to regularize Y and avoid overfitting to a particular set of patches, we slowly adapt Y over time as follows. For each particular style s, adapt ys toward the mean of all inferred vectors y∗,s corresponding to patches transformed according to style s (line 11 in algorithm 1). Averaging ys across all patches for a particular transformation encourages the style representation to be invariant to particular patch content. Without additional constraints, the algorithm above would not necessarily learn to represent content only in x and transformations only in y. In order to learn style and content separation for transformation invariance,

56

D. Grimes and R. Rao

Algorithm 1: LearnSparseBilinearModel(I, T, l, α, β, , η, ε)

1: W ⇐ RandNormalizedVectors(k, m, n) 2: Y ⇐ RandNormalizedVectors(n, r) 3: for iter ⇐ 1, · · · , maxIter do 4: P ⇐ SelectPatchLocations(I, q) 5: Z ⇐ ExtractPatches(I, P) 6: X ⇐ InferContent(W, Y(:, c), Z, α) 7: dW ⇐ Zeros(k, m, n) 8: for s ⇐ 1, · · · , r do 9: Z ⇐ ExtractPatches(I, Transform(P, T(:, s))) 10: Ybatch ⇐ InferStyle(W, X, Z, β) 11: Y(:, s) ⇐ (1 − )Y(:, s) +  · SampleMean(Ybatch ) 12: yvar ⇐ (1 − var + ε · SampleVar(Ybatch ) ε)y

γ yvar 13: gy ⇐ gy σ 2 g

14: 15: 16: 17: 18:

Y(:, s) ⇐ NormalizeVectors(Y(:, s), gy )  dW ⇐ dW + 1r dEdW(Z, W, X, Y(:, s)) end for W = W + ηdW xvar ⇐ (1 − var + ε · SampleVar(X) ε)x

19:

gx ⇐ gx

xvar σg2

γ

20: W ⇐ NormalizeVectors(W, gx ) 21: end for Algorithm 2: InferStyle(W, X, Z, β)

1: Y ⇐ ConjGradSparseFit(W, X, Z, β)

we estimate x and y in a constrained fashion. We first infer xi . Finding the initial x vector relies on having an initial y vector. Thus, we refer to one of the transformations as “canonical,” corresponding to the identity transformation. This transformation vector yc is used for initially bootstrapping the content vector xi , but besides this use is adapted exactly like the rest of the style vectors in the matrix Y. We then use the same xi to infer all yi,s vectors for each transformed patch zi,s . This ensures that a single content vector xi codes for the content in the entire set of transformed patches zi,s . Algorithm 1 presents the pseudocode for the learning algorithm. It makes use of algorithms 2 and 3 for inferring style and content, respectively. Table 1 describes the variables used in the learning of the sparse bilinear model from natural images. Capital letters indicate matrices containing column vectors. Individual columns are indexed using Matlab style “slicing,” for example, Y(:, s) is the ith column of Y yi . The indicates the element-wise product.

Bilinear Sparse Coding for Invariant Vision

57

Algorithm 3: InferContent(W, Y, Z, α)

1: X ⇐ ConjGradSparseFit(W, Y, Z, α) Table 1: Variables for Learning and Inference. Name

Size

m k I α η ε σg2 r

scalar scalar [k , l ] scalar scalar scalar scalar scalar

Description x (content) dim. z (patch) dim. Full-sized images xi prior weight W adaptation rate yvar adaptation rate xi goal variance Number of transforms

Name

Size

n l Z β  γ c T

scalar scalar [k, l] scalar scalar scalar scalar [2, r]

Description y (style) dim. Number of patches in batch Image patches yj prior weight Y adaptation rate Variance penalty Canon. transform idx. Transform parameters

4 Results 4.1 Training Methodology. We tested the algorithms for bilinear sparse coding on natural image data. The natural images we used are distributed by Olshausen and Field (1997) along with the code for their algorithm. Except where otherwise noted in an individual experiment, the training set of images consisted of 10 × 10 pixel patches randomly extracted from 10 512×512 pixel source images. The images are prewhitened to equalize large variances in frequency, helping to speed convergence. To assist convergence, all learning occurs in batch mode, where each batch consists of l = 100 image patches. The effects of varying the number of bilinear basis units m and n were investigated in depth. An obvious setting for m, the dimension of the content vector x, is to use the value corresponding to a complete basis set in the linear model (m equals the number of pixels in the patch). As we will demonstrate, this choice for m yields good results. However, one might imagine that because of the ability to merge representations of features that are equivalent with respect to the transformations, m can be set to a much lower value and still effectively learn the same basis. As we discuss later, this is true only if the features can be transformed independently. An obvious choice for n, the number of dimensions of the style vector y, is simply the number of transformations in the training set. However, we found that the model is able to perform substantial dimensionality reduction, and we present the case where the number of transformations is 49 and a basis of n = 10 is used to represent translated features. Experiments were also performed to determine values for the sparseness parameters α and β. Settings between 25 to 35 for α and between 0 and 5 for β appear to reliably yield localized, oriented features. Ranges are given

58

D. Grimes and R. Rao

here because optimum values seem to depend on n for two reasons. First, as n increases, the number of prior terms in equation 3.8 is significantly less than the mn likelihood terms. Thus, α must be increased to force sparseness on the posterior distribution on x. This intuitive explanation is reinforced by noting that α is approximately a factor of n larger than those found previously (Olshausen & Field, 1997). Second, if dimensionality reduction is desired (i.e., n is reduced), β must be lowered or set to zero, as the elements of y cannot be coded in an independent fashion. For instance, in the case of dimensionality reduction from 49 to 10 transformation basis vectors β = 0. The W step size η for gradient descent using equation 3.10 was set to 0.25. Variance normalization used ε = 0.25, γ = 0.05, and σg2 = 0.1. These parameters are not necessarily the optimum parameters, and the algorithm is robust with respect to significant parameter changes. Generally, only the amount of time required to find a good sparse representation changes with untuned parameters. In the cases presented here, the algorithm converged after approximately 2500 iterations, although to ensure that the representations had converged, we ran several runs for 10,000 iterations. The transformations for most of the experiments were chosen to be twodimensional translations in the range [−3, 3] pixels in both the axes. The experiments measuring transformation invariance (see section 4.4) considered one-dimensional translations in the range of [−8, 8] in 12 × 12 sized patches. 4.2 Bilinear Sparse Coding of Natural Images. Experimental results are analyzed as follows. First, we study the qualitative properties of the learned representation, then look quantitatively at how model parameters affect these and other properties, and finally examine the learned model’s invariance properties. Figures 4 and 5 show the results of training the sparse bilinear model on natural image data. Both show localized oriented features resembling Gabor filters. Qualitatively, these are similar to the features learned by the model for linear sparse coding. Some features appear to encode more complex features than is common for linear sparse coding. We offer several possible explanations here. First, we believe that some deformations of a Gabor response are caused by the occlusion introduced by the patch boundaries. This effect is most pronounced when the feature is effectively shifted out of the patch based on a particular translation and its canonical (or starting) location. Second, we believe that closely located and similarly oriented features may sometimes be representable in the same basis feature by using slightly different transformations representations. In turn, this may “free up” a basis dimension for representing a more complex feature. The bilinear method is able to model the same features under different transformations. In this case, horizontal translations in the range [−3, 3] were used for training the model. Figure 4 provides an example of how

Bilinear Sparse Coding for Invariant Vision

59

Figure 4: Representing natural images and their transformations with a sparse bilinear model. The representation of an example natural image patch and of the same patch translated to the left. Note that the bar plot representing the x vector is indeed sparse, having only three significant coefficients. The code for the translation vectors for both the canonical patch and the translated one is likewise sparse. The wij basis images are shown for those dimensions that have nonzero coefficients for xi or yj .

the model encodes a natural image patch and the same patch after it has been translated. Note that in this case, both the x and y vectors are sparse. Figure 5 displays the transformed features for each translation represented by a learned ys vector. Figure 6 shows how the model can account for a given localized feature at different locations by varying the y vector. As shown in the last column of the figure, the translated local feature is generated by linearly combining a sparse set of basis vectors wij . This figure demonstrates that the bilinear form of the interaction function f (x, y) is sufficient for translating features to different locations. 4.3 Effects of Sparseness on Representation. The free parameters α and β play an important role in deciding how sparse the coefficients in the vectors x and y are. Likewise, the sparseness of the vectors is intertwined with the desired local and independent properties of the wij bilinear basis features. As noted in other research on sparseness (Olshausen & Field, 1996), both the attainable sparseness and independence of features also depend on the model dimensionality—in our case, the parameters m and n. In all of our experiments, we use a complete basis (in which m = k) for content representation, assuming that the translations do not affect the number of basis features needed for representation. We believe this is justified also by the very idea that changes in style should not change the intrinsic content.

60

D. Grimes and R. Rao

Figure 5: Localized, oriented set of learned basis features. The transformed basis features learned by the sparse bilinear model for a set of five horizontal ys translations. Each block of transformed features wi is organized with values of i = 1, . . . , m across the rows and values of s = 1, . . . , r down the columns. In this case, m = 100 and r = 5. Note that almost all of the features exhibit localized and oriented properties and are qualitatively similar to Gabor features.

In theory, the style vector y could also use a sparse representation. In the case of affine transformations on the plane, using a complete basis for y means using a large value on n. From a practical perspective, this is undesirable, as it would essentially equate to the tiling of features at all possible transformations. Thus, in our experiments, we set β to a small or zero value and also perform dimensionality reduction by setting n to a fraction of the number of styles (usually between four and eight). This configuration allows

Bilinear Sparse Coding for Invariant Vision

61

Figure 6: Translating a learned feature to multiple locations. The two rows of eight images represent the individual basis vectors wij for two values of i. The yj values for two selected transformations for each i are shown as bar plots. y(a, b) denotes a translation of (a, b) pixels in the Cartesian plane. The last column shows the resulting basis vectors after translation.

learning of sparse, independent content features while taking advantage of dimensionality reduction in the coding of transformations. We also analyzed the effects of the sparseness weighting term α. Figure 7 illustrates the effect of varying α on the sparseness of the content representation x and on the log posterior optimization function E (see equation 3.7). The results shown are based on 1000 x vectors inferred by presenting the learned model with a random sample of 1000 natural image patches. For each value of α, we also average over five runs of the learning algorithm. Figure 8 illustrates the effect of the sparseness weighting term α on the kurtosis of x and the basis vectors learned by the algorithm. 4.4 Style and Content Invariance. A series of experiments were conducted to analyze the invariance properties of the sparse bilinear model. These experiments examine how transformation (y) and content representations (x) change when the input patch is translated in the plane. After learning a sparse bilinear model on a set of translated patches, we select a new test patch z0 and estimate a reference content vector x0 using InferContent(W, yc , z0 ). We then shift the patch according to transformation i: zi = Ti (z0 ). Next, we infer the new yi using InferStyle(W, x0 , zi ) (without using knowledge of the transformation parameter i). Finally, we infer the new content representation xi using a call to the procedure InferContent(W, yi , zi ). To quantify the amount of change or variance in a given transformation or content representation, we use the L2 norm of the vector difference between the reestimated vector and the initial vector. To normalize our metric and account for different scales in coefficients, we divide by the norm of the

62

D. Grimes and R. Rao 0 −100

Log likelihood

−200 −300 −400 −500 −600

Prior (sparseness) term Reconstruction (Gaussian) term Sum of terms

−700 −800 0

5

10 15 Sparseness parameter value

20

Figure 7: Effect of sparseness on the optimization function. As the sparseness weighting value is increased, the sparseness term (log P(x)) takes on higher values, increasing the posterior likelihood. Note that the reconstruction likelihood (log P(z|x, y, {wij )) is effectively unchanged, even as the sparseness term is weighted 20 times more heavily, thus illustrating that the sparse code does not cause a loss of representational fidelity.

reference vector: xi =

|xi − x0 | |x0 |

yi =

|yi − y0 | . |y0 |

(4.1)

For testing invariance, the model was trained on 12 × 12 patches and vertical shifts in the range [−8, 8], with m = 144 and n = 15. Figure 9 shows the result of vertically shifting a particular content (image patch at a particular location) and recording the subsequent representational changes. Figure 10 shows the result of selecting random patches (different content vectors x) and translating each in an identical way (same translational offset). Both figures show a strong degree of invariance of representation: the content vector x remains approximately unchanged when subjected to different translations, while the translation vector y remains approximately the same when different content vectors are subject to the same translation. While Figures 9 and 10 show two sample test sequences, Figures 11 and 12 show the transformation-invariance properties for the average of 100 runs on shifts in the range [−8, 8] in steps of 0.5. The amount of variance in x to translations of up to three pixels is less than a 2 percent change. These

Bilinear Sparse Coding for Invariant Vision

63

Figure 8: Sparseness prior yields highly kurtotic coefficient distributions. (a) The effect of weighting the sparseness prior for x (via α) on the kurtosis (denoted by k) of xi coefficient distribution. (b) A subset of the corresponding bilinear basis vectors learned by the algorithm.

64

D. Grimes and R. Rao

Figure 9: Transformation-invariance property of the sparse bilinear model. (a) Randomly selected natural image patch transformed by an arbitrary sequence of vertical translations. (b) Sequence of vertical pixel translations applied to the original patch location. (c) Effects of the transformations on the transformation (y) and patch content (x) representation vectors. Note that the magnitude of the change in y is well correlated to the magnitude of the vertical translation, while the change in x is relatively insignificant (mean x = 2.6), thus illustrating the transformation-invariance property of the sparse bilinear model.

results suggest that the sparse bilinear model is able to learn an effective representation of translation invariance with respect to local features. Figure 13 compares the effects of translation in the sparse linear versus sparse bilinear model. We first trained a linear model using the corresponding subset of parameter values used in the bilinear model. The same metric for measuring changes in representation was used by first estimating x0 for a random patch and then reestimating xi on a translated patch. As expected, the bilinear model exhibits a much greater degree of invariance to transformation than the linear model. 4.5 Interpolation for Continuous Translations. Although transformations are learned as discrete style classes, we found that the sparse bilinear model can handle continuous transformations. Linear interpolation in the style space was found to be sufficient for characterizing translations in the continuum between two discrete learned translations. Figure 14a shows

Bilinear Sparse Coding for Invariant Vision

65

Figure 10: Invariance of style representation to content. (a) Sequence of randomly selected patches (denoted “Can.”) and their horizontally shifted versions (denoted “Trans.”). (b) Plot of the the L2 distance in image space between the two canonical images. (c) The change in the inferred y for the translated version of each patch. Note that the patch content representation fluctuates wildly (as does the distance in image space), while the translation vector changes very little.

the values of the six-dimensional y vector for each learned translation. The filled circles on each line represent the value of the learned translation vector for that dimension. Plus symbols indicate the resulting linearly interpolated translation vectors. Note that generally the values vary smoothly with respect to the translation amount, allowing simple linear interpolation between translation vectors, similar to the method of locally linear embedding (Roweis & Saul, 2000). Figure 14b shows the style vectors for a model trained with only the transformations −4, −2, 0, +2, +4. Figure 14c shows examples of three reconstructed patches for two interpolated style vectors (for translations, −3 and +0.5 pixels). Also shown is the mean squared error (MSE) over all image pixels between each reconstructed patch and the actual translated patch. The MSE values for the interpolated cases are somewhat higher than those for translations in the training set (labeled “learned” in the figure) but within the range of MSE values across all image patches.

66

D. Grimes and R. Rao

Percent Norm Change

10

8 ∆x 6

4

2

0 −10

−5

0 Translation (pixels)

5

10

Figure 11: Effects of translation on patch content representation. The average relative change in the content vector x versus translation over 100 experiments in which 12 × 12 image patches were shifted by varying degrees. Note that for translations in the range (−3, +3), the relative change in x is small, yet as more and more features are shifted into the patch, the content representation must change. Error bars represent the standard deviation over the 100 experiments.

5 Discussion 5.1 Related Work. Our work is based on a synthesis of two extensively studied tracks of vision research. The first is transformation-invariant object representations, and the second is the extraction of sparse, independent features from images. The key observation is that the combination of the two tracks of research can be synergistic, effectively making each problem easier to solve. A large body of work exists on transformation invariance in image processing and vision. As discussed by Wiskott (2004), the approaches can be divided into two rough classes: (1) those that explicitly deal with different scales and locations by means of normalizing a perceived image to a canonical view and (2) those that find simple localized features that become transformation invariant by pooling across various scales and regions. The first approach is essentially generative, that is, given a representation of “what” (content) and “where” (style), the model can output an image. The bilinear model is one example of such an approach; others are discussed below. The second approach is discriminative in the sense that the model operates by extracting object information from an image and discards posi-

Bilinear Sparse Coding for Invariant Vision

Percent Norm Change

10

67

∆y

8

6

4

2

0

D4 D2 L2 L1.5 L1 L0.5 0 R0.5 R1 R1.5 R2 U2 U4 Translation (direction,amount)

Figure 12: Effects of changing patch content on translation representation. The average relative change in y for random patch content over 100 experiments in which various transformations were performed on a randomly selected patch. No discernible pattern seems to exist to suggest that some transformations are more sensitive to content than others. The bar shows the mean relative change for each transformation.

tional and scale information. Generation of an image in the second approach is difficult because the pooling operation is noninvertible and discards the “where” information. Examples of this approach include weight sharing and weight decay models Hinton, 1987; Foldi´ ¨ ak, 1991), the neocognitron (Fukushima, Miyake, & Takayukiito, 1983), LeNet (LeCun et al., 1989), and slow feature analysis (Wiskott & Sejnowski, 2002). A key property of the sparse bilinear model is that it preserves information about style and explicitly uses this information to achieve invariance. It is thus similar to the shifting and routing circuit models of Anderson and Van Essen (1987) and Olshausen, Anderson, and Van Essen (1995). Both the bilinear model and the routing circuit model retain separate pathways containing “what” and “where” information. There is also a striking similarity in the way routing nodes in the routing circuit model select for scale and shift to mediate routing, and the multiplicative interaction of style and content in the bilinear model. The y vector in the bilinear model functions in the same role as the routing nodes in the routing circuit model. One important difference is that while the parameters and structure of the routing circuit are selected a priori for a specific transformation, our focus is on learning

68

D. Grimes and R. Rao

250

Percent Norm Change

200

150

100 ∆ xi (linear) ∆ xi (bilinear)

50

0 −10

−5

0 Translation (pixels)

5

10

Figure 13: Effects of transformations on the content vector in the bilinear model versus the linear model. The solid line shows, for a linear model, the relative content representation change due to the input patch undergoing various translations. The points on the line represent the average of 100 image presentations per translation, and the error bars indicate the standard deviation. For reference, the results from the bilinear model (shown in detail in Figure 11) are plotted with a dashed line. This shows the high degree of transformation invariance in the bilinear model in comparison to the linear model, whose representation changes steeply with small translations of the input.

arbitrary transformations directly from natural images. The bilinear model is similarly related to the Lie group–based model for invariance suggested in Rao and Ruderman (1999), which also uses multiplicative interactions but in a more constrained way by invoking a Taylor-series expansion of a transformed image. Our emphasis on modeling both “what” and “where” rather than just focusing on invariant recognition (“what”) is motivated by the belief that preserving the “where” information is important. This information is critical not simply for recognition but for acting on visual information, in both biological and robotic settings. The bilinear model addresses this issue in a very explicit way by directly modeling the interaction between “what” and “where” processes, similar in spirit to the “what” and “where” dichotomy seen in biological vision (Ungerleider & Mishkin, 1982). The second major component of prior work on which our model is based is that of representing natural images through the use of sparse or statisti-

Bilinear Sparse Coding for Invariant Vision

69

Figure 14: Modeling continuous transformations by interpolating in translation space. (a) Each line represents a dimension of the tranformation space. The values for each dimension at each learned translation are denoted by circles, and interpolated subpixel coefficient values are denoted by plus marks. Note the smooth transitions between learned translation values, which allow interpolation. (b) Tranformation space values as in a when learned on coarser translations (steps of 2 pixels). (c) Two examples of interpolated patches based on the training in b. See section 4.5 for details.

70

D. Grimes and R. Rao

cally independent features. As discussed in section 1, our work is strongly based on the work of Olshausen and Field (1997), Bell and Sejnowski (1997), and Hinton and Ghahramani (1997) in forming local, sparse distributed representations directly from images. The sparse and independent nature of the learned features and their locality enables a simple, essentially linear model (given fixed content) to efficiently represent transformations of that content. Given global eigenvectors such as those resulting from principal component analysis, this would be more difficult to achieve. A second benefit of combining transformationinvariant representation with sparseness is that the multiplicity of the same learned features at different locations and transformations can be reduced by explicitly learning transformations of a given feature. Finally, our use of a lower-dimensional representation based on the bilinear basis to represent inputs and to interpolate style and content coefficients between known points in the space has some similarities to the method of locally linear embedding (Roweis & Saul, 2000). 5.2 Extension to a Parts-Based Model of Object Representation. The bilinear generative model in equation 2.2 uses the same set of transformation values yj for all the features i = 1, . . . , m. Such a model is appropriate for global transformations that apply to an entire image region such as a shift of p pixels for an image patch or a global illumination change. Consider the problem of representing an object in terms of its constituent parts (Lee & Seung, 1999). In this case, we would like to be able to transform each part independent of other parts in order to account for the location, orientation, and size of each part in the object image. The standard bilinear model can be extended to address this need as follows:   m n

 z= wij yji  xi . (5.1) i=1

j=1

Note that each object feature i now has its own set of transformation values yji . The double summation is thus no longer symmetric. Also, note that the standard model (see equation 2.2) is a special case of equation 5.1 where yji = yj for all i. We have tested the feasibility of equation 5.1 using a set of object features learned for the standard bilinear model. Preliminary results (Grimes & Rao, 2003) suggest that allowing independent transformations for the different features provides a rich substrate for modeling images and objects in terms of a set of local features (or parts) and their individual transformations relative to an object-centered reference frame. Additionally, we believe that the number of basis features needed to represent natural images could be greatly reduced in the case of independently transformed features. A single “template” feature could be learned for a particular orientation and then

Bilinear Sparse Coding for Invariant Vision

71

translated to represent a localized, oriented image feature. 6 Summary and Conclusion A fundamental problem in vision is to simultaneously recognize objects and their transformations (Anderson & Van Essen, 1987; Olshausen et al., 1995; Rao & Ballard, 1998; Rao & Ruderman, 1999; Tenenbaum & Freeman, 2000). Bilinear generative models provide a tractable way of addressing this problem by factoring an image into object features and transformations using a bilinear function. Previous approaches used unconstrained bilinear models and produced global basis vectors for image representation (Tenenbaum & Freeman, 2000). In contrast, recent research on image coding has stressed the importance of localized, independent features derived from metrics that emphasize the higher-order statistics of inputs (Olshausen & Field, 1996, 1997; Bell & Sejnowski, 1997; Lewicki & Sejnowski, 2000). This paper introduces a new probabilistic framework for learning bilinear generative models based on the idea of sparse coding. Our results demonstrate that bilinear sparse coding of natural images produces localized oriented basis vectors that can simultaneously represent features in an image and their transformation. We showed how the learned generative model can be used to translate a basis vector to different locations, thereby reducing the need to learn the same basis vector at multiple locations, as in traditional sparse coding methods. We also demonstrated that the learned representations for transformations vary smoothly and allow simple linear interpolation to be used for modeling transformations that lie in the continuum between training values. Finally, we showed that the object representation (the “content”) in the sparse bilinear model remains invariant to image translations, thus providing a basis for invariant vision. Our current efforts are focused on exploring the application of the model to learning other types of transformations such as rotations, scaling, and view changes. We are also investigating a framework for learning parts-based object representations that builds on the bilinear sparse coding model presented in this article. Acknowledgments This research is supported by NSF Career grant 133592, ONR YIP grant N00014-03-1-0457, and fellowships to R.P.N.R. from the Packard and Sloan foundations. References Anderson, C. H., & Van Essen, D. C. (1987). Shifter circuits: A computational strategy for dynamic aspects of visual processing. Proceedings of the National Academy of Sciences, 84, 1148–1167.

72

D. Grimes and R. Rao

Attneave, F. (1954). Some informational aspects of visual perception. Psychological Review, 61(3), 183–193. Barlow, H. B. (1961). Possible principles underlying the transformation of sensory messages. In W. A. Rosenblith (Ed.), Sensory communication (pp. 217– 234). Cambridge, MA: MIT Press. Bell, A. J., & Sejnowski, T. J. (1997). The “independent components” of natural scenes are edge filters. Vision Research, 37(23), 3327–3338. Besag, J. (1986). On the Statistical Analysis of Dirty Pictures. J. Roy. Stat. Soc. B, 48, 259–302. Foldi´ ¨ ak, P. (1991). Learning invariance from transformation sequences. Neural Computation., 3(2), 194–200. Foldi´ ¨ ak, P., & Young, M. P. (1995). Sparse coding in the primate cortex. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (pp. 895–898). Cambridge, MA: MIT Press. Fukushima, K., Miyake, S., & Takayukiito (1983). Neocognitron: A neural network model for a mechanism of visual pattern recognition. IEEE Transactions on Systems, Man, and Cybernetics, 13(5), 826–834. Grimes, D. B., & Rao, R. P. N. (2003). A bilinear model for sparse coding. In S. Becker, S. Thrun, ¨ & K. Obermayer (Eds.), Advances in neural information processing systems, 15. Cambridge, MA: MIT Press. Hinton, G. E. (1987). Learning translation invariant recognition in a massively parallel network. In G. Goos & J. Hartmanis (Eds.), PARLE: Parallel architectures and languages Europe (pp. 1–13). Berlin: Springer-Verlag. Hinton, G. E., & Ghahramani, Z. (1997). Generative models for discovering sparse distributed representations. Philosophical Transactions Royal Society B, 352, 1177–1190. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4), 541–551. Lee, D. D., & Seung, H. S. (1999). Learning the parts of objects by non-negative matrix factorization. Nature, 401, 788–791. Lewicki, M. S., & Sejnowski, T. J. (2000). Learning Overcomplete Representations. Neural Computation, 12(2), 337–365. Olshausen, B., Anderson, C., & Van Essen, D. (1995). A multiscale routing circuit for forming size- and position-invariant object representations. Journal of Computational Neuroscience, 2, 45–62. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607–609. Olshausen, B. A., & Field, D. J. (1997). Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research, 37, 33113325. Rao, R. P. N., & Ballard, D. H. (1998). Development of localized oriented receptive fields by learning a translation-invariant code for natural images. Network: Computation in Neural Systems, 9(2), 219–234. Rao, R. P. N., & Ballard, D. H. (1999). Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive field effects. Nature Neuroscience, 2(1), 79–87.

Bilinear Sparse Coding for Invariant Vision

73

Rao, R. P. N., & Ruderman, D. L. (1999). Learning Lie groups for invariant visual perception. In M. S. Kearns, S. Solla, & D. Cohn (Eds.), Advances in neural information processing systems, 11 (pp. 810–816). Cambridge, MA: MIT Press. Roweis, S., & Saul, L. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500), 2323–2326. Schwartz, O., & Simoncelli, E. P. (2001). Natural signal statistics and sensory gain control. Nature Neuroscience, 4(8), 819–825. Tenenbaum, J. B., & Freeman, W. T. (2000). Separating style and content with bilinear models. Neural Computation, 12(6), 1247–1283. Ungerleider, L., & Mishkin, M. (1982). Two cortical visual systems. In D. Ingle, M. Goodale, & R. Mansfield (Eds.), Analysis of visual behavior (pp. 549–585). Cambridge, MA: MIT Press. Wiskott, L. (2004). How does our visual system achieve shift and size invariance? In J. L. van Hemmen & T. J. Sejnowski (Eds.), Problems in systems neuroscience. New York: Oxford University Press. Wiskott, L., & Sejnowski, T. (2002). Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14(4), 715–770. Received November 4, 2003; accepted June 4, 2004.

LETTER

Communicated by Jaap van Pelt

A Probabilistic Framework for Region-Specific Remodeling of Dendrites in Three-Dimensional Neuronal Reconstructions Rishikesh Narayanan [email protected]

Anusha Narayan [email protected]

Sumantra Chattarji [email protected] National Centre for Biological Sciences, Bangalore 560065, India

Dendritic arborization is an important determinant of single-neuron function as well as the circuitry among neurons. Dendritic trees undergo remodeling during development, aging, and many pathological conditions, with many of the morphological changes being confined to certain regions of the dendritic tree. In order to analyze the functional consequences of such region-specific dendritic remodeling, it is essential to develop techniques that can systematically manipulate three-dimensional reconstructions of neurons. Hence, in this study, we develop an algorithm that uses statistics from precise morphometric analyses to systematically remodel neuronal reconstructions. We use the distribution function of the ratio of two normal distributed random variables to specify the probabilities of remodeling along various regions of the dendritic arborization. We then use these probabilities to drive an iterative algorithm for manipulating the dendritic tree in a region-specific manner. As a test, we apply this framework to a well-characterized example of dendritic remodeling: stress-induced dendritic atrophy in hippocampal CA3 pyramidal cells. We show that our pruning algorithm is capable of eliciting atrophy that matches biological data from rodent models of chronic stress. 1 Introduction Dendrites are the primary sites for receiving synaptic inputs from other neurons. The structure and biophysical properties of the dendritic arbor critically modulate synaptic integration. The discovery of the presence of numerous voltage-gated ion channels in dendrites has increased the importance of the role of dendritic arbor in neuronal function (Stuart, Spruston, & H¨ausser, 1999; Magee, Hoffman, Colbert, & Johnston, 1998; Johnston, Magee, Colbert, & Cristie, 1996; Migliore & Shepherd, 2002). Theoretical and experimental studies point to a significant role for dendritic morphology in modulating neuronal firing patterns (Mainen & Sejnowski, 1996; Neural Computation 17, 75–96 (2005)

c 2004 Massachusetts Institute of Technology


76

R. Narayanan, A. Narayan, and S. Chattarji

Krichmar, Nasuto, Scorcioni, Washington, & Ascoli, 2002; van Ooyen, Duijnhouwer, Remme, & van Pelt, 2002), calcium dynamics (Regehr & Tank, 1994), propagation of action potentials (Vetter, Roth, & H¨ausser, 2001), axonal competition (van Ooyen, Willshaw, & Ramakers, 2000), and other important biophysical properties of neurons (Segev & London, 2000; H¨ausser & Mel, 2003). The functional importance of dendritic morphology is also reflected in its strict regulation during development and in the adult brain (Stuart et al., 1999). During development, dendrites undergo dramatic changes in arborization in order to formulate appropriate synaptic contacts with other neurons (Cline, 2001; Wong & Ghosh, 2002). Structural plasticity in dendrites also plays a prominent role in changes elicited by aging (Duan et al., 2003; Pyapali & Turner, 1996), hibernation (Popov, Bocharova, & Bragin, 1992), Alzheimer’s disease (Anderton et al., 1998; Brizzee, 1987; Geula, 1998), temporal lobe epilepsy (Bothwell et al., 2001), brain injury and lesions (Jones & Schallert, 1994; Kevyani & Schallert, 2002), syndromes related to mental retardation (Kaufmann & Moser, 2000; Ramakers, 2002), retinal degeneration (Marc, Jones, Watt, & Strettoi, 2003), and chronic stress (McEwen, 1999; Vyas, Mitra, Rao, & Chattarji, 2002). Detailed morphometric analyses of dendritic plasticity indicate that such remodeling is often confined to specific regions of the dendrites. For instance, stress-induced dendritic atrophy in hippocampal CA3 pyramidal cells is restricted largely to the stratum radiatum, with other regions undergoing little or no atrophy (McEwen, 1999; Vyas et al., 2002). The possibility that the functional correlates of such localized structural changes also remain localized offers exciting roles for dendritic computation and signal processing. Further, such localized dendritic remodeling promises to provide interesting and important insights into neuronal function for a number of reasons (Johnston & Amaral, 1997; Migliore & Shepherd, 2002; Segev & London, 2000; Stuart et al., 1999): • Inputs to different regions along the dendritic tree arrive from different brain areas (see Table 1). • Inputs to different regions along the dendritic tree have different synaptic properties (see Table 1). • Dendritic ion channels are not uniformly distributed (e.g., the A-type transient potassium channel, cation non-specific hyperpolarizationactivated channel, Ih , and the calcium-dependent channels in CA1 pyramidal cells). This could leave similar magnitudes of dendritic remodeling with different effects on various neuronal conductances, depending on the region of remodeling. • The effects of excitatory postsynaptic potentials (EPSP) originating from a remote dendritic location, on voltage changes at the soma, depend on the distance of that dendritic region from the soma.

Region-Specific Remodeling of Dendrites

77

Table 1: Region Specificity of Inputs to the Dendritic Tree of a CA3 Pyramidal Neuron. Dendritic Region Stratum

oriensc,d

Stratum pyramidalec,d Stratum lucidumb

Distance from Soma (µm)

Receives Inputs from

Receptors

0–400

Commissural/associational, interneurons Interneurons Dentate gyrus, interneurons Commissural/associational, interneurons Entorhinal cortex, interneurons

NMDA, AMPA GABAA GABAA Kainate, AMPA GABAA NMDA, AMPA GABAA , GABAB NMDA, AMPA GABAA , GABAB

Soma 0–100

Stratum radiatumc,d

100–350

Stratum lacunosum molecularea

350–550

Notes: Stratum oriens forms the basal side of the tree. All other strata, except for pyramidale, form the apical side. a Berzhanskaya, Urban, and Barrionuevo (1998). b Cossart et al. (2002). c Johnston and Amaral (1997). d Traub, Jefferys, and Whittington (1999). AMPA = α-amino-3-hydroxy-5methyl-4-isoxazole propionate. GABA = gamma aminobutyric acid. NMDA = N-methylD-aspartate.

• Remodeling of different dendritic regions leads to different effects on synaptic integration, which also depends critically on the distance from the cell body and on the distribution of dendritic ion channels. • Dendrites are capable of eliciting localized changes in excitability (Frick, Magee, & Johnston, 2004) and spatial integration (Wang, Xu, Wu, Duan, & Poo, 2003) by modifying dendritic ion channels. The expression of such localized intrinsic mechanisms can be modulated by localized dendritic remodeling. These issues highlight the need for a formalism that allows us to systematically manipulate dendritic morphology in three-dimensional reconstructions of neurons. Hence, the goal of this study is to develop an algorithm that would systematically manipulate neuronal reconstructions to match biological data on the modulation of dendritic architecture. To this end, we use experimental data from a well-established model of structural plasticity of dendrites in the hippocampus, dendritic atrophy in CA3 pyramidal neurons induced by chronic or repeated stress, to specify the extent and location of dendritic remodeling. 2 Methodological Overview The flow diagram in Figure 1 provides an overview of the framework used to develop and implement the analysis presented here. Previous studies (Vyas et al., 2002) from our laboratory, using Sholl’s analysis of Golgi-impregnated CA3b pyramidal neurons in the rat hippocampus, have demonstrated that

R. Narayanan, A. Narayan, and S. Chattarji

Data from Vyas et al., 2002

78

Control Animals

CIS Animals

2D Sholl’s Analysis on CA3b Pyramidal Neurons Shell-wise DL and BP Atrophy Statistics

3D Sholl’s Analysis

Shell-wise DL and BP Pruning Specifications

Section 5

Section 6 Shell-wise DL and BP Pruning Probabilities Section 7

3D CA3b Pyramidal Neuron Reconstructions

Pruning Algorithm Section 8

DSArchive Pruned CA3b Pyramidal Neurons

Figure 1: Overview of the methodological framework used in the study.

chronic immobilization stress (2 hours a day for 10 days) elicits specific patterns of region-specific atrophy (reduction in dendritic length, DL) and debranching (reduction in the number of branch points, BP). Thus, our experimental data (see the dotted box in Figure 1) provide region-wise statistics, for both control and stress-treated neurons, on atrophy and debranching in each concentric shell that constitutes the basic unit of two-dimensional morphometric analysis in Sholl’s method (see Figure 2). These provide us with the reductions in DL and BP observed in neurons from stress-treated animals with respect to control animals. The goal of the proposed algorithm is to enforce these experimentally observed reductions on three-dimensional (3D) neuronal reconstructions in a region-specific manner. We use digital reconstructions of CA3b pyramidal neurons from the Duke-Southampton Archive (DSArchive) as inputs to our algorithm (Cannon, Turner, Pyapali, & Wheal, 1998). There are two problems

Apical k=A

Region-Specific Remodeling of Dendrites

79

7

50 Pm

6

SAB(6)=2

5

SAD(5)

4 3 2 1

Basal k=B

0

50 Pm

Figure 2: Illustration of Sholl’s analysis and notations used. Two-dimensional projection of 3D neuronal reconstruction overlaid with concentric shells radiating out in steps of 50 µm from the center of gravity (CoG) of the soma. The number within each shell corresponds to n; the apical and basal sides of the dendritic tree are differentiated by k = A and k = B, respectively. In this example, SAD (5) corresponds to the dendritic length in shell number 5 along the apical side of the tree and is represented by the shaded region. SAB (6) corresponds to the number of BPs in shell number 6 along the apical side of the tree and is equal to 2 in this case (the locations of the two BPs are indicated by arrows).

80

R. Narayanan, A. Narayan, and S. Chattarji

that we face in specifying the actual reductions in each concentric shell used in Sholl’s analysis. First, there are considerable differences in values of DL and BP for neurons from the experimental data (two-dimensional, 2D) and DSArchive (3D). Second, each dendritic tree, for a given neuron from either database, has its own intershell variations in actual values for DL and BP. We overcome these problems by implementing an algorithm that uses the experimentally observed ratios of reduction in DL and BP as the key parameter. This ratio in turn is enforced by the algorithm on 3D neuronal reconstructions from the DSArchive. In implementing this, in section 5, we first subject digital reconstructions of CA3b pyramidal neurons, from the DSArchive, to 3D Sholl’s analysis to obtain the DLs and BPs at various distances. Next, we combine this output with experimental statistics to arrive at the pruning specifications for this neuron. We accomplish this in section 6 by arriving at DL and BP pruning specifications at various distances by sampling the ratio distribution of corresponding random variables, which are specified by experimental statistics. These specifications are then used to set the probabilities of pruning along various distances (see section 7). Finally, an iterative algorithm (see section 8) employs these probabilities to subject the 3D reconstruction to levels of dendritic atrophy and debranching that are the same as those observed in our experiments (Vyas et al., 2002). We present the results of the proposed algorithm in section 9 and discuss the implications of the study in section 10.

3 Notations As a first step in our algorithm, we divide the dendritic tree into concentric spherical shells in order to perform Sholl’s analysis on the neuron to be pruned (see section 5). Figure 2 illustrates this diagrammatically, along with providing a reference to the notations. Nk , with k ∈ {A, B}, represents the number of spherical shells along apical (k = A) and basal (k = B) sides of the dendritic tree. For the neuron shown in Figure 2, NA = 8 and NB = 4. skl (n) and ckl (n) represent discrete random processes defining the BP and DL statistics in stress-treated and control animals, respectively. k, as above, denotes the class of the dendrite (apical or basal), whereas l represents whether the random process corresponds to BPs (l = B) or to DL (l = D), that is, l ∈ {B, D}, n = 0, 1, . . . , Nk − 1. For instance, sAD (5) represents the random variable defining the DL in the fifth shell (250–300µm) along the apical branches in neurons from stress-treated animals (see Figure 2). µskl (n) and µckl (n) represent the mean value of the process skl (n) and ckl (n), respectively, and σkls (n) and σklc (n) represent their respective standard deviations. These are obtained directly from the DL and BP statistics of stresstreated and control animals (Vyas et al., 2002).

Region-Specific Remodeling of Dendrites

81

Skl (n) represents the DL, obtained by subjecting the neuron to Sholl’s analysis (see below), of a given 3D neuronal reconstruction with l = D and l = B, respectively (see Figure 2 for an illustration). As above, k ∈ {A, B} represents the side of the shell (apical or basal), and n = 0, 1, . . . , Nk − 1 gives the shell number. ζkl (n), n = 0, 1, . . . , Nk − 1, k ∈ {A, B}, and l ∈ {B, D} are the pruning specifications, giving the amount of DL or BP to be removed from each given shell on the apical and basal sides of the neuron. Explicitly, running Sholl’s analysis at the end of the pruning process should yield Skl (n)−ζkl (n) ∀ k, l, n. The probabilities for pruning DL and BP are denoted by ρkl (n), k ∈ {A, B}, l ∈ {B, D}, n = 1, . . . , Nk − 1, which will be derived from the pruning specifications (see section 7). As the algorithm is an iterative one, it reduces 1 µm of dendritic length or one branching point at a time. In order to update the specifications for and probabilities of pruning at each iteration, it is necessary to keep track of the current levels of pruning, which is referred to as φkl (n), k ∈ {A, B}, l ∈ {B, D}, n = 0, 1, . . . , Nk − 1. 4 Assumptions 1. Radial isotropy is maintained in stress-induced dendritic atrophy, that is, statistics of atrophy observed in 2D morphometric data extends to the 3D case. 2. The random processes skl and ckl , corresponding to stress and control statistics, respectively, are independent. 3. The random variables skl (n), n = 0, 1, . . . , Nk − 1 are independent, as are ckl (n), n = 0, 1, . . . , Nk − 1. This assumption, given the nature of dendritic arborization, does not hold. For instance, if the number of points in shell Nk − 2 is zero, then the number of points in shell Nk − 1 has to be zero as well. However, the impact of this assumption is reduced because the pruning process takes the connectivity of the dendrites into account. 4. The random variables skl (n) and ckl (n), n = 0, 1, . . . , Nk − 1 conform to a normal distribution. The normal distribution with mean µ and standard deviation σ is denoted by N(x; µ, σ ): N(x; µ, σ ) = √


(x − µ)2 . exp − 2σ 2 2πσ 1

(4.1)

5 Three-Dimensional Sholl’s Analysis The first step in reproducing dendritic remodeling of model neurons similar to experimentally observed atrophy involves performing Sholl’s analysis (Sholl, 1953) on digitally reconstructed neurons obtained from the

82

R. Narayanan, A. Narayan, and S. Chattarji

DSArchive. Neuronal morphology in the DSArchive is stored in the SWC format (Cannon et al., 1998), which consists of seven field data lines, each defining a neuronal compartment. Each SWC data line has information about the 3D coordinates of the compartment, its radius, its parent in the dendritic tree, and a type code specifying the location of the compartment (in soma, axon, basal dendrite, or apical dendrite). The 3D coordinates of the various compartments (taken from the corresponding SWC data lines) are used to reconstruct the neuron by connecting straight lines between them. Once this is done, we have access to the exact coordinates of all points along the soma and the dendritic tree. We then perform 3D Sholl’s analysis on these neurons as follows (see Figure 2): 1. Calculate center of gravity (CoG) of the soma. Calculate the mean of the 3D coordinates of points associated with the soma and assign that as the center of gravity of the soma. 2. Assign shells. Construct a set of concentric spheres radiating out in steps of 50 µm from the CoG of the soma. The annular regions between these spheres form the shells, which form the basic unit of morphometric analysis using Sholl’s analysis (see Figure 2). In keeping with the 3D nature of our analysis (see assumption 1, above), we employ spherical shells instead of circular shells, which are used in the traditional 2D version of Sholl’s analysis (Sholl, 1953; Vyas et al., 2002). 3. Measure DL. Measure the length of dendrites within a given shell, n, by superimposing the dendritic tree on these set of concentric spheres. Assign this as SkD (n). 4. Count BPs. Count the number of points in a given shell, n, having more than one child, and assign that as SkB (n). In both of the above cases, the counts for the apical (k = A) and basal (k = B) sides are done separately. 6 Specification of the Pruning Schedule The goal of the specification process is to ensure that the loss of dendritic arborization in model neurons reflects experimental data on stress-induced dendritic atrophy, where apical dendrites undergo greater atrophy than their basal counterparts and each shell along both apical and basal dendrites undergoes atrophy depending on their distance from the soma. For example, it has previously been reported (Watanabe, Gould, & McEwen, 1992; McEwen, 1999; Vyas et al., 2002) that CA3 apical dendrites in the stratum radiatum layer undergo maximal atrophy, which is in sharp contrast to little or no atrophy of basal dendrites in the stratum oriens layer. The specification process involves setting the portion of DL to be removed from SkD and the number of BPs to be removed from SkB , in each of the concentric shells used in Sholl’s analysis. These reductions, denoted

Region-Specific Remodeling of Dendrites

83

as ζkl (n), n = 0, 1, . . . , Nk − 1, k ∈ {A, B}, l ∈ {B, D}, are set as samples of the ratio distribution (Marsaglia, 1965) involving the DL and the BP statistics of stressed (sAD (n)) and control (cAD (n)) animals (Vyas et al., 2002). For instance, to specify the amount of DL to be pruned in shell n on the apical (n) side, we take a sample z of the distribution csAD (see equation 6.3) and set AD (n) ζAD (n) = (1 − z)SAD (n). The following algorithm elaborates on this specification procedure:1 Algorithm SetSpecifications Inputs: µs , µc , σ s , σ c Output: ζ 1. Obtain random numbers ζ s and ζ c corresponding to the (independent) distributions N(x; µs , σ s ) and N(x; µc , σ c ), respectively, using s the Polar method (Knuth, 1997). Find the ratio ζζ c . 2. The ratio r=

ζs ζc

is an instance of the random variable,

µs + σ s x , µc + σ c y

(6.1)

where x and y are independent standard normal distributed random variables. Map r to the representation of Marsaglia’s ratio distribution (Marsaglia, 1965) as follows: r=k

a+x ; b+y

k=

µs µc σs , a = s, b = c. c σ σ σ

(6.2)

a+x is given as (Marsaglia, The probability density function of kr = b+y 1965) 
 q q exp(−0.5(a2 + b2 )) g(z)dz , 1 + f (t) = π(1 + t2 ) g(q) o at + b , (6.3) q= √ 1 + t2

where g(z) is the density function of the standard normal deviate. s As ζζ c is an instance of r, to obtain the corresponding instance of kr = s a+x , scale ζ c by k and set  t as: b+y

ζ

1 ζs σ cζ s  t= = . k ζc σ sζ c

(6.4)

1 For brevity, a general method to generate ζ , given S, µs , µc σ s , and σ c , is presented. This holds for all ζkl (n) given Skl (n), µskl (n), µckl (n) σkls (n) and σklc (n); k ∈ {A, B}, l ∈ {B, D}, n = 0, 1, . . . , Nk − 1.

84

R. Narayanan, A. Narayan, and S. Chattarji

3. Find f (t =  t) from equations 6.3 and 6.4. 4. if (0 ≤ f ( t) ≤ 1 & f ( t) ≥ 0.75 ∗ maxt ( f (t))) accept

ζs ζc

as a representative sample of r.   s return ζ = S 1 − ζζ c as the amount to be pruned. else goto Step 1. Step 4 of the above algorithm ensures the following: • f (t =  t) lies within the [0, 1] range, such that there is no negative pruning and there is no specification greater than 100% of actual BP or DL. • f (t =  t) is always greater than 75% of the maximum value of f (t). This is to make sure that the pruning specification approaches the mean distribution of the original data. The problem of matching DLs arises when the algorithm is applied to neurons obtained from the DSArchive. Experimental data provide pruning statistics for apical DLs up to 400 µm and 300 µm for basal dendrites (cf. Figure 1 of Vyas et al., 2002). However, CA3b pyramidal neurons from the DSArchive have DLs greater than these experimentally reported values. In order to circumvent this problem, statistics for shells with DL less than and equal to these values are set with statistics from Vyas et al. (2002). The statistics of the last shell (shell 8 on the apical side and shell 6 on the basal side) are replicated for shells with DL greater than these values. 7 Specification of Pruning Probabilities As outlined in Figure 1, once the pruning specifications are set, we need to set probabilities of pruning for DL or BP in any given shell based on the specifications obtained from the SetSpecifications algorithm. Intuitively, if the specification mentions that a given shell has to undergo higher pruning with respect to the other shells, then the probability of pruning for that shell has to be higher. Because the pruning algorithm we employ is an iterative one, within a given shell, we also take into account the current pruning values to specify the probabilities of subsequent pruning. 7.1 DL Pruning Probabilities. These are computed by normalizing the current pruning specifications such that the sum of the probabilities is one. Specifically, the DL pruning probabilities are set as follows: ρkD (n) = 

ζkD (n) − φkD (n) , n = 1, . . . , Nk − 1. k,n (ζkD (n) − φkD (n))

(7.1)

7.2 BP Pruning Probabilities. Although DL and BP pruning are different numbers in terms of specifications and experimental statistics, they are

Region-Specific Remodeling of Dendrites

85

physically linked, as they constitute the same dendritic tree. An important goal of our algorithm is to ensure that the reduction schedules of BP and DL remain in synchrony through all the iterations of the algorithm. Given that for a specific shell n, ζkB (n) number of BPs have to be pruned for a ζkD (n) reduction in DL, we enforce this by setting the BP pruning prob(n) abilities such that, on average, one BP is removed for every ζζkD reduction kB (n) in DL: ρkB (n) = ρkD (n)P

with

P=

ζkD (n)φkB (n) . ζkB (n)φkD (n)

(7.2)

In order to explain the motivation behind the above equation, we rewrite it as the following set of equations: ρ kB (n) = ρkD (n)P1

with

ρkB (n) = ρ kB (n)P2

with

ζkD (n) ζkB (n) φkB (n) P2 = . φkD (n) P1 =

(7.3) (7.4)

The expression for ρ kB (n), equation 7.3, states that for one BP to be pruned, as per the specifications, on average, P1 units of DL are required to be pruned. Equation 7.4 modulates this base expression with current pruning values φkl (n), ensuring that BP pruning runs according to schedule with respect to DL pruning. It may be noted here that the value of P in equation 7.2 determines whether BP pruning is on par or not with respect to DL pruning. This can be easily confirmed if P is seen as the ratio of P1 and P12 . If P > 1, then BP pruning is ahead of schedule; in this case, ρkB (n) is set to be less than ρkD (n) as per equation 7.2, which means that the probability of pruning a BP has been decreased with respect to that of pruning a DL, thus forcing BP pruning to slow down. Similarly, if P > 1, BP pruning would be running behind schedule and ρkB (n) > ρkD (n), meaning that BP pruning becomes more probable than DL pruning. Finally, BP pruning is on schedule with respect to DL pruning if P = 1. Effectively, P, which gets updated at each iteration of the algorithm (being dependent on φkl (n)), ensures the requirement that, (n) on average, one BP is removed for every ζζkD reduction in DL. kB (n) 8 Pruning the Dendritic Tree Our pruning algorithm is an iterative process that selects a given shell in each iteration, based on the pruning probabilities derived above, and prunes either 1 µm of DL or one BP. An adaptation of the rejection method for generating random numbers (Knuth, 1997) is employed to select the shells within which pruning of DL and BP is to be executed. Similar methods have been used in models of dendritic growth and for pruning simple, virtual dendritic trees (van Pelt, Dityatev, & Uylings, 1997; van Pelt, 1997).

86

R. Narayanan, A. Narayan, and S. Chattarji

The steps involved in pruning DL (in steps of 1 µm) or a BP, as part of a single iteration, are: 1. Select a BP or a terminal dendritic point randomly using a uniformly distributed random number generator. 2. Find the shell number  n to which the point belongs. 3. Generate a random number rand from a uniform distribution. 4. DL pruning: If (rand < ρkD ( n)), prune 1 µm length starting from the terminal point, reduce φkD ( n) by unit value, and update ρkD ( n) and ρkB ( n) as in equations 7.1 and 7.2, respectively.  n) m∈S ρkD (m)Rm ), prune the dendrite 5. BP pruning: If (rand < ρkB ( that connects the BP to a terminal point, reduce φkB ( n) by one, and update φkD ( n). Modify ρkD ( n) and ρkB ( n) as in equations 7.1 and 7.2, respectively. kl (n) The algorithm converges when maxk,l,n ζkl (n)−φ < T (with a default ζkl (n) value of T = 0.001), where T gives the allowance threshold for the maximum error in specifications. If the algorithm does not converge satisfactorily in certain shells due to the organization of the dendrites in some trees, postprocessing is done on those shells alone to make sure that the pruned tree meets the specifications. In the above scheme, we use the fact that a BP ceases to exist if one of the two subtrees originating from it is completely removed. We remove a BP by removing all the dendritic points present between it and its corresponding terminal point (with no other BP in between). For instance, if the BP represented by BPt in Figure 3 is to be pruned, one has to remove either DLtt1 or DLtt2 , so that BPt ceases to be a BP. This removal of dendritic length should also be taken into account in setting the probability of pruning a BP. This is done by revising the probability for pruning the BP as (step 5 above):

  ρkB (n) = ρkB ( n)



ρkD (m)Rm ,

(8.1)

m∈P

n) is as in equation 7.2. The set {Rm : m ∈ P } refers to the denwhere ρkB ( dritic points that are present between the pruned BP and the corresponding terminal point, where P corresponds to the set of all the Sholl shells through which these sets of points traverse. With reference to the illustration in Figure 3, if DLtt1 is chosen for pruning, then P = {3, 4, 5} as DLtt1 traverses through shells 3, 4, and 5 (see Figure 3). Pruning a nonterminal branch point involves more constraints than pruning a terminal branch point. The difference between pruning a terminal and nonterminal branches is depicted in Figure 3. The probability of pruning the terminal branch BPt is a product of the probability of pruning BPt alone and pruning DLtt1 through P , as given by equation 8.1. On the other hand,

Region-Specific Remodeling of Dendrites

87

DLtt2

5 4

1

tt DL

BPt

3 BPn

t

n DL

2

1 Figure 3: Illustration of pruning terminal and nonterminal BPs. Dendritic subtree overlaid on concentric shells, used in Sholl’s analysis, indicating typical terminal (BPt ) and nonterminal (BPn ) BPs. DLnt corresponds to the DL between BPn and BPt . DLtt1 and DLtt2 represent DLs between BPt and its terminal points. The numerical indices on the left correspond to the shell numbers.

the probability of pruning the nonterminal branching point BPn would be a product of the probabilities of pruning BPn alone, pruning DLnt , pruning BPt , and pruning DLtt1 and DLtt2 . This product turns out to have a very low value for all nonterminal branch points, and omitting these did not cause any difference in the pruning procedure. Hence we do not consider nonterminal branching points for pruning unless they eventually become terminal branches during the pruning process. 9 Results A Linux system running on a Pentium IV processor is used for all computations. The pruning algorithm is implemented in C++. A typical execution of the algorithm takes less than 1 minute for a CA3b pyramidal cell from the DSArchive. The total DL of typical CA3 pyramidal neurons is in the range of 1.2 mm to 1.5 mm, with the apical tree covering around 0.7 mm of the total length (approximately 60–65% of the total DL). The total number of BPs is around 60 to 75 in these neurons.

88

R. Narayanan, A. Narayan, and S. Chattarji

In order to test the algorithm, we generate 100 sets of specifications using the SetSpecifications algorithm, and run the pruning algorithm with these specifications. We then perform Sholl’s analysis on these pruned neurons and plot their statistics (see Figure 4). Figure 4A illustrates, as a function of the radial distance from the soma, the region-specific reduction in DL of the pruned neurons (equivalent to an experimental neuron that has undergone stress-induced atrophy) with respect to the original neuron (equivalent to a control unstressed neuron). Figure 4B displays the effects on the number of BPs. Several features of the results depicted in Figure 4 are particularly relevant with respect to experimental data on stress-induced dendritic atrophy. First, the most significant pruning is evident at a distance of 100 to 300 µm from the soma on apical dendrites, that is, in the stratum radiatum of area CA3. This is in agreement with experimental observations in several studies (Vyas et al., 2002; McEwen, 1999; Watanabe et al., 1992). Second, we carried out a more quantitative analysis to further validate this qualitative agreement between experimental data and results obtained from our pruning algorithm (see Figure 5). As described in section 6, the algorithm is designed to emulate the ratio of pruning BP and DL along all dendritic shells. Thus, we compare the modes of the reduction distribution2 obtained from experimental data and from the outcomes of our algorithm for each shell along the apical and basal sides. The methods used to compute the experimental and algorithmic modes of the reduction distribution are explained in Figure 5A. This illustration employs the dendritic length of a shell located at 400 µm from the soma on the apical side as an example. We use biological atrophy data (Vyas et al., 2002) and equation 6.3 to generate the distribution of percentage reductions in the dendritic length. The percentage value at which this distribution attains its maximum corresponds to the experimental mode of the distribution (see Figure 5A, left). The distributions that are used for computing these modes are 1 − csklkl (n) (n) n = 0, 1, . . . , Nk − 1, k ∈ {A, B}, l ∈ {B, D}, and

(8) . for the given example, it is 1 − csAD AD (8) We then determine the algorithm modes by following these steps (see Figure 5A, right): (1) generate 100,000 different specifications on a given model neuron, (2) prune the given neuron exactly to these specifications using the algorithm, (3) obtain Sholl’s analysis for each of the 100,000 outcomes, (4) plot the histograms of each of Skl (n) for n = 0, 1, . . . , Nk − 1, k ∈ {A, B}, l ∈ {B, D} (i.e., the output of Sholl’s analysis) using all the 100,000 outcomes, and (5) compute the mode of the histograms. Figure 5A also

s (n)

2 kl ckl (n) corresponds to the ratio distribution of statistics from the stressed and control animals. Reduction distribution corresponds to the distribution of reduction in DL/BP in s (n) stressed neurons with respect to control neurons and is represented as 1 − ckl (n) . kl

Region-Specific Remodeling of Dendrites

A

89

Dendritic length (Pm)

Apical

Basal 2000

Original Pruned

1500 1000 500

500

400

300

200

100

0

100

200

300

Distance from soma (Pm)

Number of branching points

B

500

Apical

Basal 15

Original Pruned 10

5

400

300

200

100

0

100

200

Distance from soma (Pm) Figure 4: Pruning algorithm replicates experimental dendritic atrophy in 3D reconstructions of CA3 pyramidal cells. (A) Dendritic length (µm) as a function of the radial distance from the soma (µm). Open circles: Shell-wise DL for unpruned neuron; filled circles: mean and SEM of shell-wise DL of 100 neurons pruned with specifications set in the range of changes experimentally observed following chronic stress. (B) Branching points as a function of the radial distance from the soma (µm). Open circles: Shell-wise count of BPs on apical and basal sides for unpruned neuron; Filled circles: mean and SEM of shell-wise BPs of 100 neurons used in A.

90

R. Narayanan, A. Narayan, and S. Chattarji

0.3 0.2 0.1 0.0

73.3%

-100 0 100 200 Percentage of reduction in DL

Mode of dendritic length reduction distribution

B

Experimental 0.4

Normalized frequency

Probability of occurrence

A

Mode of branching point reduction distribution

C

Algorithm

0.008

MAX

0.006 0.75*MAX 0.004 0.002 0.000

74.2%

-100 0 100 200 Percentage of reduction in DL 90

Apical

80 70

Basal

60 50 40 30 20 10 0 50 100 150 200 250 400 350 300 250 200 150 100 50 Distance from soma (µm) Experimental 80 Algorithm Apical 70 Basal 60 50 40 30 20 10 0 50 100 150 250 200 150 100 50 Distance from soma (µm)

brings out the accuracy of the pruning procedure by showing that within the valid specification range (indicated by the solid vertical lines, Figure 5A, right), there is complete overlap between the pruning specifications (gray curve) and the outcomes of the algorithm (black curve). Using the process depicted in Figure 5A, modes of percentage reduction distribution for DL (see Figure 5B) and BP (see Figure 5C) are plotted for each shell as a function of its distance from the soma. As Figures 5B and 5C depict, this analysis confirmed that the modes of the reduction distributions

Region-Specific Remodeling of Dendrites

91

obtained from biological data (see Figures 5B and 5C, “experimental,” open bars) match the outcomes of our pruning algorithm (see Figures 5B and 5C, “algorithm,” gray bars) in all dendritic regions. The maximum discrepancy is found to be less than 5% in all comparisons involving the modes of DL and BP reduction along all shells on the apical and basal sides. Finally, a further point of agreement between our results and experimental data is with reference to the reduction in total dendritic length. Stressinduced atrophy in total dendritic length has been reported to be at around 30% in CA3 pyramidal cells (Watanabe et al., 1992; Vyas et al., 2002). This is in agreement with the outcome of our algorithm, which prunes 30% to 45% of the total dendritic length, while maintaining region specificity in the process (see Figure 6). It may also be observed from Figure 6 that the stratum radiatum (the region between the two dotted lines) has undergone maximal pruning relative to the other regions of the neuron, which is also consistent with the experimentally obtained data on stress-induced dendritic atrophy (Watanabe et al., 1992; Vyas et al., 2002). While the pruning algorithm is capable of eliciting specific patterns of dendritic remodeling that match experimental observations rather well, it should also be noted that our algorithm can prune a given tree to various degrees of atrophy under a specified probabilistic regime (see Figure 6). This helps in obtaining trees of any desired level of atrophy with the relative reductions in BP and DL across various shells maintained as per the specifications. This feature is a direct outcome of the iterative nature of the

Figure 5: Facing page. Agreement between shell-specific modes of percentage reduction distributions observed biologically and calculated from the algorithmic outcomes. (A) Illustration of the calculation of modes of the reduction distribution for DL, using the example of a shell located at 400 µm from the soma on the apical side. Experimental mode (left) is computed by finding the point at which the reduction distribution (obtained from equation 6.3) attains its maximum value. The mode of reduction distribution corresponding to the algorithm (right) is obtained by finding the histogram of Sholl’s analysis outcomes of the pruned neurons (curve highlighted in black). The plot in gray depicts the samples of the reduction distribution that were used to arrive at the pruning specifications. The 0.75*MAX value (dotted horizontal line) indicates the constraint imposed by the SetSpecifications algorithm (step 4) to determine the samples that are considered valid specifications. This, along with the other constraint that the sample has to lie between 0 and 100% (step 4 of the SetSpecifications algorithm), constitutes the difference between the gray and black curves. Within the valid specification range (indicated by the solid vertical lines), there is complete overlap between the pruning specifications (gray curve) and the outcomes of the algorithm (black curve). Using the process depicted in A, modes of reduction distribution for DL (B) and BP (C) are computed for each shell as a function of distance from the soma.

92

R. Narayanan, A. Narayan, and S. Chattarji

Stratum Radiatum

75%

60%

30%

Experimentally observed atrophy

45%

15%

0% 50 Pm

Figure 6: A wide range of dendritic atrophy can be achieved using the pruning algorithm. Projections of 3D reconstructions of a CA3 pyramidal cell and its various pruned versions. The percentage of pruning is given to the left of each projection. Pruning in the range of 30% to 45% reduction (reconstructions within the gray box) corresponds to the outcomes of the algorithm with specifications set in the range of atrophy observed experimentally after chronic stress in animal models (Vyas et al., 2002; Watanabe et al., 1992). The stratum radiatum (delineated by parallel dotted lines) undergoes maximal dendritic pruning and debranching, which is consistent with experimental data (Vyas et al., 2002; Watanabe et al., 1992).

Region-Specific Remodeling of Dendrites

93

algorithm, which can be stopped at any given iteration to obtain the tree of desired length. The relative reduction across shells is kept constant because the choice to prune a DL or BP is guided by the probabilistic regime through all the iterations. 10 Discussion In this study, we have employed chronic stress-induced dendritic atrophy in the hippocampus as a model to develop an algorithm to elicit systematic, region-specific remodeling of 3D reconstructions of CA3 pyramidal neurons. While it is standard practice to find the ratio of the means of two distributions to determine the amount of reduction, in our model, we have employed the actual ratio distribution to arrive at the statistics of the ratio. Using this ratio distribution, we have calculated region-based distributions of reduction in dendritic arborization, which are used to characterize the region-specific pruning probabilities at any given distance from the soma. These probabilities then drive an iterative algorithm, which probabilistically reduces either 1 µm of dendritic length or removes one branching point in each iteration. There are two distinct advantages of this algorithm. First, it prunes BP and DL in parallel, both within the same probabilistic framework, thus ensuring that the average pruning in DL per BP is maintained through all iterations (see equation 7.2). Second, it provides precise control over the remodeling process at any point of interest with the relative pruning across various regions still conforming to biological data. This helps in analyzing the causal structure-function relationship over a range of remodeling by maintaining region specificity throughout, thereby allowing us to monitor the evolution of the neuron’s biological function as the remodeling proceeds. It is also important to emphasize that although we used stress-induced dendritic atrophy as a test case, the algorithm presented here is a generalized one that can be applied to a wide range of biological problems involving dendritic remodeling. In other words, this algorithm can be applied to induce both growth and pruning of neuronal reconstructions. In problems involving dendritic growth, the only difference would be to add a BP or 1 µm of DL once a specific region is probabilistically selected. Further, for simulating dendritic growth, the diameter of the newly formed dendrites should also be fixed, which can be done using Rall’s branching rule (Rall, 1977). A related class of algorithms for generating virtual dendritic trees to match a given statistical framework is the L-Neuron package (Ascoli & Krichmar, 2000; Ascoli, Krichmar, Scorcioni, Nasuto, & Senft, 2001). Our algorithm is different from these because here we remodel real 3D reconstructions based on experimental data, whereas L-Neuron generates virtual neurons based on morphometric data. Further, for analyzing the effects of localized remodeling, our algorithm has the advantage of generating neu-

94

R. Narayanan, A. Narayan, and S. Chattarji

rons for possible analysis of causal structure-function relationships. This is possible because we remodel dendrites of a given neuron to various levels of atrophy, and the extent and location of the reduction are well known. This is in contrast to, say, having two groups of neurons generated to match the stress and control statistics and using them for analyzing the structurefunction relationship. Such an analysis can provide us with only correlative relationships rather than precise causal relationships, as the analysis is not confined to remodeling a given neuron. Given that systematic manipulation of dendritic morphology is currently impossible with biological neurons, the framework presented here provides us with a tool to quantitatively address questions related to specific contributions of dendritic structure to neuronal function during both neuronal development and experience-induced plasticity in adults. The region specificity of the outputs of the algorithm opens new avenues to address the exciting possibility that localized changes in dendritic structure translate to localized changes in neuronal function. It also allows us to analyze the effects of possible local modulation of ionic and synaptic channels as a result of dendritic remodeling. Such analysis may have significant implications for recent findings related to intrinsic plasticity in neurons (Frick et al., 2004). Acknowledgments This work was supported by a research grant from the Department of Biotechnology, India. We express grateful thanks to Ajai Vyas and Rupshi Mitra for helpful discussions. We also thank the anonymous reviewer for the positive comments and suggestions, which helped substantially in improving the presentation of this article. References Anderton, B. H., Callahan, L., Coleman, P., Davies, P., Flood, D., Jicha, G. A., Ohm, T., & Weaver, C. (1998). Dendritic changes in Alzheimer’s disease and factors that may underlie these changes. Progress in Neurobiology, 55, 595–609. Ascoli, G., & Krichmar, J. L. (2000). L-Neuron: A modeling tool for the efficient generation and parsimonious description of dendritic morphology. Neurocomputing, 32–33, 1003–1011. Ascoli, G., Krichmar, J., Scorcioni, R., Nasuto, S., & Senft, S. (2001). Computer generation and quantitative morphometric analysis of virtual neurons. Anatomy and Embryology, 204, 283–301. Berzhanskaya, J., Urban, N. N., & Barrionuevo, G. (1998). Electrophysiological and pharmacological characterization of the direct perforant path input to hippocampal area CA3. Journal of Neurophysiology, 79, 2111–2118. Bothwell, S., Meredith, G. E., Phillips, J., Staunton, H., Doherty, C., Grigorenko, E., Glazier, S., Deadwyler, S., O’Donovan, C. A., & Farrell, M. (2001). Neuronal hypertrophy in the neocortex of patients with temporal lobe epilepsy. Journal of Neuroscience, 21, 4789–4800.

Region-Specific Remodeling of Dendrites

95

Brizzee, K. R. (1987). Neurons numbers and dendritic extent in normal aging and Alzheimer’s disease. Neurobiology of Aging, 8, 579–580. Cannon, R., Turner, D., Pyapali, G., & Wheal, H. (1998). An on-line archive of reconstructed hippocampal neurons. Journal of Neuroscience Methods, 84, 49–54. Cline, H. T. (2001). Dendritic arbor development and synaptogenesis. Current Opinion in Neurobiology, 11, 118–126. Cossart, R., Epsztein, J., Tyzio, R., Becq, H., Hirsch, J., Ben-Ari, Y., & Crepel, V. (2002). Quantal release of glutamate generates pure kainate and mixed AMPA/kainate EPSCs in hippocampal neurons. Neuron, 35(1), 147– 159. Duan, H., Wearne, S. L., Rocher, A. B., Macedo, A., Morrison, J. H., & Hof, P. R. (2003). Age-related dendritic and spine changes in corticocortically projecting neurons in macaque monkeys. Cerebral Cortex, 13, 950–961. Frick, A., Magee, J., & Johnston, D. (2004). LTP is accompanied by an enhanced local excitability of pyramidal neuron dendrites. Nature Neuroscience, 7, 126– 135. Geula, C. (1998). Abnormalities of neural circuitry in Alzheimer’s disease: Hippocampus and cortical cholinergic innervation. Neurology, 51, S18–S29. H¨ausser, M., & Mel, B. (2003). Dendrites: Bug or feature? Current Opinion in Neurobiology, 13, 372–383. Johnston, D., & Amaral, D. G. (1997). Hippocampus. In G. Shepherd (Ed.), Synaptic organization of the brain (4th ed.). New York: Oxford University Press. Johnston, D., Magee, J. C., Colbert, C. M., & Cristie, B. R. (1996). Active properties of neuronal dendrites. Annual Review of Neuroscience, 19, 165–186. Jones, T. A., & Schallert, T. (1994). Use-dependent growth of pyramidal neurons after neocortical damage. Journal of Neuroscience, 14, 2140–2152. Kaufmann, W. E., & Moser, H. W. (2000). Dendritic anomalies in disorders associated with mental retardation. Cerebral Cortex, 10, 981–991. Keyvani, K., & Schallert, T. (2002). Plasticity-associated molecular and structural events in the injured brain. Journal of Neuropathology and Experimental Neurology, 61, 831–840. Knuth, D. E. (1997). The art of computer programming, Vol. 2: Seminumerical algorithms (3rd ed.). Reading, MA: Addison-Wesley. Krichmar, J. L., Nasuto, S. J., Scorcioni, R., Washington, S. D., & Ascoli, G. A. (2002). Effects of dendritic morphology on CA3 pyramidal cell electrophysiology: A simulation study. Brain Research, 941, 11–28. Magee, J., Hoffman, D., Colbert, C., & Johnston, D. (1998). Electrical and calcium signaling in dendrites of hippocampal pyramidal neurons. Annual Review of Physiology, 60, 327–346. Mainen, Z., & Sejnowski, T. (1996). Influence of dendritic structure on firing pattern in model neocortical neurons. Nature, 382, 363–366. Marc, R. E., Jones, B. W., Watt, C. B., & Strettoi, E. (2003). Neural remodeling in retinal degeneration. Progress in Retinal and Eye Research, 22, 607–655. Marsaglia, G. (1965). Ratios of normal variables and ratios of sums of uniform variables. American Statistical Association Journal, 60, 193–204. McEwen, B. (1999). Stress and hippocampal plasticity. Annual Review of Neuroscience, 22, 105–122.

96

R. Narayanan, A. Narayan, and S. Chattarji

Migliore, M., & Shepherd, G. M. (2002). Emerging rules for the distributions of active dendritic conductances. Nature Reviews Neuroscience, 3, 362–370. Popov, V. I., Bocharova, L. S., & Bragin, A. G. (1992). Repeated changes of dendritic morphology in the hippocampus of ground squirrels in the course of hibernation. Neuroscience, 48, 45–51. Pyapali, G. K., & Turner, D. A. (1996). Increased dendritic extent in hippocampal CA1 neurons from aged F344 rats. Neurobiology of Aging, 17, 601–611. Rall, W. (1977). Core conductor theory and cable properties of neurons. In E. R. Kandel (Ed.), Handbook of physiology (Sec. 1, Vol. 1, pp. 39–97). Bethesda, MD: American Physiological Society. Ramakers, G. J. (2002). Rho proteins, mental retardation and the cellular basis of cognition. Trends in Neuroscience, 25, 191–199. Regehr, W. G., & Tank, D. W. (1994). Dendritic calcium dynamics. Current Opinion in Neurobiology, 4, 373–382. Segev, I., & London, M. (2000). Untangling dendrites with quantitative models. Science, 290, 744–750. Sholl, D. A. (1953). Dendritic organization in the neurons of the visual and motor cortices of the cat. Journal of Anatomy, 87, 387–406. Stuart, G., Spruston, N., & H¨ausser, M. (Eds.). (1999). Dendrites. New York: Oxford University Press. Traub, R. D., Jefferys, J. G. R., & Whittington, M. A. (1999). Fast oscillations in cortical circuits. Cambridge, MA: MIT Press. van Ooyen, A., Duijnhouwer, J., Remme, M. W., & van Pelt, J. (2002). The effect of dendritic topology on firing patterns in model neurons. Network: Computation in Neural Systems, 13, 311–325. van Ooyen, A., Willshaw, D. J., & Ramakers, G. J. A. (2000). Influence of dendritic morphology on axonal competition. Neurocomputing, 32–33, 255–260. van Pelt, J. (1997). Effect of pruning on dendritic tree topology. Journal of Theoretical Biology, 186, 17–32. van Pelt, J., Dityatev, A. E., & Uylings, H. B. M. (1997). Natural variability in the number of dendritic segments: Model-based inferences about branching during neurite outgrowth. Journal of Comparative Neurology, 387, 325–340. Vetter, P., Roth, A., & H¨ausser, M. (2001). Propagation of action potentials in dendrites depends on dendritic morphology. Journal of Neurophysiology, 85, 926–937. Vyas, A., Mitra, R., Rao, B. S. S., & Chattarji, S. (2002). Chronic stress induces contrasting patterns of dendritic remodeling in hippocampal and amygdaloid neurons. Journal of Neuroscience, 22, 6810–6818. Wang, Z., Xu, N. L., Wu, C. P., Duan, S., & Poo, M. M. (2003). Bidirectional changes in spatial dendritic integration accompanying long-term synaptic modifications. Neuron, 37, 463–472. Watanabe, Y., Gould, E., & McEwen, B. S. (1992). Stress induces atrophy of apical dendrites of hippocampal CA3 pyramidal neurons. Brain Research, 588(2), 341–345. Wong, R. O. L., & Ghosh, A. (2002). Activity-dependent regulation of dendritic growth and patterning. Nature Reviews Neuroscience, 3, 803–812. Received February 3, 2004; accepted June 2, 2004.

LETTER

Communicated by Helge Ritter

Analysis of Cyclic Dynamics for Networks of Linear Threshold Neurons H. J. Tang [email protected]

K. C. Tan [email protected] Department of Electrical and Computer Engineering, National University of Singapore, Singapore 117576

Weinian Zhang [email protected] Department of Mathematics, Sichuan University, Chengdu, Sichuan, P.R. China 610064

The network of neurons with linear threshold (LT) transfer functions is a prominent model to emulate the behavior of cortical neurons. The analysis of dynamic properties for LT networks has attracted growing interest, such as multistability and boundedness. However, not much is known about how the connection strength and external inputs are related to oscillatory behaviors. Periodic oscillation is an important characteristic that relates to nondivergence, which shows that the network is still bounded although unstable modes exist. By concentrating on a general parameterized two-cell network, theoretical results for geometrical properties and existence of periodic orbits are presented. Although it is restricted to two-dimensional systems, the analysis can provide a useful contribution to analyze cyclic dynamics of some specific LT networks of high dimension. As an application, it is extended to an important class of biologically motivated networks of large scale: the winner-take-all model using local excitation and global inhibition. 1 Introduction Networks of linear threshold (LT) neurons with nonsaturating transfer functions have attracted growing interest (Ben-Yishai, Bar-Or, & Sompolinsky, 1995; Hahnloser, 1998; Wersing, Beyn, & Ritter, 2001; Xie, Hahnloser, & Seung, 2002; Yi, Tan, & Li, 2003). The prominent properties of an LT network are embodied at least in two categories. First, it is believed to be more biologically plausible, for example, in modeling cortical neurons, which rarely operate close to saturation (Douglas, Koch, Mahowald, Martin, & Suarez, 1995). Second, the LT network is ready for performing multistability analysis. A multistable network is not as computationally restrictive as a Neural Computation 17, 97–114 (2005)

c 2004 Massachusetts Institute of Technology


98

H. Tang, K. Tan, and W. Zhang

monostable network, such as in decision making. Its computational abilities have been explored in a wide range of applications. A class of multistable winner-take-all (WTA) networks was presented in Hahnloser (1998), and an LT network composed of competitive layers was employed to accomplish perceptual grouping (Wersing, Steil, & Ritter, 2001). An LT network designed in microelectronic circuits demonstrated both computation of analog amplification and digital selection (Hahnloser, Sarpeshkar, Mahowald, Douglas, & Seung, 2000). The transfer function is an important factor in bounding the network dynamics. Unlike the sigmoid function in Hopfield networks (Hopfield, 1984) and the piecewise linear sigmoid function in cellular networks (Chua & Yang, 1988), both of which ensure bounded dynamics and at least one equilibrium, nonsaturating transfer function may give rise to unbounded dynamics and even the nonexistence of equilibrium (Forti & Tesi, 1995). The existence and uniqueness of equilibrium of LT networks have been investigated (Hadeler & Kuhn, 1987; Feng & Hadeler, 1996). A stability condition such that local inhibition is sufficient to achieve nondivergence of LT networks was established (Wersing, Beyn et al., 2001). A wide scope of dynamics analysis, including boundedness, global attractivity, and complete convergence, was conducted in Yi et al. (2003). However, in analysis of the dynamics of the LT networks, little attention has been paid to theoretically analyzing the geometric properties of equilibria and periodic oscillations. It is often difficult to perform theoretical analysis for the cyclic dynamics of a two-dimensional system and impossible to study that of high-dimensional systems. It was reported that slowing the global inhibition produced periodic oscillations in a WTA network and the epileptic network approached a limit cycle by computer simulations (Hahnloser, 1998). Nevertheless, there was a lack of theoretical proof to the existence of periodic orbits. It also remains unclear what factors affect the amplitude and period of the oscillations. This article provides an in-depth analysis of these issues based on parameterized networks with two LT neurons. Generally, the analytical result is applicable only to two-cell networks; however, it can be extended to a special class of high-dimensional networks, such as a generalized WTA network using local excitation and global inhibition. The principal analysis of the planar system would provide a useful framework to study the dynamical behaviors and give insight into such LT networks. 2 Preliminaries The network with n linear threshold neurons is described by the additive recurrent model, ˙ = −x(t) + Wσ (x(t)) + h, x(t)

(2.1)

Analysis of Cyclic Dynamics

99

where x, h ∈ Rn are the neural states and external inputs and W = (wij )n×n the synaptic connection strengths. σ (x) = (σ (xi )) is called the LT function defined by σ (xi ) = max{0, xi }, i = 1, . . . , n. Due to the simple nonlinearity of threshold, network 2.1 can be divided into pieces of linear systems, and there are 2n such partitions for n neurons. In each partition, we are able to study the qualitative behavior of the dynamics in details. In general, equation 2.1 can be linearized as ˙ = −x(t) + W + x(t) + h, x(t)

(2.2)

and it must correspond to each partition. For a given partition, W + is com+ puted simply by w+ ij = wij for xj ≥ 0 and wij = 0 for xj < 0, for all i, j. Then it is easy to compute the equilibria of equation 2.2 by solving linear equation Jx∗ + h = 0, where J = (W + − I) is the Jacobian and I is the identity matrix. If J is singular, the network may possess infinite equilibria, for example, a line attractor (necessary conditions for line attractors have been studied in Hahnloser, Seung, & Slotine, 2003). If J is nonsingular, a unique equilibrium exists, which is explicitly given by x∗ = −J−1 h. It should be noted that if x∗ lies in this region, it is called a true equilibrium; otherwise, it is called a virtual equilibrium. 3 Geometrical Properties of Equilibria For a planar system, the phase plane is divided into four quadrants: R2 = D1 ∪ D2 ∪ D3 ∪ D4 , where D1 = {(x1 , x2 ) : x1 ≥ 0, x2 ≥ 0}, D2 = {(x1 , x2 ) : x1 < 0, x2 ≥ 0}, D3 = {(x1 , x2 ) : x1 < 0, x2 < 0}, and D4 = {(x1 , x2 ) : x1 ≥ 0, x2 < 0}. Obviously, D1 is a linear region, which is unsaturated. D2 and D4 are partially saturated, but D3 is saturated. The Jacobians of the two-cell network in each region are formulated explicitly,
w11 − 1 w21


−1 w12 , w22 − 1 0


w12 −1 , 0 w22 − 1


w11 − 1 0 , −1 w21

 0 , −1

adding the subscript 1, . . . , 4 when it is necessary to indicate their corresponding regions. Define δ =det J, µ =trace J, and = µ2 − 4δ. The dynamical properties of the network are determined by the relations among δ, µ, and according to the linear system theory in R2 (Perko, 2001): (1) If δ < 0 then x∗ is a saddle. (2) If δ > 0 and ≥ 0, then x∗ is a node that is stable if µ < 0 and unstable if µ > 0. (3) If δ > 0, < 0, and µ = 0, then x∗ is a focus that is stable if µ < 0 and unstable if µ > 0. (4) If δ > 0 and µ = 0, then x∗ is a center. Based on the theory, for the sake of conciseness, the following theorem is presented without detailed analysis:

100

H. Tang, K. Tan, and W. Zhang

Table 1: Properties and Distributions of the Equilibria. Distribution

Conditions

D1

(w11 − 1)(w22 − 1) < w12 w21 (w22 − 1)h1 ≥ w12 h2 (w11 − 1)h2 ≥ w21 h1 (w11 − 1)(w22 − 1) > w12 w21 (w11 − w22 )2 ≥ −4w12 w21 d < 2 (w22 − 1)h1 ≤ w12 h2 d>2 (w11 − w22 )2 < −4w12 w21 d < 2 (w11 − 1)h2 ≤ w21 h1 d>2 d=2 w22 < 1, h1 < w12 h2 /(w22 − 1), h2 ≥ 0 w22 > 1, h1 < w12 h2 /(w22 − 1), h2 ≤ 0 h1 < 0, h2 < 0 w11 < 1, h1 ≥ 0, h2 < w21 h1 /(w11 − 1) w11 > 1, h1 ≤ 0, h2 < w21 h1 /(w11 − 1)

D2 D3 D4

Properties Saddle

s-node u-node s-focus u-focus center s-node saddle s-node s-node saddle

Note: d = w11 + w22 , s: stable, u: unstable.

Theorem 1. A distribution of equilibria of the two-cell LT network, equation 2.1, and their geometrical properties are given for various cases of connection strengths and external inputs in Table 1.

4 Existence and Boundary of Periodic Orbits According to the results in theorem 1, the equilibrium x∗ is an unstable focus or a center if it satisfies that  w w < − 14 (w11 − w22 )2 ,    12 21 (w22 − 1)h1 < w12 h2 , (4.1) (w11 − 1)h2 < w21 h1 ,    w11 + w22 > 2(unstable focus), w11 + w22 = 2(center). If network 2.1 has a topological center in D1 , then it always exhibits oscillations around the equilibrium in D1 . However, the outmost of the periodic orbits needs further study. When x∗ is an unstable focus, the network may tend to be unstable, and it is nontrivial to determine if the periodic orbits exist. Remark 1. The Poincar´e-Bendixson theorem (Zhang, Ding, Huang, & Dong, 1992) holds that if there exists an annular region ⊂ R2 , containing no equilibrium, such that both orbits from its outer boundary γ2 and from its inner boundary γ1 enter into , then there exists a (may not be unique) periodic orbit γ in . If the vector field is analytic (can be expanded as a convergent power series), then the periodic orbits in are limit cycles. Since σ is not analytic, our attention is devoted to the existence of periodic orbits

Analysis of Cyclic Dynamics

101

instead of making more efforts to assert that of limit cycles. Actually, they do not make significant difference in the oscillatory behavior, though they are different in geometrical meaning. Let L1 and L2 be the two lines that satisfy x˙ 1 = 0 and x˙ 2 = 0, respectively:  L1 : (w11 − 1)x1 + w12 x2 + h1 = 0, L2 : w21 x1 + (w22 − 1)x2 + h2 = 0. When w12 < 0, w11 > 1, w22 < 1, w21 > 0, equation 4.1 describes a vector field as shown in Figure 1. The trajectories that evolve in D1 and D2 are also intuitively illustrated. R is the right trajectory in D1 that starts from point p and ends at point q(0, q). L is the left trajectory in D2 that starts from point q and ends at point r(0, r). The arrows indicate the directions of the vector fields. It can be seen that there exists a rotational vector field around the equilibrium. Theorem 2. The two-cell LT network, equation 2.1, must have one periodic orbit if it satisfies that w12 < 0, w22 < 1 and equation 4.1.

Figure 1: Vector fields described by the oscillation prerequisites. The trajectories in D1 and D2 (R and L , respectively) forced by the vector fields are intuitively illustrated.

102

H. Tang, K. Tan, and W. Zhang

Proof.

The equilibrium in D2 is computed by

x∗ =

1 1 − w22




 w12 h2 + (1 − w22 )h1 . h2

It is noted that x∗1 > 0, x∗2 > 0; hence, it is a virtual equilibrium lying in D1 . Although it does not practically exist, it still drives the trajectory to approach it or diverge from it along its invariant eigenspace. When w22 = 0, the Jacobian matrix in D2 has two distinct eigenvalues, λ1 = −1, λ2 = w22 − 1, as well as two linear independent eigenvectors, T 12 e1 = (1, 0)T , e2 = ( w w22 , 1) . First, consider the two cases for w22 < 1, where the subspaces spanned by e1 and e2 are both stable: i. 0 < w22 < 1. In this case, λ1 < λ2 , and let Ess = Span{e1 } and Ews = Span{e2 } denote the strong stable subspace and weakly stable subspace, respectively. Figure 2 shows the phase portrait. Each trajectory except the invariant line Ess approaches the equilibrium along a well-defined tangent line, Ews . Since (0, p) lies in Ess , each trajectory must intersect the x2 -axis at some point (0, r) where r > p. Therefore, a simple closed curve  can be constructed such that  = R ∪ L ∪ rp, which can be used as an outer boundary of an annular region. ii. w22 < 0. In this case, λ2 < λ1 , and let Ess = Span{e2 } and Ews = Span{e1 } denote the strong stable subspace and weakly stable sub-

Figure 2: Phase portrait for 0 < w22 < 1.

Analysis of Cyclic Dynamics

103

Figure 3: Phase portrait for w22 < 0.

space, respectively. The phase portrait is shown in Figure 3. Each trajectory except the invariant line Ess approaches the equilibrium along the well-defined tangent line, Ews . Since (0, p) lies in Ess , each trajectory that starts from (0, q) must intersect the x2 -axis at some point (0, r) where r > p. Thus, a simple closed curve  can be constructed such that  = R ∪ L ∪ rp. iii. w22 = 0. In this case, there is only a single eigenvector e1 , which corresponds to the negative eigenvalue of −1. Figure 4 shows there is only a single invariant subspace. Each trajectory approaches the equilibrium along a well-defined tangent line determined by e1 . Thus, the left trajectory intersects the x2 -axis at a point above (0, p). To summarize the results of the above cases, an outer boundary  of an annular region can be constructed such that  = R ∪ L ∪ rp. According to the Poincar´e-Bendixson theorem, there must be a periodic orbit in the interior of the domain enclosed by . Remark 2. It can be shown that the two-cell network, equation 2.1, can have no periodic orbit if w22 > 1 and w12 < 0. As analyzed in theorem 3, x∗1 < 0, x∗2 < 0, the equilibrium is a virtual equilibrium lying in D3 . Notice that the T 12 system has two linear independent eigenvectors, e1 = (1, 0)T , e2 = ( w w22 , 1) ; s u thus, it has a stable subspace E = Span{e1 } and an unstable subspace E = Span{e2 }. All the trajectories in D2 leave away from x∗ along the invariant space Eu . Therefore, no periodic orbit exits in this case.

104

H. Tang, K. Tan, and W. Zhang

Figure 4: Phase portrait for w22 = 0.

The periodic orbits resulting from a center type equilibrium are investigated below. The proof can be found in the appendix. Theorem 3. If τ = 0, w12 < 0, w22 < 1 and equation 4.1 holds for network 2.1, then the network produces periodic orbits of center type. The amplitude of its outermost periodic orbit is confined in the closed curve  = R ∪ L ∪ rp, as shown in Figure 1, where h2 , and 1 − w22 (w11 − 1)h2 − w21 h1 w12 h2 + (1 − w22 )h1 + . q = −w22 (1 − w22 )w12 w12 w21 − (w11 − 1)(w22 − 1)

p=

From the geometrical view, the trajectory can switch only between a stable mode and an unstable mode; switching between two unstable modes or two stable modes is not permitted. It is difficult to calculate the amplitude and period of a periodic orbit. Fortunately, by virtue of the analytical results for center-type periodic orbits in theorem 3 and the trajectory equations in temporal domain (referred to in the appendix), it is possible to estimate their variation trends—how they change with respect to θi , τ , and hi . An approximate relationship describing the amplitude, period, and external inputs can be stated as (1) q ∝ hi , q ∝ τ 2 , q indicating the amplitude, and (2) the period increases as τ increases, while analysis procedures are omitted.

Analysis of Cyclic Dynamics

105

5 Winner-Take-All Network A WTA network is an interesting and meaningful application of LT networks. The study of its dynamics, fixed points, and stability has great merit. A biologically motivated WTA model with local excitation and global inhibition was established and studied by Hahnloser (1998). A generalized model of such a network is formulated as the following differential equations: x˙ i (t) = −xi (t) + θi σ (xi (t)) − L + hi , n  ˙ = −L(t) + θj σ (xj (t)), τ L(t)

(5.1a) (5.1b)

j=1

where θi > 0 denotes local excitation, i = 1, . . . , n, L is a global inhibitory neuron, and τ is a time constant reflecting the delay of inhibition. It is noted that the dynamics of L has a property of nonnegativity: if L(0) ≥ 0, then L(t) ≥ 0, and if L(0) < 0, then L(t1 ) ≥ 0 after some transient time t1 > 0. In an explicit and compact form for x = (x1 , . . . , xn )T , the WTA model can be rewritten as









˙  −1 x(t) h x(t) σ (x(t)) , (5.2) + + = − ˙ 0 L(t) L(t) L(t) v 1 − τ1 where  = diag(θ1 , . . . , θn ), v= τ1 (θ1 , . . . , θn ), and 1 is a columnar vector of ones. Unlike in Hahnloser (1998), the generalized WTA model, equations 5.1a and 5.1b allow each neuron to have an individual self-excitation θi , which can be different from each other. Hence, it is believed to be more biologically plausible than requiring the same excitation for all neurons. By generalizing 1 the analysis (Hahnloser, 1998), it can be concluded that if θi > 1 and τ < θi −1 for all i, the network performs the WTA computation: xk = hk ,

(5.3a)

xi = hi − θk hk , i = k,

(5.3b)

L = θ k hk ,

(5.3c)

where k indicates the winning neuron. Nevertheless, it may not be the neuron having the largest external input. And all θi = 1 will produce an absolute WTA network, in the sense that the winner xk is the neuron that receives the largest external input hk . Hahnloser (1998) claimed that a periodic orbit may occur by slowing global inhibition somewhat. However, its occurrence is not known a priori. Consider a single excitatory-inhibitory pair of the dynamic system 5.1a and 5.1b, that is,







˙ θ −1 x(t) x(t) h σ (x(t)) + . (5.4) = − + θ ˙ L(t) 0 L(t) L(t) 1 − τ1 τ

106

H. Tang, K. Tan, and W. Zhang

Since each neuron is coupled only with a global inhibitory neuron (though such a global inhibitory neuron reflects the collective activities of all the excitatory neurons), it does not interact directly with other neurons. Therefore, the dynamics of n-dimensional WTA network can be clarified in the light of studying a two-cell network with a single excitatory neuron. The established results for any two-dimensional network in theorem 2 allow us to put forward the theorem on the existence of periodic orbits for the two-cell WTA model. Theorem 4. The single excitatory-inhibitory WTA network, equation 5.4, must have a periodic orbit if its local excitation and global inhibition satisfy θ > 1 and √ 1 θ −1

<τ <

Proof.

( θ+1)2 . (θ−1)2

The connection matrix and external inputs are, respectively,

W=

θ θ τ



−1 h ˆ . 1 , and h = 0 1− τ

Obviously, the first two conditions of theorem 2 are met since −1 < 0, 1− τ1 < 1. Now it remains to prove that condition 4.1 is also met by synaptic weights 1 and external inputs. From θ−1 < τ , it is easy to show that θ + 1 − τ1 > 2, which verifies the last inequality of equation 4.1. On the other hand, its first inequality is equivalent to
 θ 1 1 2 − <− . θ −1+ τ 4 τ

(5.5)

Then it immediately yields √ √ ( θ + 1)2 ( θ − 1)2 < τ < . (θ − 1)2 (θ − 1)2 Apparently,

√ ( θ−1)2 (θ−1)2

<

1 θ−1 .

Recalling τ >

√ ( θ + 1)2 1 . <τ < θ −1 (θ − 1)2

(5.6) 1 θ−1 ,

it is obtained that (5.7)

The next two inequalities of equation 4.1 are easy to verify by considering that hˆ 1 > 0, hˆ 2 = 0. Therefore, according to theorem 2, the WTA network must have a periodic orbit. The above analysis indicates that the parameters (θ and τ ), that is, the synaptic connections between excitatory and inhibitory neurons, are the

Analysis of Cyclic Dynamics

107

dominating factors on the stability and oscillation behavior. On the other hand, adding a new excitatory neuron to equation 5.4 only increases the strength of the global inhibition L, which would resemble adding a new excitatory-inhibitory pair (x2 , L) to the existing one (x1 , L). As a consequence, the stability of the existing pair would not change if the new pair is stable; hence, the whole network (x1 , x2 , L) remains stable. Otherwise, the existing pair would lose stability if the new pair is oscillating, and so the whole network oscillates. Based on the heuristic discussion above, we conjecture the following conditions that lead to oscillation behaviors of n-dimensional WTA networks, equations 5.1a and 5.1b. Rigorous proof of them is formidable. Nevertheless, extensive simulation studies support our observations. Proposition 1. The WTA network, equations 5.1a and 5.1b, loses stability and periodic oscillation√ occurs if its local excitation and global inhibition satisfy θi > 1 2 1 i +1) < τ < ( (θθ−1) and θi −1 2 for any i = 1, . . . , n. i

As mentioned, the strengths of local excitation and global inhibition (θi and τ ) play a determining role on the dynamical properties of the recurrent model. A complete description of the dynamics of the WTA model now can be concluded as i = 1, . . . , n, • θi ≤ 1 for all i results in a global stability. 1 for all i give rise to a WTA computation; it is • θi > 1 and τ < θi −1 absolute if all θi = 1.

• θi > 1 and

1 θi −1

<τ < √

√ ( θi +1)2 , (θi −1)2

for any i lead to periodic oscillations.

i +1) • θi > 1 and τ > ( (θθ−1) 2 for any i, implying too slow a global inhibition, i result in unstable dynamics due to a very strong unstable mode (in two dimensions, it is exactly an unstable focus) that give birth to divergent activities. 2

The study on cyclic dynamics of WTA networks is of special interest for cortical processing and short-term memory and efforts have been paid (Ellias & Grossberg, 1975; Ermentrout, 1992). A different WTA model using global inhibition was analyzed (Ermentrout, 1992), and conditions on the existence of periodic orbits were proven for a single excitatory-inhibitory pair by virtue of the Poincar´e-Bendixson theorem. Unlike model 5.4, the neuronal activities were saturated; thus, the outer boundary of an annular region is straightforward. In contrast, it is nontrivial to construct such an outer boundary for network 5.4 since nonsaturating transfer functions may give rise to unbounded activities. Different treatment of applying the Poincar´eBendixson theorem is also required by the fact that the WTA network of LT neurons differs from various models studied in Ermentrout (1992) due

108

H. Tang, K. Tan, and W. Zhang

to the nondifferentiability of LT transfer functions, which imposes another difficulty on cyclic dynamics analysis. Most interesting, observations similar to our analysis were made: the topology of the connections is the main determining factor for such WTA dynamics, and oscillatory behavior begins when the inhibition slows. 6 Examples and Discussions In this section, examples of computer simulations are provided to illustrate and verify the theories developed. Consider an example of WTA network 5.1a and 5.1b with six neurons. When τ = 2, θi = 2 for all i, the network becomes periodically oscillating (see Figures √ 5 and 6). It is verified by proposition 1, since τ is in the region 1 < τ < ( 2 + 1)2 where periodic orbit (undamped oscillation) occurs. As τ < 1, the network is asymptotically stable and performs the WTA computation, whereas it becomes unstable when √ τ > ( 2 + 1)2 .

25 20 15

Neural States

10 5 0 −5 −10 −15 −20 0

10

20

30

40

50

Time Figure 5: Cyclic dynamics of the WTA network with six excitatory neurons. τ = 2, θi = 2 for all i, h = (1, 1.5, 2, 2.5, 3, 3.5)T . The trajectory of each neural state is illustrated in 50 seconds. The dashed curve shows the state of the global inhibitory neuron L.

Analysis of Cyclic Dynamics

109

20

15

L

10

5

0

−5 −5

0

5

x

10

15

6

Figure 6: A periodic orbit constructed in the x6 − L plane. It shows that the trajectories starting from five random points eventually approach the periodic orbit.

To show the periodic orbits of center type, a simple network is taken as an example:
dx 2 = −x + 2 dt

−1 0



 σ (x1 (t)) 2 . + 1 σ (x2 (t))

Figure 7 gives an illustration of the trajectories starting from three different points: (0, 0.5), (0, 1), and (0, 1.2). As can be seen, each finally approaches a periodic orbit, where the inner closed curve shows a center. It is verified by theorem 3, as a computer illustration of Figure 1, where p = 1 and q = 3. The value of q can also be determined by solving the temporal equations of neural states (see the appendix). Further evaluating the temporal equations, A.10 to A.12, shows exactly 1 < r < 2. The established analytical results imply that slowing global inhibition in a multistable WTA network (Hahnloser, 1998) is equivalent to giving rise to an unstable focus, which would result in periodic orbits. Another important issue about LT networks is continuous attractors, for example, line attractors

110

H. Tang, K. Tan, and W. Zhang 6

5

x

2

4

q

3

2

r 1

p

0 −0.5

0

0.5

1

1.5

2

2.5

3

x1 Figure 7: Periodic orbits of the center type. The trajectories with three different initial points (0, 0.5), (0, 1) and (0, 1.2) approach a periodic orbit that crosses the boundary of D1 . The trajectory starting from (0, 1) constructs an outer boundary  = p qr ∪ rp such that all periodic orbits lie in its interior.

as discussed in Seung (1998) and Hahnloser et al. (2003). On the other hand, as shown in this article, a stable periodic orbit is another type of continuous attractor, which encircles an unstable stationary state. 7 Conclusion By concentrating on the analysis of a general parameterized two-cell network, this article studied the geometrical properties of equilibria and oscillation behaviors of the LT networks. The conditions for the existence of periodic orbits were established, and the factors affecting its amplitude and period were also revealed. Generally, the theoretical results are applicable only to a two-cell network; however, they can be extended to a special class of biologically motivated networks of more than two neurons: a WTA network using local excitation and global inhibition. The theory for the cyclic dynamics of such a WTA network was presented. Finally, simulation results illustrated the theory developed.

Analysis of Cyclic Dynamics

111

Appendix A.1 Proof of Theorem 3. Based on vector field analysis for D1 and D2 , it is known that the trajectory starting from a point (0, q) with q > s will intersect the x2 -axis again at a point (0, r) above (0, p). The trajectory starting from (0, r) must approach a periodic orbit at t → ∞. Thus, the right trajectory R , which starts from (0, p), and ends at (0, q), and its continued left trajectory L prescribe an outermost boundary such that every periodic orbit lies in the interior of the closed curve . Define x1 (t) = x1 (t) − x∗1 , x2 (t) = x2 (t) − x∗2 for t ≥ 0. Applying the trajectory equation in D1 , it is obtained that 
w212 2(w11 − 1)w12 (w11 − 1)2 2 x1 (t)2 x x (t) + (t) x (t) + 1 + 2 1 2 ω2 ω2 ω2
2 w12 w11 − 1 − x1 (0)2 = 0. (A.1) x1 (0) + x2 (0) − ω ω Suppose the trajectory intersects the x2 -axis again at (0, q), where it exists after time tR . Let x1 (tR ) = x1 (0) = 0 − x∗1 = −x∗1 . Then 
w212 2(w11 − 1)w12 (w11 − 1)2 2 x1 (0)2 x x (t ) + (0) x (t ) + 1 + 2 R 1 2 R ω2 ω2 ω2 w212 (w11 − 1)2 2 x x2 (0)2 (0) − 1 ω2 ω2 2w12 (w11 − 1) − x2 (0) − x1 (0)2 = 0, x1 (0) ω2 −

(A.2)

that is, w212 x2 (tR )2 + 2(w11 − 1)w12 x1 (0) x2 (0)2 x2 (tR ) − w212 − 2w12 (w11 − 1) x1 (0) x2 (0) = 0.

(A.3)

To solve the above equation, let φ = 4(w11 − 1)2 w212 x1 (0)2 + 4w212 (w212 x2 (0)2 + 2w12 (w11 − 1) x1 (0) x2 (0)) 2 2 = 4w12 ((w11 − 1) x2 (0)) , x1 (0) + w12 and it is obtained that x2 (tR ) =

−(w11 − 1) x1 (0) ∓ |(w11 − 1) x1 (0) + w12 x2 (0)| . w12

(A.4)

Since (w11 − 1) x2 (0) x1 (0) + w12 = (w11 − 1)x1 (0) + w12 x2 (0) − ((w11 − 1)x∗1 + w12 x∗2 ),

(A.5)

112

H. Tang, K. Tan, and W. Zhang

and x1 (0) = 0, x2 (0) = p = − w22h2−1 , (w11 − 1)x∗1 + w12 x∗2 + h1 = 0, it holds that
 h2 x1 (0) + w12 x2 (0) = 0 + w12 − (w11 − 1) − (−h1 ) > 0. (A.6) w22 − 1 Therefore, x2 (tR ) =

x2 (0)) x1 (0) ∓ ((w11 − 1) x1 (0) + w12 −(w11 − 1) . w12

(A.7)

x2 (0) when tR = 0, and as tR > 0, It is noted that one of the roots is x2 (tR ) = x2 (tR ) =

−(w11 − 1)(0 − x∗1 ) −

w12 h2 +(1−w22 )h1 1−w22

w12 w22 h2 w22 h1 =− − 1 − w22 w12 w12 h2 + (1 − w22 )h1 = −w22 . (1 − w22 )w12

(A.8)

Hence, q = x2 (tR ) + x∗2 w12 h2 + (1 − w22 )h1 (w11 − 1)h2 − w21 h1 = −w22 + . (1 − w22 )w12 w12 w21 − (w11 − 1)(w22 − 1)

(A.9)

A.2 Neural States Computed in Temporal Domain. The orbit of the system in D1 is computed by

µ 
w11 −w22 sin ωt + cos ωt 2ω t w21 2 ω sin ωt e × (x(0) − x ),

x(t) − xe = exp

w22 −w11 2ω

 sin ωt sin ωt + cos ωt

w12 ω

(A.10)

√ where ω = 12 | |. The orbit of the system in D2 is computed by
exp(−t) x(t) − x = 0 e

w12 w22 (exp((w22

− 1)t) − exp(−t)) exp((w22 − 1)t)

× (x(0) − xe ), w22 = 0,



(A.11)

or else
exp(−t) x(t) − x = 0 e

 w12 t exp(−t) w22 t exp(−t) + exp(−t)

× (x(0) − xe ), w22 = 0.

(A.12)

Analysis of Cyclic Dynamics

113

References Ben-Yishai, R., Bar-Or, R. L., & Sompolinsky, H. (1995). Theory of orientation tuning in visual cortex. Proc. Nat. Acad. Sci. USA, 92, 3844–3848. Chua, L. O., & Yang, L. (1988). Cellular neural networks: Theory. IEEE Trans. Circuits Systems, 35, 1257–1272. Douglas, R., Koch, C., Mahowald, M., Martin, K., & Suarez, H. (1995). Recurrent excitation in neocortical circuits. Science, 269, 981–985. Ellias, S. A., & Grossberg, S. (1975). Pattern formation, contrast control, and oscillations in the short term memory of shunting on-center off-surround networks. Biol. Cybernetics, 20, 69–98. Ermentrout, B. (1992). Complex dynamics in winner-take-all neural nets with slow inhibition. Neural Networks, 5, 415–431. Feng, J., & Hadeler, K. P. (1996). Qualitative behavior of some simple networks. Journal of Physics A, 29, 5019–5033. Forti, M., & Tesi, A. (1995). New conditions for global stability of neural networks with application to linear and quadratic programming problems. IEEE Trans. Circuits Systems, 42(7), 354–366. Hadeler, K. P., & Kuhn, D. (1987). Stationary states of the Hartline-Ratliff model. Biological Cybernetics, 56, 411–417. Hahnloser, R. H. R. (1998). On the piecewise analysis of networks of linear threshold neurons. Neural Networks, 11, 691–697. Hahnloser, R. H. R., Seung, H. S., & Slotine, J. J. (2003). Permitted and forbidden sets in symmetric threshold-linear networks. Neural Computation, 15(3), 621– 638. Hahnloser, R. H. R., Sarpeshkar, R., Mahowald, M. A., Douglas, R. J., & Seung, H. S. (2000). Digital selection and analog amplification coexist in a cortexinspired silicon circuit. Nature, 405, 947–951. Hopfield, J. J. (1984). Neurons with grade response have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. USA, 81, 3088–3092. Perko, L. (2001). Differential equations and dynamical systems. New York: SpringerVerlag. Seung, H. S. (1998). Continuous attractors and oculomotor control. Neural Networks, 11, 1253–1258. Wersing, H., Beyn, W. J., & Ritter, H. (2001). Dynamical stability conditions for recurrent neural networks with unsaturating piecewise linear transfer functions. Neural Computation, 13(8), 1811–1825. Wersing, H., Steil, J. J., & Ritter, H. (2001). A competitive layer model for feature binding and sensory segmentation. Neural Computation, 13(2), 357–387. Xie, X., Hahnloser, R. H. R., & Seung, H. S. (2002). Selectively grouping neurons in recurrent networks of lateral inhibition. Neural Computation, 14(11), 2627– 2646. Yi, Z., Tan, K. K., & Lee, T. H. (2003). Multistability analysis for recurrent neural networks with unsaturating piecewise linear transfer functions. Neural Computation, 15(3), 639–662.

114

H. Tang, K. Tan, and W. Zhang

Zhang, Z., Ding, T., Huang, W., & Dong, Z. (1992). Qualitative theory of differential equations (W. K. Leung, transl.). Providence, RI: American Mathematical Society.

Received October 14, 2003; accepted June 2, 2004.

LETTER

Communicated by Dennis Barbour

Nonlinear and Noisy Extension of Independent Component Analysis: Theory and Its Application to a Pitch Sensation Model Shin-ichi Maeda [email protected] Nara Institute of Science and Technology, Ikoma, Nara 630–0192, Japan

Wen-Jie Song [email protected] Osaka University, Suita, Osaka 565–0871, Japan

Shin Ishii [email protected] Nara Institute of Science and Technology, Ikoma, Nara, 630-0192 Japan

In this letter, we propose a noisy nonlinear version of independent component analysis (ICA). Assuming that the probability density function (p.d.f.) of sources is known, a learning rule is derived based on maximum likelihood estimation (MLE). Our model involves some algorithms of noisy linear ICA (e.g., Bermond & Cardoso, 1999) or noise-free nonlinear ICA (e.g., Lee, Koehler, & Orglmeister, 1997) as special cases. Especially when the nonlinear function is linear, the learning rule derived as a generalized expectation-maximization algorithm has a similar form to the noisy ICA algorithm previously presented by Douglas, Cichocki, and Amari (1998). Moreover, our learning rule becomes identical to the standard noise-free linear ICA algorithm in the noiseless limit, while existing MLE-based noisy ICA algorithms do not rigorously include the noise-free ICA. We trained our noisy nonlinear ICA by using acoustic signals such as speech and music. The model after learning successfully simulates virtual pitch phenomena, and the existence region of virtual pitch is qualitatively similar to that observed in a psychoacoustic experiment. Although a linear transformation hypothesized in the central auditory system can account for the pitch sensation, our model suggests that the linear transformation can be acquired through learning from actual acoustic signals. Since our model includes a cepstrum analysis in a special case, it is expected to provide a useful feature extraction method that has often been given by the cepstrum analysis.

Neural Computation 17, 115–144 (2005)

c 2004 Massachusetts Institute of Technology


116

S. Maeda, W.-J. Song, and S. Ishii

1 Introduction 1.1 Review of Independent Component Analysis. Blind source separation (BSS) is a problem that recovers original sources that are assumed to be independent of each other, from observations that are linear mixtures of the original sources. Let s = [s1 , . . . , sm ]T denote an m-dimensional original source variable, A an n × m linear mixing matrix, and an n-dimensional observation variable x = [x1 , . . . , xn ]T given as x = As.

(1.1)

The probability density function (p.d.f.) of the sources, p(s), is unknown, except that the mean of s is fixed at zero. When the mixing matrix A is square (n = m), a demixing matrix W for recovering the sources can be estimated in which y = Wx is component-wise independent. Due to the insufficiency of constraints in the BSS, however, the demixing matrix W is not uniquely determined and contains ambiguities with respect to permutation and scale of sources, such that W = PDA−1 where P and D are any n × n permutation and diagonal matrices, respectively. In the following we ignore these indeterminacies by identifying a demixing matrix W with PDW or a mixing matrix A with ADP. Various algorithms have been proposed to obtain an n × n square mixing or, equivalently, demixing matrix by defining varieties of cost functions, each based on the corresponding independence measure. Some of the cost functions are based on Kullback-Leibler (KL) divergence between p(y) and 
p(y) ˜ ˜ p(y) ≡ ni=1 pi (yi ), KL[p(y)  p(y)] = p(y) log p(y) ˜ dy, where y = Wx and  pi (yi ) = p(y)dy¯ i (Amari, Cichocki, & Yang, 1996). Here, y¯ i is a marginalized random vector, whose ith component yi is removed from the vector y, ˜ and p(y) is a distribution in which the components of y are independent of each other. Since the KL divergence is always positive, or zero if and only if ˜ p(y) is equivalent to p(y), it is appropriate to minimize the KL divergence in order to recover the independent sources. This approach, however, needs to know the marginal distribution pi (yi ). Another approach for obtaining the mixing matrix A is to maximize the entropy of z = f (y), where y = Wx and f is a component-wise monotonically increasing nonlinear function (Bell & Sejnowski, 1995). When the nonlinear function f becomes a cumulative distribution function of y, the entropy maximization of z is equivalent to the minimization of the KL divergence. This approach, however, also needs to ˜ know pi (yi ). The KL divergence between p(y) and p(y) has a relation  to “ne˜ gentropy” J(y) ≡ H(yg ) − H(y), namely, KL[p(y)  p(y)] = J(y) − ni=1 J(yi ) (Hyv¨arinen, 1998b). Here, H(y) is the entropy of random variable y, and yg is a gaussian random variable whose covariance matrix is the same as that of y. Since the negentropy J(y) is invariant with respect to an application of any invertible linear transformation to y, the minimization of the KL divergence is equivalent to the maximization of the sum of marginal negentropies

Nonlinear and Noisy Extension of Independent Component Analysis

117

n

J(yi ). This optimization problem also requires knowledge of pi (yi ). Accordingly, all of these approaches based on information-theoretical cost functions require knowledge of pi (yi ), though it is computationally difficult to estimate it. If we use kernel density estimation for approximating pi (yi ) from sample points, then, for example, the evaluation of pi (yi ) needs computation that is proportional to the sample number. Moreover, reestimation of pi (yi ) is necessary after every update of the demixing matrix W, which may occur after every data observation. To reduce the computational cost, therefore, the marginal distribution pi (yi ) is often restricted, and the estimation is performed in an on-line manner. For example, if the source’s p.d.f. is close to a gaussian distribution, it is well approximated by using several of the first terms of Gram-Chalier expansion or Edgeworth expansion (Gaeta & Lacoume, 1990; Amari et al., 1996), and those moments can be estimated in an online manner. Alternatively, pi (yi ) is parameterized or
fixed, hence is restricted. A supposition that the source distribution p(s) = ni=1 pi (si ) is parameterized or fixed is related to whether p(y) is parameterized or fixed, respectively. When p(s) is parameterized or fixed, the above approaches based on information-theoretical cost functions are similar to maximum likelihood estimation (MLE) of p(x|A), for given observation x. However, the MLE does not guarantee that the obtained parameter is the true one, when the assumed p.d.f. parametric family (model) does not include the true parameter. If the model incorporates not only an unknown parameter but also an unknown function like pi (yi ), suggesting that an infinite number of parameters are required to be free from the unknown function, then such an estimation problem is said to be semiparametric. Therefore, the BSS is a semiparametric problem. One way to deal with a semiparametric problem is to define an estimation function, a function of the parameter, which is zero if the parameter is identical to the true one, or nonzero otherwise, whatever the unknown function is. Although an approach based on an estimation function does not suppose any cost function, making the estimation function zero is a type of MLE in the case of BSS, because it is known that when the source distribution of BSS is parameterized or fixed, the derivative of the likelihood function with respect to the parameter can be regarded as a proper estimation function (Amari & Cardoso, 1997).1 Some other BSS algorithms based on cost functions such as a higher-order cumulant can also be regarded as the optimization of certain estimation functions (Cardoso & Souloumiac, 1993). Although all of the above approaches assume a linear noiseless square mixture, several extensions have been studied. One is a noisy mixture case, i=1

1 Although a proper estimation function guarantees that the optimized parameter is the true one, it describes neither the convergence nor convergence speed. See Amari, Chen, and Cichocki (1997) for the stability analysis.

118

S. Maeda, W.-J. Song, and S. Ishii

in which the observation process is modeled as x = As + n,

(1.2)

where n is a noise, usually obeying a gaussian. To deal with a noisy mixture case, not only an MLE approach with a certain parametrization of the source distribution (Belouchrani & Cardoso, 1995; Hyv¨arinen, 1998a; Bermond & Cardoso, 1999; Lewicki & Sejnowski, 2000; Zibulevsky & Pearlmutter, 2001; Girolami, 2001) but also an approach based on an estimation function that is free from the source distribution assumption (Hyv¨arinen, 1999; Kawanabe & Murata, 2000), have been proposed. The MLE approach treats the source signal s as a hidden variable. In this case, the likelihood function includes an integration over the hidden variable. Although the expectationmaximization (EM) algorithm (Dempster, Laird, & Rubin, 1977) is often used for MLE when the model includes hidden variables, the algorithm also requires an integration over the hidden variable to calculate expected likelihood. Since such integrations are often intractable, they are approximated by using point approximation with a delta function (Belouchrani & Cardoso, 1995; Olshausen & Field, 1996; Hyv¨arinen, 1998a; Rao & Ballard, 1999), Laplace approximation (Bermond & Cardoso, 1999; Lewicki & Sejnowski, 2000; Girolami, 2001), or mean-field approximation (HøjenSørensen, Winther, & Hansen, 2002). Another extension is a nonlinear mixture case. Because of indeterminacies in a general nonlinear BSS, an appropriate nonlinearity should be introduced. One such nonlinearity has been introduced in a postnonlinear mixture model (Jutten & Karhunen, 2003): x = f (As),

(1.3)

where the nonlinear function f is a component-wise monotonic function and the mixing matrix A is square (n = m). In this case, the nonlinear function and the mixing matrix are both uniquely determined, because the KL divergence is invariant with respect to the transformation by a componentwise monotonic function. In a nonlinear noise-free case, therefore, either the minimization of the KL divergence or the entropy maximization is used to estimate the demixing matrix W (Lee, Koehler, & Orglmeister, 1997; Taleb & Jutten, 1999; Almeida, 2004). However, such an information-theoretical approach suffers from ill-conditionness, because the independent sources cannot be obtained by a deterministic transformation like A−1 f −1 by itself, when the observation process involves a noise or the transformation is overcomplete (n < m). Estimation functions available for noise-free nonlinear mixture cases are currently unknown. In this letter, we deal with a situation in which the mixture is nonlinear and noisy: x = f (As) + n.

(1.4)

Nonlinear and Noisy Extension of Independent Component Analysis

119

There have been few studies that deal with noisy nonlinear mixture cases excepting a case that the source distribution is gaussian (Lappalainen & Honkela, 2000).2 Estimation functions that can be applied to noisy nonlinear mixture cases are unknown. For simplicity, the source distribution is fixed in this letter, though an extension to the case that the source distribution is a mixture of gaussians (Attias, 1999) will be possible. As a theoretical discussion, we present the relation between the MLE and the entropy maximization based on a deterministic transformation (deterministic entropy maximization), by means of the notion of probabilistic entropy maximization, which is an extension of deterministic entropy maximization to what allows a probabilistic sampling instead of deterministic transformation. Our discussion generalizes the equivalence between the MLE and the (deterministic) entropy maximization, presented by Cardoso (1997). To evaluate the likelihood function in the noisy nonlinear ICA, an integral over the hidden variable s is also necessary. For this reason, we employ an EM algorithm to maximize the likelihood, and the integral is performed by the Laplace approximation with a Taylor series expansion. In a special case that the nonlinear function is linear, an analytical solution for the M-step condition is obtained and the derived algorithm becomes identical to the noisy linear ICA presented by Bermond and Cardoso (1999). In a nonlinear case, on the other hand, an analytical solution is not obtained, though the algorithm is still derived as a generalized EM (GEM) algorithm. When the nonlinear function is linear and the noise is sufficiently small, the GEM algorithm becomes similar to the noisy linear ICA by using a “bias removal” technique (Douglas, Cichocki, & Amari, 1998). Moreover, the GEM algorithm under the assumption of linear mixture becomes identical in the noiseless limit to the standard ICA algorithm, which assumes a square noise-free linear mixture. The important difference is that our noisy nonlinear ICA algorithm based on MLE includes the noise-free linear ICA algorithm, while the other MLE-based noisy linear ICA algorithms do not include the noise-free linear ICA algorithm even in their special cases. We will later discuss why this discrepancy arises. 1.2 Pitch Sensation Model. A possible implication of ICA in the sensory cortex is feature extraction. Since the major function of the sensory cortex is to process its input “information,” it is natural that information theory provides some insights to the processing (e.g., Barlow, 1961; Atick, 1992). As an algorithm to realize information theory, ICA can provide a principle underlying the feature extraction done by the sensory cortex. Actually, ICA and a sparse coding scheme that assumes a sparse distribution for the sources in ICA successfully demonstrated the emergence of orientation-selective neu-

2 Because the standard BSS assumes the source distribution is nongaussian, this existing algorithm was called nonlinear factor analysis.

120

S. Maeda, W.-J. Song, and S. Ishii

rons (e.g., Olshausen & Field, 1996; Bell & Sejnowski, 1997; Rao & Ballard, 1999) and extraclassical receptive-field phenomena (Rao & Ballard, 1999) in the visual cortex. In the auditory system, Lewicki (2002) showed that the cochlear filter can be realized by bases trained from natural sounds within the formulation of ICA. In this article, we apply our nonlinear noisy ICA to reproducing some aspects of pitch sensation. Here, we briefly introduce the history of pitch sensation modeling and why ICA has a relationship with this psychoacoustic topic. Pitch is a psychological quantity that represents how high or low the sound is perceived when compared with the perception of a pure tone composed of a single sinusoidal wave. A sound wave, an air wave of condensation and rarefaction, is essentially a one-dimensional signal. Although there should be arbitrariness in detecting individual auditory sources (auditory streams) from such a onedimensional signal, animals are able to detect several or more auditory streams. This grouping of auditory streams is often called auditory stream segregation (Bregman & Campbell, 1971). Perceiving a pitch is an instance of such an auditory grouping. From its definition, it is easily understood that the perceived pitch is significantly dependent on frequency components of heard sound signals. However, various other factors are also involved with it. For example, one perceives a “middle C3 code” on a piano as a simple pitch, but the code consists of not only the pure tone corresponding to the perceived pitch, but also its harmonic components (Lindsay & Norman, 1977). Seebeck (1843) showed that even if the fundamental frequency of a harmonic combination tone is removed from the combination, one still perceives the same pitch as the tone frequency of the “missing fundamental.” This perception is called “virtual pitch,” “residue pitch,” or “low pitch.” We term this phenomenon “virtual pitch” in the rest of this article. Several hypotheses have been presented to account for the pitch perception. Helmholtz (1863) proposed a place theory in which the perceived pitch corresponds to the place that vibrates most on the basilar membrane of a cochlea, and virtual pitch occurs because the difference in components of a combination tone makes a similar place vibrate to that most vibrated by a missing fundamental tone, in a nonlinear fashion. However, later studies found that place theory cannot explain a pitch-shift phenomenon: one perceives an upward (or downward) shift of virtual pitch proportional to the upward (or downward) shift of a harmonic combination tone (de Boer, 1956; Schouten, Ritsma, & Cardozo, 1962). Because virtual pitch is due to the frequency difference of neighboring components in a harmonic combination, according to place theory, the pitch is not allowed to change when the harmonic combination is shifted with maintaining the combination structure. Place theory was, moreover, unsuccessful in explaining the observation in a masking experiment, in which band-limited noise is imposed on the fundamental frequency. In the experiment, a strong noise was incurred so as to disturb the nonlinear influence around the cochlear membrane vibrated

Nonlinear and Noisy Extension of Independent Component Analysis

121

most by the fundamental frequency. Although virtual pitch should disappear according to place theory, it was still perceived in the experiment. On the other hand, Schouten (1940) proposed a temporal theory, which explained the pitch perception in terms of temporal relationships among components in a combination tone. This theory supposed that one detects peaks of a sonic wave and perceives the interval of the peaks as pitch. However, later studies found that the pitch perception is not actually very sensitive to the phase shift of a combination tone, although the shift greatly changes the temporal pattern (Patterson, 1973), and pitch perception is not impaired when a combination of even-numbered multiple tones is presented to one ear and that of odd-numbered tones to the other ear (Houtsma & Goldstein, 1972). To explain these pitch perception phenomena, various central pattern transformation models were also proposed (Wightman, 1973; Goldstein, 1973; Terhardt, 1974), which fundamentally supposed linear transformation in the central auditory system after a peripheral spectral analysis. For example, in Wightman’s model, matrix  the transformation  (i−1)(2j−1) 2 is component-wise represented as Wij = n cos π for i = 0 2n  and Wij = n1 for i = 0, and an internal representation for pitch is represented as s = Wx, where x denotes the power spectrum of input sounds. When x is obtained in the continuous frequency domain, the above transformation matrix is simply represented by an interval-dependent function ∞ 1 s(τ ) = 2π x(ω) cos ωτ dω or, equivalently, an autocorrelation function −∞ T 1 s(τ ) = limT→∞ 2T −T v(t)v(t − τ )dt, where v(t) denotes the instantaneous sound pressure at time t and T the sound duration. As can be seen in the above expression, Wightman’s pattern transformation model is not influenced by a phase shift of the sounds, which is a desirable property of a pitch perception model. Existing central pattern transformation models usually have this property, because they often involve a peripheral spectrum analysis. Later studies revealed that various central pattern transformation models are equivalent to each other under specific conditions (de Boer, 1977; Yoshioka, 1980). Although the central pattern transformation models successfully accounted for several aspects of virtual pitch and pitch shift phenomena, the transformation matrix, which plays an essential role in the models, must be determined beforehand by the model’s designer. It is more likely that such a transformation matrix is acquired through interaction with environments, that is, through learning from external inputs. The aim of this study is to show that the transformation matrix can be obtained as a stable point through learning from natural speech and music signals, according to our noisy nonlinear ICA. Although the existing studies on the central pattern transformation models assume (noisy) linear transformation matrices, the nonlinearity is important in our learning-based approach to cover the dynamic range of natural sounds, which tend to be distributed in a log-normal

122

S. Maeda, W.-J. Song, and S. Ishii

fashion. Due to this feature of natural sounds, the sound intensity has often been measured in decibels to represent its psychological effect. Noisy setting is advantageous to relaxing the assumption of ICA; for example, it allows dimensional disagreement between input sound data and internal representation. After learning, virtual pitch phenomena were simulated, and the existence region of virtual pitch in the model was examined; the results are consistent with those by psychophysical experiments. Accordingly, our study implies that animals may have a mechanism to perform nonlinear noisy ICA instead of an inherent transformation matrix. This letter is organized as follows. In section 2, we show the equivalence between the MLE and the probabilistic entropy maximization. In section 3, we formulate our noisy nonlinear ICA and derive its learning rules, then discuss the relationship with existing ICA models. In section 4, we show pitch sensation simulations. Section 5 is presents our conclusion. 2 Probabilistic Entropy Maximization First, we show that the entropy maximization is equivalent to MLE in a more general condition than that assumed in the existing study (Cardoso, 1997). MLE is defined as maximize E[log p(x|θ)], θ

(2.1)

where x = [x1 , . . . , xn ]T is an n-dimensional random variable representing an input. E[·] denotes the expectation with respect to the true p.d.f. of x, q(x). We assume that the probabilistic model p(x|θ) has an m-dimensional hidden variable y,  p(x|θ ) =

p(y)p(x|y, θ)dy,

(2.2)

where the prior distribution for the hidden variable, p(y), is assumed to be uniform within a certain domain, for example, an m-dimensional hypercube, (0, 1)m . We further assume that x is generated by a parametric transformation φ from y plus a gaussian noise; namely, p(x|y, θ ) is given by     T 1 (2.3) exp −β x − φ(y; θ) −1 x − φ(y; θ ) , Z(θ)  where Z(θ) is a normalization term such that the integral p(x|y, θ )dx is 1. β > 0 is a constant hyperparameter, and 2 /β is a covariance matrix of the gaussian noise. φ(y; θ) denotes a basis function parameterized by parameter θ. The superscript T denotes the transpose. Probabilistic entropy maximization is defined by means of probabilistic sampling as follows. If y is p(x|y, θ) =

Nonlinear and Noisy Extension of Independent Component Analysis

123

sampled according to the sampling distribution p(y)p(x|y, θ ) p(x|θ ) and the model distribution p(x|θ) is sufficiently close to the true distribution q(x), the distribution of y, q(y), is approximated as q(y) ≡ ∫ q(x)p(y|x, θ )dx = p(x|y,θ ) p(y) ∫ q(x) p(x|θ) dx ≈ p(y). Since we have assumed that p(y) is uniform, q(y) also becomes uniform, implying that the entropy of q(y) is maximized under the condition that y’s support is fixed. The probabilistic entropy maximization maximizes the entropy of q(y) where y is probabilistically sampled as above. Since the above process is mathematically equivalent to the approximation of q(x) by p(x|θ), MLE is a method that performs probabilistic entropy maximization. However, probabilistic entropy maximization does not necessarily employ the same cost function as that of MLE; the solution by the probabilistic entropy maximization corresponds to the MLE solution only when the model p(x|θ ) is able to represent the true distribution q(x). Therefore, an MLE employing equations 2.2 and 2.3 is a special case of the probabilistic entropy maximization. Now we show that the probabilistic entropy maximization is reduced to the deterministic entropy maximization when the basis function φ(y; θ ) is invertible and β approaches infinity. By defining u ≡ φ(y) and ψ ≡ φ −1 , the following relations between u and y hold: dy = |J(u)|du

(2.4)

y = ψ(u),

(2.5)

where

|J(u)| is the absolute determinant of the Jacobian matrix J(u) ≡ ∂y ∂u. Then equation 2.3 becomes p(x|y) =

  1 exp −β(x − u)T −1 (x − u) ≡ p(u|x). Z(θ)

(2.6)

 p(u|x) ≡ p(x|y) is regarded as a p.d.f. of u because p(u|x)du = 1 and p(u|x) > 0. p(u|x) approaches a Dirac’s delta function, p(u|x) ≈ δ(u − x), as β goes to infinity. This means the sampling by p(y|x, θ ) ∝ p(x|y, θ )p(y) approaches the deterministic transformation y = ψ(x). By substituting equations 2.4, 2.5, and 2.6 into equation 2.2, we derive p(x|θ ) = |J(x)|,

(2.7)

where J(x) ≡ ∂y ∂x and y = ψ(x). Since the entropy of the transformed  variable y = ψ(x; θ) is given by H(y) = H(x) + q(x) log |J(x; θ )| dx, maximization of equation 2.7 is identical to the deterministic entropy maximization with respect to y, indicating that the probabilistic entropy maximization is reduced to the deterministic entropy maximization when the basis function φ(y; θ) is invertible and β approaches infinity. Also, an MLE of

124

S. Maeda, W.-J. Song, and S. Ishii

equations 2.2 and 2.3 is identical to the maximization of equation 2.7 when the basis function φ(y; θ) is invertible and β approaches infinity. The above discussion suggests two points: (1) the probabilistic entropy maximization is a generalization of the deterministic entropy maximization, and (2) the deterministic entropy maximization can be solved by MLE. This generalization is useful for understanding the relationship between our model and existing ICA models based on the deterministic entropy maximization, as described later. Since the only assumption we have used is that the basis function φ(y; θ ) is an invertible function, our result about the above equivalence between entropy maximization and MLE is more general than that obtained by Cardoso (1997), who showed the equivalence when ψ(x) is a deterministic function such that ψ(x) = f (Wx) where W is a square and regular matrix and f (x) is a component-wise monotonically increasing nonlinear function. 3 Model and Algorithm 3.1 Probabilistic Model. Here, we describe the details of the probabilistic model we use. Suppose x is an n-dimensional input signal and s is an m-dimensional hidden
variable that obeys a component-wise independent distribution p(s) = m i=1 pi (si ). We consider a square or undercomplete situation, that is, n is not smaller than m (n ≥ m). Our probabilistic model is then given more specifically by  p(x|A, θ) = p(s)p(x|s, A, θ )ds (3.1) p(x|s, A, θ) =

p(s) =

1 Z(A, θ)   T   × exp −β g(x; θ) − As (x; θ ) g(x; θ ) − As (3.2) m  1 i=1

2

(1 − tanh2 (si )),

(3.3)

where A is an n×m matrix, g(x; θ) is a component-wise monotonic function,  and Z(A, θ) is a normalization term such that the integral p(x|s, θ, A)dx is 1. The stochastic generative process p(x|s, θ, A) is based on a postnonlinear ICA model, namely, x = f (As) + n, where f (·) is the inverse function of g(x; θ ). n is a gaussian noise whose mean and covariance are 0 and a diagonal matrix (2β)−1 I, respectively, where I denotes the identity matrix. When (x − f (As)) is close to zero, it is approximated as x − f (As) ≈ f  (g(x; θ ); θ )(g(x; θ ) − As), where f  (x; θ) is a diagonal matrix whose (i, i) ∂ f (xi ;θi ) element is [ f  (x; θ )]ii = ∂xi . Using this approximation, g(x; θ ) obeys a gaussian distribution whose mean and covariance are 0 and a diagonal matrix (x; θ ), respectively. The (i, i) element of (x; θ ) is [ (x; θ )]ii =

Nonlinear and Noisy Extension of Independent Component Analysis

125

{ f  (g(xi ; θ ); θ )}2 . Note that our model cannot be described by a noisy linear ICA after the application of a nonlinear transformation, y = Aˆs + n and y = f (x; θ ), because the covariance of the noise n is dependent on x in our formulation. In this study, we use g(xi ; θ ) = log(xi ) + ci , with which f (yi ; θ ) = exp(yi − ci ) and [ (x; θ)]ii = { f  (g(xi ; θ ); θ )}2 = x2i . Parameters of our probabilistic model are A and θ ≡ {ci |i = 1, . . . , n}. β is a hyperparameter. The prior defined by equation 3.3 corresponds to the application of a nonlinear transformation tanh(y) when the model is reduced to a deterministic entropy maximization model. This prior is a gaussian-like unimodal distribution, but more kurtotic than the gaussian distribution. Our model is identical to a cepstrum analysis in a special parameter setting. The cepstrum analysis first transforms a given sound into an amplitude spectrum by applying a Fourier transformation. The amplitude spectrum is then transformed by a logarithm function, and again another Fourier transformation is applied. The first step is also done as a preprocess in our model. The latter step is also realized in our model if the component-wise nonlinear function is given as g(x; θ) = log(x), the mixing matrix A is fixed at a square Fourier transformation matrix, and the hyperparameter β approaches infinity. Our model becomes a deterministic transformation in the limit of β going to infinity. In this case, our model becomes equivalent to the cepstrum analysis in the above-mentioned setting. 3.2 Generalized EM Algorithm. Because there is a hidden variable s, an EM algorithm (Dempster et al., 1977) is used for the maximization of the log likelihood 3.1 with respect to parameters A and θ . ¯ we obtain • E-step: By using the current parameter estimators, A¯ and θ, the expected log likelihood,

¯ θ) ¯ = Q(A, θ |A,

T  1 ¯ θ¯ ) log p(xt , st |A, θ )dst , p(st |xt , A, T t=1

(3.4)

where {xt } is the set of T input samples and {st } is the set of corresponding hidden variables. ¯ θ) ¯ with respect to parameters A and θ . • M-step: Maximize Q(A, θ|A, 3.2.1 E-Step. Since the parameter-dependent terms in log p(x, s|A, θ ) = −β(g(x; θ ) − As)T (x; θ)(g(x; θ) − As) − log Z(A, θ ) − log p(s) include only the first- and second-order statistics of the hidden variable s, the integral in equation 3.4 is calculated by the Laplace approximation. ¯ θ¯ ) appearing in equaAccording to the Laplace approximation, p(s|x, A, tion 3.4 is approximated as a gaussian distribution peaked at the MAP esti-

126

S. Maeda, W.-J. Song, and S. Ishii

¯ θ) ¯ θ¯ ) + log p(s)}: ¯ = arg maxs {log p(x|s, A, mator, sˆ ≡ arg max p(s|x, A,

 ¯ θ¯ ) ∝ exp − 1 (s − sˆ )T H(ˆs)(s − sˆ ) , p(s|x, A, 2

(3.5)

¯ θ). ¯ where H(ˆs) ≡ −∇∇ log p(ˆs|x, A, To obtain the MAP estimator sˆ , a gradient-ascent method is used: ¯ + ϕ(s), ¯ ¯ − As) s ∝ 2β A¯ T (g(x; θ)

(3.6)

∂ log p(s)

where ϕ(s) ≡ = −2 tanh(s). ∂s Then the first and second moments are calculated as  ¯ θ)sds ¯ s ≡ p(s|x, A, = sˆ  sT s ≡

¯ θ)ss ¯ T ds = sˆ sˆ T + H(ˆs)−1 , p(s|x, A,

(3.7) (3.8)

where H(ˆs)−1 is estimated under the assumption of low noise such that β is fairly large:  −1 1 ≈ (AT A)−1 H(ˆs)−1 = 2βAT A − ∇∇ log p(ˆs) 2β 1 + 2 (AT A)−1 ∇∇ log p(ˆs)(AT A)−1 . 4β

(3.9)

3.2.2 M-Step. Using the statistics, equations 3.7 and 3.8, and neglecting the normalization term (− log Z(A, θ)) under the assumption of low noise, ¯ θ¯ ) are calculated as the derivatives of the expected log likelihood Q(A, θ|A, T   ∂Q 2β ≈ t et sˆ Tt − t AH(ˆst )−1 ∂A T t=1

(3.10)

T 2β ∂Q ≈− et,i ( f  (g(xt,i ; θi ); θi ))2 , ∂ci T t=1

(3.11)

where et,i and xt,i denote the ith elements of et ≡ (g(xt ; θ ) − Aˆst ) and xt , respectively. The parameter c is analytically obtained from equation 3.11 as   m 2 log x − ˆ s x a i,t ij j,t t=1 i,t j=1 , T 2 x t=1 i,t

T ci = −

(3.12)

Nonlinear and Noisy Extension of Independent Component Analysis

127

while some constraint is needed to obtain analytically the matrix A. When the function g(xt ; θ) is linear in equation 3.10, the M-step equation ∂Q/∂A = 0 is analytically solved as T 1 A= g(xt ; θ)ˆsTt T t=1



T   1 sˆ t sˆ Tt + H(ˆst )−1 T t=1

−1 ,

(3.13)

which provides the noisy ICA presented previously (Bermond & Cardoso, 1999; Girolami, 2001). In a nonlinear case, however, the analytical solution is not directly obtained, and a gradient-based optimization method is used in our algorithm. Since the M-step is based on a gradient-ascent method in this case, our EM procedure is an instance of the generalized EM algorithm.

Using the natural gradient AAT ∂Q ∂A instead of the gradient ∂Q/∂A, the learning is much accelerated in practice (Amari et al., 1996; Lewicki & Sejnowski, 2000): AAT

T    2β ∂Q ≈ AAT t g(xt ; θ) − Aˆst sˆ Tt − AAT t AH(ˆst )−1 . ∂A T t=1

Since the MAP estimator sˆ should satisfy s = 0 or, equivalently, 2βAT (g(x; θ ) − As) + ϕ(s) = 0, the following equation holds: AAT

T   1 ∂Q A φ(ˆst )ˆsTt + 2βAT t AH(ˆst )−1 . ≈− ∂A T t=1

(3.14)

When the noise is low, that is, β is large enough, H(ˆs)−1 is approximated as 1 H(ˆs)−1 ≈ 2β (AT A)−1 + 4β1 2 (AT A)−1 ∇∇ log p(ˆs)(AT A)−1 , and this leads to AAT

∂Q ∂W



T ∂Q 1 1 A I + ϕ(ˆst )ˆsTt + ≈− ∇∇ log p(ˆst )(AT t A)−1 . ∂A T t=1 2β

(3.15)

In particular, when A has an inverse matrix W ≡ A−1 , noting the fact that T = −AT ∂Q ∂A A , the natural gradient 3.15 becomes ∂Q T ∂Q T T W W = −AT A W W ∂W ∂A

 T 1 1 −1 T T I + φ(ˆst )ˆst + ∇∇ log p(ˆst )W t W W, = T t=1 2β

(3.16)

which is closely related to the noisy ICA which uses the bias removal technique, presented previously by Douglas et al. (1998). In their noisy ICA,

128

S. Maeda, W.-J. Song, and S. Ishii

however, the sign of the third right-hand-side term in equation 3.16 was opposite, and sˆ was given as sˆ = Wx instead of the MAP estimator we use. Note that local convergence is assumed in our generalized EM algorithm, while the convergence of the above bias removal technique remains 1 unclear. If the term ( 2β ∇∇ log p(ˆst )W t−1 W T ) is ignored in equation 3.16, in addition, it becomes the learning rule of noise-free ICA, which has been proposed from various perspectives as described in section 1. Although the noise-free ICA is a special case of the noisy ICA, existing noisy ICA learning rules based on MLE (e.g., Belouchrani & Cardoso, 1995; Hyv¨arinen, 1998a; Bermond & Cardoso, 1999; Højen-Sørensen et al., 2002; Girolami, 2001) do not include the noise-free ICA learning rule, except for the one by Lewicki and Sejnowski (2000). Lewicki and Sejnowski derived a noisefree ICA learning rule by omitting a term that is unrelated to the likelihood ∫ p(s)p(x|s, θ)ds, but our learning rule does not include such a term. Instead, 1 the term ( 2β ∇∇ log p(ˆst )W t−1 W T ), which appears in our learning rule, automatically vanishes in the noiseless limit. They reported that the additional term makes the learning unstable, whereas ours does not. This difference arises probably because Lewicki and Sejnowski directly evaluated the like lihood function p(s)p(x|s)ds by approximating p(s)p(x|s) as a gaussian distribution centered at the MAP estimator sˆ , which is dependent on the estimation of A, whereas our EM formulation introduced the expected log ¯ θ) ¯ in which the MAP estimator sˆ is independent of the likelihood Q(A, θ|A, estimation of parameters A and θ. Although their approximation incorporated how the MAP estimator sˆ is affected by changing the parameter A, that effect was difficult to estimate and hence needed approximation, which might have caused the instability. Many of the conventional noisy ICA algorithms based on MLE calculate the integral over the hidden variable s by using point approximation with a Dirac’s delta function (Belouchrani & Cardoso, 1995; Olshausen & Field, 1996; Hyv¨arinen, 1998a; Rao & Ballard, 1999). For example, if the term t AH(ˆst )−1 is removed from equation 3.10, which corresponds to the case that the posterior is approximated by a delta function at the MAP estimator sˆ , then the updating rule becomes equivalent to the original sparse coding method (Olshausen & Field, 1996): A ∝

T 2β t (g(xt ; θ) − Aˆst )ˆsTt . T t=1

(3.17)

It should be noted that if g(xt ; θ) is linear, t becomes the identity matrix in equation 3.17. In the noiseless limit, equation 3.10 is reduced to noise-free ICA while equation 3.17 is not, and this difference stems from the approximation accuracy of the posterior. Accordingly, our new learning rule not only improves existing noisy ICA algorithms but also gives an appropriate nonlinear extension of those algorithms.

Nonlinear and Noisy Extension of Independent Component Analysis

129

Our learning rule also generalizes a noise-free nonlinear ICA derived by Lee et al. (1997) or Almeida (2004), because those algorithms are based on deterministic entropy maximization which is equivalent to MLE in the noiseless limit as mentioned in section 2. 3.3 Step Size Determination. According to our GEM algorithm, parameter A is obtained by a gradient-based optimization method, A := A + A. Here, we discuss the way to determine an appropriate value for the step size . In the GEM algorithm, the step size should be determined so that ¯ θ) ¯ is maximized. The optimization for the expected log likelihood Q(A, θ|A, ¯ θ¯ ) ≥ Q(A, ¯ θ |A, ¯ θ¯ ). We parameter A in the M-step should achieve Q(A, θ |A, then define a cost function, ¯ θ) ¯ θ|A, ¯ θ), ¯ − Q(A, ¯ F() ≡ Q(A, θ |A,

(3.18)

where A = A¯ + A.  is determined such that F() is maximized. We can see F() has a simple quadratic form if either the noise is fairly low or the nonlinear function g(xt ; θ) is linear,

 b 2 b2 F() ≈ −a  − (3.19) + , 2a 4a  T T where a = Tβ Tt=1 (Tr(st sTt AT t A)) and b = 2β t=1 (g(xt ; θ ) t A T T T ¯ Tr(·) denotes the matrix trace. The derivation of st  − Tr(st st A t A)). equation 3.19 is described in section A.1. Since a and b are positive (see section A.2), the optimal step size is determined as =

b , 2a

(3.20)

with which the GEM algorithm achieves a line search for parameter A. ¯ θ¯ ) decreases as the The gain of the expected log likelihood Q(A, θ |A, noise decreases, and the optimal step size equals zero in the noiseless limit because the parameter a diverges while the parameter b converges in that limit. This indicates that the EM algorithm is no longer effective in the noiseless limit. Interestingly, although the update of A according to the GEM algorithm does not increase the expected log likelihood in this case, the update is still meaningful because it increases the likelihood itself. This implies that when the noise is small enough, the step size , which is optimal for maximizing the likelihood, is larger than that optimal for maximizing the expected likelihood. 4 Virtual Pitch Simulation 4.1 Condition. Human speech and music were used for training our model. Speech was uttered by four American, four Spanish, and two Japanese speakers; five of them were male and the other five female. The total

130

S. Maeda, W.-J. Song, and S. Ishii

duration of speech was 232 seconds. Music was classified into a category of singing voice and no singing voice. The former consisted of pop music and relaxation music, and was extracted from 25 songs so that the total duration was 116 seconds. The latter consisted of a piano concerto, a violin concerto, and an orchestra and was extracted from 24 tunes so that the total duration was 116 seconds. All the sound data were obtained from commercial compact discs randomly. No period of silence was included. The data set consisted of 10,000 segments, each of which contained 512 points. They were quantized into 16 bits and converted to the sampling frequency of 11,025 Hz to cover the existence region of virtual pitch (Ritsma, 1962). To obtain spectral information, 512-point discrete Fourier transformation (DFT) of the entire acoustic data was employed, which transformed the acoustic signals into 10,000 points of 256-dimensional spectral patterns. Before performing DFT, we applied a Hanning window to the data set for removing wideband artifactual energy. We used Fourier bases for mimicking peripheral spectral analysis for the following reason, though bandpass filters whose center frequencies are arranged in a logarithmic scale are more natural than the Fourier filters; to distinguish high-frequency components with a resolution sufficient for reproducing virtual pitch phenomena, the number of filters required by the logarithmic arrangement becomes much larger than the number of Fourier filters by the linear arrangement. We adopted the definition of pitch matching, used in the existing study (Wightman, 1973). We assume that the peak value of on MAP estimator sˆ represents the internal pitch representation: we suppose two inputs are perceived as having the identical pitch if the same component of sˆ is most activated in response to both of the two inputs. We set the size of the mixing matrix A at 256×30. The learning results were dependent on the hyperparameter β and the size and the initial value of the matrix A. Since the hyperparameter β balances the prior p(s), which makes the internal representation s small with the conditional probability p(x|s), which makes the representation error (g(x; θ) − As)T (x; θ )(g(x; θ ) − As) small, β should be set appropriately. In the simulation, β was set to 500, the initial value of A was set closely to the one used in Wightman’s model (1973), and the parameter c was constrained so that the mean E[g(x; θ )] was zero. 4.2 Results. Figure 1 shows the mixing matrix A after learning, in which the magnitude of all elements of matrix A is depicted. The jth column vector of matrix A, aj , can be interpreted as an n-dimensional spectral basis pattern due to the jth internal representation sˆj , and an input x is approximated   m as a linear summation of the column vector aj as x ≈ exp j=1 aj sˆj + c . Then we call each aj a basis vector and regard sˆ as an internal representation of input x. Figure 2 shows six basis vectors out of matrix A after learning, which correspond to the column vectors surrounded by rectangles in Figure 1. Many harmonic peaks can be observed in those basis vectors. A circle

Nonlinear and Noisy Extension of Independent Component Analysis

131

Figure 1: The mixing matrix A after learning, in which the magnitude of all elements of matrix A is shown. The jth column vector of matrix A, aj , can be interpreted as an n-dimensional spectral pattern due to the jth internal representation sˆj , and an input x is approximated
as the exponential of the linear  summation of those column vectors, x ≈ exp

m

j=1

aj sˆj + c . Note that both of

the components of a column vector aj and an internal representation sˆj can be negative, because we used an exponential function as a nonlinear function in this simulation. The column vectors surrounded by rectangles correspond to the basis vectors shown in Figure 2.

indicates a harmonic peak, and the title denotes the fundamental frequency calculated from the circled harmonic peaks such to minimize the mean square error between multiples of the fundamental frequency and the circled harmonic frequencies. A harmonic structure is clear especially for low fundamental frequencies (see Figure 2A), while sometimes not very clear for high fundamental frequencies (see Figure 2D). For high fundamental frequencies, the basis vectors sometimes contain large values (see Figure 2C). Such basis collapses for high fundamental frequencies occur presumably because the training data are not well represented by harmonic bases whose fundamental frequencies are high. In other words, the harmonic structure is supposed to arise especially for low fundamental frequencies because the training data are well represented by the acquired harmonic bases. A few basis vectors (see Figure 2F) exhibit no prominent harmonic structure.

132

S. Maeda, W.-J. Song, and S. Ishii $

+]

 

ZHLJKWVRIDEDVLV

í

'



%

+]







í



í

í 



+]



(



+] 

í 



)

















+]

&







í

í 



IUHTXHQF\+]

Figure 2: Six column vectors out of matrix A after learning, each of which represents a basis vector for an internal representation sj . Each panel corresponds to a rectangle in Figure 1. A circle indicates a harmonic peak, and the title denotes a fundamental frequency calculated from the circled harmonic peaks. Most of the column vectors exhibit harmonic structures. We therefore suppose that each basis vector represents a spectral pattern of a certain pitch, which tends to have a harmonic structure. Especially for high fundamental frequencies, basis vectors sometimes contain components with large values relative to the other components (e.g., C), and a few basis vectors do not exhibit prominent harmonic structures (e.g., F).

Ritsma (1962) examined whether two stimuli—one consisting of a fundamental frequency g, its twice frequency 2g and three-times frequency 3g, and the other consisting of three higher-order harmonics of the fundamental frequency, ( f − g), f and ( f + g)—provide the same pitch sensation. More precisely, the higher-order harmonics of the fundamental frequency tone v(t) in that experiment had a modulation parameter m that controlled the mixture ratio: v(t) = m sin 2π( f − g)t + sin 2π f t + m sin 2π( f + g)t. We call the former and the latter tones a fundamental tone (FT) and a higher harmonics tone (HHT), respectively (the spectrum of each tone is shown in Figure 3), and call frequencies g and f , a spacing frequency and a center frequency, respectively. Figure 4 shows the region in which an FT and an HHT provide the identical pitch (Ritsma, 1962). A pair of an FT and an HHT is specified by parameters g and f in Figure 4. This figure shows several aspects of virtual pitch phenomena. When the center frequency f is too large, virtual pitch does not occur. When the spacing frequency g is too small or too large, virtual pitch does not occur either. Therefore, virtual pitch can be observed for moderate values of f and g. r is a reference variable denoting the rate f/g. Note that there are no common frequency components between the FT

Nonlinear and Noisy Extension of Independent Component Analysis

DPSOLWXGHVSHFWUXP

P 

VSDFLQJIUHTXHQF\

J J J

+LJKHUKDUPRQLFV

% DPSOLWXGHVSHFWUXP

IXQGDPHQWDOWRQH

$

IJ I IJ

VSDFLQJIUHTXHQF\

J J J

IUHTXHQF\

DPSOLWXGHVSHFWUXP

' DPSOLWXGHVSHFWUXP

P 

FHQWHUIUHTXHQF\

IUHTXHQF\

IUHTXHQF\

&

133

FHQWHUIUHTXHQF\

IJ I IJ

IUHTXHQF\

Figure 3: Spectra of a fundamental tone (FT) and a higher harmonics tone (HHT) for m = 100 and m = 50. An FT consists of a fundamental frequency g, its twice frequency 2g, and three-times frequency 3g, and an HHT consists of three higherorder harmonics of the fundamental frequency, ( f −g), f and ( f +g). Frequencies g and f are called a spacing frequency and a center frequency, respectively. A virtual pitch is judged to occur if these two stimuli, left and right, provide the same pitch sensation.

and the HHT in the region of r = f/g ≥ 5. The modulation parameter m represents the clarity of the HHT (see above). As the m value becomes large, the harmonic structure becomes prominent, and then the region of virtual pitch enlarges. Figure 5 shows activations of internal representation sˆ in our model when presented with FTs and HHTs. We used m = 100% for all simulations in this section. In each of Figures 5A, 5B, and 5C, the upper and lower rows show the amplitude spectrum of inputs and the activation pattern against the inputs, respectively. In each lower panel is a mark that indicates whether the most activated internal representation is the same between when presented with an FT and when presented with an HHT; a circle indicates a match, a cross a mismatch, and a triangle the case that the internal representation most activated by an FT was the second most activated one by an HHT. When FTs and HHTs were moderate (corresponding to from 160 to 660 Hz), the most activated internal representation was almost the same between when presented with an FT and when presented with an HHT, as can be

134

S. Maeda, W.-J. Song, and S. Ishii

Figure 4: The existence region of virtual pitch examined by Ritsma (1962), in which whether an FT and an HHT provide the same pitch sensation was psychologically examined. The two stimuli are specified by a spacing frequency g and a center frequency f . This figure shows several aspects of virtual pitch. When either the center frequency f is too large or the spacing frequency g is too small or too large, virtual pitch does not occur. Therefore, virtual pitch can be observed for moderate values of f and g. r is a reference variable denoting the rate f/g. m is a modulation parameter that represents the clarity of an HHT (see the text for the exact definition). As the m value becomes large, the harmonic structure becomes prominent and then the region of virtual pitch enlarges.

seen in Figures 5A and 5B.3 This indicates that such an FT-HHT pair led to the identical pitch sensation. In contrast, the most activated component of internal representation showed discrepancies for lower or higher fundamental frequencies, as can be seen in Figure 5C; namely, such an FT-HHT pair did not lead to the identical pitch sensation in our simulation. To examine what kind of feature of the input sound data was extracted by internal representation s, we reconstructed input sounds from the internal representation. An example is shown in Figure 6. The original input (upper panels) and the corresponding internal representation are the same as those in Figure 5B, though the y-axis has log-scale for visibility. From this figure, we see that the reconstructed spectrum had a clear harmonic structure, but the reconstruction was degraded as the center frequency was high. This result

3 FTs and HHTs are dependent on the resolution of DFT. Although individual activation of internal representation s was affected by the resolution of DFT, the overall characteristics of activation patterns did not vary regardless of the resolution.

Nonlinear and Noisy Extension of Independent Component Analysis IXQGDPHQWDOWRQH 

    í 

 



   í 

I +] 

 



   í 

I +]

% J +]



 

 

DFWLYLW\

DPSOLWXGH VSHFWUXP











  



IUHTXHQF\+]    í



   í 







  í 



   í 



   í 

I +] 



I +] 

 

 



I +] 

 

   

I +] 

  

  



I +]

I +] 

 

   

I +] 

  

  

  

& J +]

KLJKHUKDUPRQLFV I +]

$ J +]

135



 

   





I +] 

 





   í 

 



LQGH[RIKLGGHQYDULDEOHV

Figure 5: Activations of internal representation s when presented with FTs (the left-most column) or HHTs (the other columns) (m = 100%). (A–C) The upper and lower rows show the amplitude spectrum of input tones and the activation patterns against the inputs, respectively. In each lower panel is a mark that indicates whether the most activated internal representation is the same between when presented with an FT and when presented with an HHT. A circle indicates a match, a cross a mismatch, and a triangle the case that the internal representation most activated by an FT is the second most activated one by an HHT. A light gray bar indicates the component most activated by the FT. When FTs and HHTs are moderate (corresponding to from 160 to 660 Hz), the most activated internal representation was almost the same between when presented with an FT and when presented with the corresponding HHT, as can be seen in A and B; namely, such an FT-HHT pair produces the identical pitch sensation in our simulation. In contrast, the most activated component of internal representation is different for lower or higher fundamental frequencies, as can be seen in C; namely, such an FT-HHT pair does not produce the identical pitch sensation in our simulation.

indicates that the sound data used for the learning involve many harmonic structures that are prominent especially for low frequencies. The learning by our nonlinear noisy ICA then made the reconstruction error small, especially for low-frequency components. Although we show a single example here, the above tendency can be seen in other cases. A more intensive result is summarized in Figure 8 (for details, see the figure caption). Figure 7 shows the existence region of virtual pitch in our model—the region where the same component of sˆ was most activated in response

136

S. Maeda, W.-J. Song, and S. Ishii

IXQGDPHQWDOWRQH

DPSOLWXGH VSHFWUXP

J +]

KLJKHUKDUPRQLFV I +]

I +]

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I +]

   

 

 

 

 

 

 

 

 

 

 

 

 











 



 



 



I +]

 

 

 

 

IUHTXHQF\+]

Figure 6: Comparison between an input sound spectrum and a spectrum reconstructed from internal representation sˆ . The original input (upper panels) and the corresponding internal representation are the same as those in Figure 5B, though the y-axis has a log scale for visibility. Each reconstructed spectrum has a clear harmonic structure, but the reconstruction is degraded as the center frequency is high.

to both an FT and the corresponding HHT. We show only the region of f/g ≥ 5. The circles and triangles have the same meaning as in Figure 5. A shaded dot corresponds to a cross in Figure 5. When the center frequency f was fairly large, we did not perform simulations because the frequency resolution sufficient for producing test sounds could not be realized in the current simulation condition. When the spacing frequency g was too small or too large, the number of shaded dots increased, suggesting that virtual pitch hardly occurred in that region. This tendency can also be seen in the psychophysical experiment in Figure 4. The reason that virtual pitch was observed in the moderate region can be considered as follows. When the spacing frequency was too small, virtual pitch did not occur, possibly because the frequency resolution was not sufficient; although this insufficient resolution is the case of our simulation, similar features may exist in the actual cochlear filter. When the spacing frequency was too large, virtual pitch did not occur because the harmonic structure of the basis vectors collapsed for high fundamental frequencies. We also compared the pitch sensation by our simulation with that by Wightman’s central pattern transformation model. The result is shown in Figure 9. The overall characteristics of the Wightman’s model are similar to those by our simulation, but there are some discrepancies. The main difference is that Wightman’s model produced virtual pitch even for high spacing frequencies (g > 800 Hz), whereas our simulation did not produce in that region. Because the harmonic structures of basis vectors are clearly prepared even for high spacing frequencies in Wightman’s model, the model produced virtual pitch. Note that virtual pitch does not actually occur when the spacing frequency is high (see Figure 4).

Nonlinear and Noisy Extension of Independent Component Analysis

U 





137



VSDFLQJIUHTXHQF\J+]

  

&



%



$

    

















FHQWHUIUHTXHQF\I+] Figure 7: Existence region of virtual pitch in our model ( f/g ≥ 5). This figure corresponds to the situation with m = 100% in Figure 4. A circle indicates a parameter pair of g and f , with which an FT and an HHT produced the same pitch, and a triangle indicates that the most activated internal representation by an FT corresponded to the next most activated internal representation by an HHT. A dot indicates a parameter pair with which the pitch sensation showed discrepancy (examples of activation pattern sˆ are shown in Figure 5, and those examples correspond to the simulation conditions surrounded by light square rectangles). When the spacing frequency g is too small or too large, the number of shaded dots increases, suggesting that virtual pitch hardly occurs in that region. This tendency can also be seen in the psychophysical experiment in Figure 4.

5 Discussion In this letter, we proposed a noisy nonlinear ICA that not only generalizes the formerly proposed noisy linear ICA, but also corresponds to the noisefree linear ICA in the noiseless limit. Exploring the estimation function that is free from the assumption of source p.d.f. in the noisy nonlinear ICA is our future work. We trained our model by using speech and music and examined the pitch sensation by the trained model. We found that the existence region of virtual pitch, obtained by our computer simulation, is qualitatively similar to

138

S. Maeda, W.-J. Song, and S. Ishii

U 



VSDFLQJIUHTXHQF\J+]

  





 㨪   㨪 㨪 㨪

      

















FHQWHUIUHTXHQF\I+] Figure 8: Reconstruction error between an input sound spectrum and a sound spectrum reconstructed from internal representation sˆ for various combinations of f and g. The reconstruction error denotes absolute difference between the amplitude spectrum of input sounds and the reconstructed amplitude spectrum, averaged over frequencies. A dark (light) circle indicates that the reconstruction error is large (small). While most of the error takes a relatively small value (error < 10), sometimes it becomes a very large value. Therefore, the coloring is performed in a nonlinear fashion for visibility (see inset).

that by the existing psychological experiment. Chang and Merzenich (2003) reported that continuous exposure to moderate-level noise disturbs the normal development of topographic representational order of an infant rat’s primary auditory cortex and the primary auditory cortex still has a plasticity after the exposure to the noise; namely, the topographic representation can be reorganized in a new sound environment. Their study is consistent with our hypothesis that the transformation in the central auditory system can be acquired and changed through learning, depending on the environment. In particular, the primary auditory cortex seems to be a responsible region for such learning, for several reasons; neurons in the primary auditory cortex have frequency selectivity (Merzenich, Knight, & Roth, 1975; Reale & Imig, 1980), and the primary auditory cortex is known to have neurons activated when presented with a pure tone and presented with a harmonic

Nonlinear and Noisy Extension of Independent Component Analysis

U 





139



VSDFLQJIUHTXHQF\J+]

          















FHQWHUIUHTXHQF\I+] Figure 9: Existence region of virtual pitch in Wightman’s model ( f/g ≥ 5). This figure is plotted similar to that in Figure 7. This figure also corresponds to the situation with m = 100%. When the spacing frequency g was larger than 260 Hz, virtual pitch was observed; it was not observed when the spacing frequency g was small. This tendency of Wightman’s model can be seen in our simulation, but there are some discrepancies. The major difference is that Wightman’s model produced virtual pitch even for high spacing frequencies (g > 800 Hz), whereas our simulation did not produce in that region. We consider that this arose because the basis vectors in the Wightman’s model were prepared to have clear harmonic structures even for high spacing frequencies. However, such a feature cannot be observed in the psychoacoustic experiment (see Figure 4).

combination tone that induces a virtual pitch phenomenon (Pantev, Hoke, Lutkenhoner, & Lehnertz, 1989; Riquimaroux & Hashikawa, 1994). Since our model involves a cepstrum analysis as a special case, from an engineering viewpoint, it may be possible to obtain a pitch sensation model or a feature extraction system that can be applied to, for example, detection of the fundamental frequency, through learning from actual speech and music signals, based on our framework of nonlinear noisy ICA.

140

S. Maeda, W.-J. Song, and S. Ishii

Appendix ¯ θ¯ ) A.1 Derivation of Equation 3.19. Expected log likelihood Q(A, θ |A, is given by ¯ θ¯ ) = − 1 Q(A, θ|A, T

= −β

T  T ¯ θ)(g(x ¯ β p(st |xt , A, t ; θ ) − Ast ) t=1

 × t (g(xt ; θ) − Ast )dst + log Z(A, θ )

T  1 g(xt ; θ)T t g(xt ; θ) − 2g(xt ; θ)T t Ast  T t=1  + Tr(st sTt AT t A) + log Z(A, θ ).

When the noise is low or the nonlinear function g(xt ; θ ) is linear, terms that are not multiplied by β can be neglected. Then, ¯ θ) ¯ θ|A, ¯ θ) ¯ − Q(A, ¯ F() ≡ Q(A, θ|A, T   1 −2g(xt ; θ)T t A st  + Tr(st sTt AT t A) ≈ −β T t=1 T   1 ¯ . −2g(xt ; θ)T t A¯ st  + Tr(st sTt A¯ T t A) +β T t=1 Substituting A for A¯ + A, it becomes = 2β

T   1 g(xt ; θ)T t (A¯ + A) st  − g(xt ; θ )T t A¯ st  T t=1

T   1 ¯ −Tr(st sTt (A¯ + A)T t (A¯ + A))+Tr(st sTt A¯ T t A) T t=1 T   β Tr(st sTt AT t A) = − 2 T t=1

+β

+

T   2β ¯ . g(xt ; θ)T t A st  − Tr(st sTt AT t A) T t=1

This is identical to equation 3.19. A.2 Sign of Parameters a and b. The parameter b is positive because the gradient descent always satisfies F ( = 0) > 0, which yields b > 0. Determining whether the parameter a is positive is a bit complicated.

Nonlinear and Noisy Extension of Independent Component Analysis

141

The term Tr(st sTt AT t A) is a sum of all elements of the matrix that is a Hadamard product of two matrices st sTt  and AT t A. A Hadamard product of two matrices P and Q is represented as P  Q, and the (i, j) element of P  Q is denoted as pij qij , where pij and qij are the (i, j) elements of matrix P and Q, respectively. To determine the positivity of the second term, the following lemmas are used. Lemma 1. When a matrix U is (semi-) positive definite, the sum of all elements of U is positive. Lemma 2. When two n × n matrices P and Q are (semi-) positive definite, the Hadamard product of them, U = P  Q, is also (semi-) positive definite. Lemma 3.

The matrices st sTt  and AT t A are (semi-) positive definite.

When a matrix U is (semi-) positive definite, the sum of all elements of U, 1T U1, is (semi-) positive. This fact is identical to lemma 1. Next, we prove lemma 2. When n×n matrices P and Q are (semi-) positive definite, P and Q are represented as P = LpT Lp and Q = LTq Lq , where Lp and Lq are lower triangular matrices of P and Q, respectively. Let aij or bij be the (i, j) element of the matrix Lp or Lq , respectively. Then the (i, j) element of U = P  Q, Uij , is calculated as Uij = [(LpT Lp )  (LTq Lq )]ij n n = aki akj bli blj k=1 l=1

=

n2

(αm,i )(αm,j ),

m=1

where αm,i = aq(m)i br(m)i , q(m) = m/n, and r(m) = m − q(m)n. Let M be an n2 × n matrix whose (i, j) element is αi,j . Then the above relation can be described in a matrix form: U = MT M. This proves that the matrix U is (semi-) positive definite. Next, we prove lemma 3. The matrix st sTt  is obviously (semi-) positive definite. AT t A is also positive definite because t is positive definite, and for any nonzero vector s, sT AT t As = (As)T t (As) > 0 is satisfied. Lemmas 1, 2, and 3 indicate that the term Tr(H(ˆst )−T AT t A) is (semi-) positive.

142

S. Maeda, W.-J. Song, and S. Ishii

References Almeida, L. B. (2004). MISEP—linear and nonlinear ICA based on mutual information. Signal Processing, 84(2), 231–245. Amari, S., & Cardoso, J. F. (1997). Blind source separation—semiparametric statistical approach. IEEE Transaction on Signal Processing, 45(11), 2692– 2700. Amari, S., Chen, T.-P., & Cichocki, A. (1997). Stability analysis of learning algorithms for blind source separation. Neural Networks, 10(8), 1345–1351. Amari, S., Cichocki, A., & Yang, H. H. (1996). A new learning algorithm for blind signal separation. In D. Touretzky, M. Mozer, & M. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 752–763). Cambridge, MA: MIT Press. Atick, J. J. (1992). Could information theory provide an ecological theory of sensory processing? Network, 3, 213–251. Attias, H. (1999). Independent factor analysis. Neural Computation, 11(4), 803– 851. Barlow, H. B. (1961). Sensory communication. Cambridge, MA: MIT Press. Bell, A. J., & Sejnowski, T. J. (1995). An information maximization approach to blind separation and blind deconvolution. Neural Computation, 7(6), 1129– 1159. Bell, A. J., & Sejnowski, T. J. (1997). The “independent components” of natural scenes are edge filters. Vision Research, 37(23), 3327–3338. Belouchrani, A., & Cardoso, J. F. (1995). Maximum likelihood source separation by the expectation-maximization technique: Deterministic and stochastic implementation. In Proceedings of NOLTA (pp. 49–53). Las Vegas, NV. Bermond, O., & Cardoso, J. F. (1999). Approximate likelihood for noisy mixtures. In ICA ‘99 (pp. 325–330). Aussois, France. Bregman, A. S., & Campbell, J. (1971). Primary auditory stream segregation and perception of order in rapid sequences of tones. Journal of Experimental Psychology, 89(2), 244–249. Cardoso, J. F. (1997). Infomax and maximum likelihood for source separation. IEEE Trans. Auto. Contr. Sys., 4(4), 112–114. Cardoso, J. F., & Souloumiac, A. (1993). Blind beamforming for non gaussian signals. In IEEE Proceedings-F, 140(6), 362–370. Chang, E. F., & Merzenich, M. M. (2003). Environmental noise retards auditory cortical development. Science, 300, 498–502. de Boer, E. (1956). Pitch of inharmonic signals. Nature, 178, 535–536. de Boer, E. (1977). Psychophysics and physiology of hearing. San Diego, CA: Academic Press. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of Royal Statistical Society. Series B, Statistical Methodological, 39, 1–38. Douglas, S. C., Cichocki, A., & Amari, S. (1998). Bias removal technique for blind source separation with noisy measurements. Electronics Letters, 34(14), 1379–1380.

Nonlinear and Noisy Extension of Independent Component Analysis

143

Gaeta, M. & Lacoume, J. L. (1990). Source separation without a priori knowledge: the maximum likelihood solution. In EUSIPCO (Vol. 24, pp. 621–624). Berlin: Springer-Verlag. Girolami, M. (2001). A variational method for learning sparse and overcomplete representations. Neural Computation, 13(11), 2517–2532. Goldstein, J. L. (1973). An optimum processor theory for the central formation of the pitch of complex tones. J. Acoust. Soc. Am., 54, 1496–1516. Helmholtz, H. L. F. (1863). Die Lehre von den Tonempfindungen als Physiologische ¨ die Theorie der Musik. Braunschweig. Grundlage fur Højen-Sørensen, P. A., Winther, O., & Hansen, L. K. (2002). Mean-field approaches to independent component analysis. Neural Computation, 14(4), 889–918. Houtsma, A. J. M., & Goldstein, J. L. (1972). The central origin of the pitch of complex tones: Evidence from musical interval recognition. J. Acoust. Soc. Am., 51, 520–529. Hyv¨arinen, A. (1998a). Independent component analysis in the presence of gaussian noise by maximizing joint likelihood. Neurocomputing, 22, 49–67. Hyv¨arinen, A. (1998b). New approximations of differential entropy for independent component analysis and projection pursuit. In M. Kearns, M. Jordan, & S. Solla (Eds.), Advances in neural information processing systems, 10 (pp. 273–279). Cambridge, MA: MIT Press. Hyv¨arinen, A. (1999). Gaussian moments for noisy independent component analysis. IEEE Signal Processing Letters, 6(6), 145–147. Jutten, C., & Karhunen, J. (2003). Advances in nonlinear blind source separation. In Proceedins of ICA 2003 (pp. 245–256). Nara, Japan. Kawanabe, M., & Murata, N. (2000). Independent component analysis in the presence of gaussian noise (Tech. Rep. 2000-03). Tokyo: University of Tokyo. Lappalainen, H., & Honkela, A. (2000). Advances in independent component analysis. Berlin: Springer-Verlag. Lee, T.-W., Koehler, B., & Orglmeister, R. (1997). Blind separation of nonlinear mixing models. In IEEE International Workshop on Neural Networks for Signal Processing (pp. 406–415). Lewicki, M. S. (2002). Efficient coding of natural sounds. Nature Neuroscience, 5(4), 356–363. Lewicki, M. S., & Sejnowski, T. J. (2000). Learning overcomplete representations. Neural Computation, 12, 337–365. Lindsay, P. H., & Norman, D. A. (1977). Human information processing: An introduction to psychology. San Diego, CA: Academic Press. Merzenich, M. M., Knight, P. L., & Roth, G. L. (1975). Representation of cochlea within primary auditory cortex in the cat. J. Neurophysiol., 38, 231–249. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607–609. Pantev, C., Hoke, M., Lutkenhoner, B., & Lehnertz, K. (1989). Tonotopic organization of the auditory cortex: Pitch versus frequency representation. Science, 246, 486–488. Patterson, R. D. (1973). Physical variables determining residue pitch. J. Acoust. Soc. Am., 53, 1565–1572.

144

S. Maeda, W.-J. Song, and S. Ishii

Rao, R. P., & Ballard, D. H. (1999). Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1), 79–87. Reale, R. A., & Imig, T. J. (1980). Tonotopic orginization in auditory cortex of the cat. J. Comp. Neurol., 192, 265–291. Riquimaroux, H., & Hashikawa, T. (1994). Units in the primary auditory cortex of the Japanese monkey can demonstrate a conversion and place pitch in the central auditory system. Journal de physique IV, C5, 419–425. Ritsma, R. J. (1962). Existence region of the tonal residue. I. J. Acoust. Soc. Am., 42(1), 191–198. Schouten, J. F. (1940). The perception of pitch. Philips Technical Review, 5, 286–294. Schouten, J. F., Ritsma, R. J., & Cardozo, B. L. (1962). Pitch of the residue. J. Acoust. Soc. Am., 34, 1418–1424. Seebeck, A. (1843). Uber die sirene. Ann. Phys. Chem., 60, 449–481. Taleb, A. & Jutten, C. (1999). Source separation in post-nonlinear mixtures. IEEE Transaction on Signal Processing, 47(10), 2807–2820. Terhardt, E. (1974). Pitch, consonance, and harmony. J. Acoust. Soc. Am., 55, 1061–1069. Wightman, F. L. (1973). The pattern-transformation model of pitch. J. Acoust. Soc. Am., 54, 407–416. Yoshioka, M. (1980). Mathematical relations between three pitch theories. In Spring Meeting of the Acoustical Society of Japan (pp. 683–684). Zibulevsky, M., & Pearlmutter, B. A. (2001). Blind source separation by sparse decomposition in a signal dictionary. Neural Computation, 13(4), 863–882.

Received April 8, 2003; accepted June 4, 2004.

LETTER

Communicated by Edward Harrington

Online Ranking by Projecting Koby Crammer [email protected]

Yoram Singer [email protected] School of Computer Science and Engineering, Hebrew University, Jerusalem 91904, Israel

We discuss the problem of ranking instances. In our framework, each instance is associated with a rank or a rating, which is an integer in 1 to k. Our goal is to find a rank-prediction rule that assigns each instance a rank that is as close as possible to the instance’s true rank. We discuss a group of closely related online algorithms, analyze their performance in the mistake-bound model, and prove their correctness. We describe two sets of experiments, with synthetic data and with the EachMovie data set for collaborative filtering. In the experiments we performed, our algorithms outperform online algorithms for regression and classification applied to ranking. 1 Introduction The ranking problem we discuss in this article shares common properties with both classification and regression problems. As in classification problems, the goal is to assign one of k possible labels to a new instance. Similar to regression problems, the set of k labels is structured, as there is a total order relation between the labels. We refer to the labels as ranks and without loss of generality assume that the ranks constitute the set {1, 2, . . . , k}. Settings in which it is natural to rank or rate instances rather than classify are common in tasks such as information retrieval and collaborative filtering. We use the latter as our running example. In collaborative filtering, the goal is to predict a user’s rating on new items such as books or movies given the user’s past ratings of the similar items. The goal is to determine whether a movie fan will like a new movie and to what degree, which is expressed as a rank. An example for possible ratings might be, “run-to-see, very-good, good, only-if-you-must, and do-not-bother.” While the different ratings carry meaningful semantics, from a learning-theoretic point of view, we model the ratings as a totally ordered set (whose size is five in the example above). The interest in ordering or ranking of objects is by no means new and is still the source of ongoing research in many fields, such as mathematical economics, social science, and computer science. For an overview of rankNeural Computation 17, 145–175 (2005)

c 2004 Massachusetts Institute of Technology 

146

K. Crammer and Y. Singer

ing problems from a learning-theoretic point of view see Cohen, Schapire, and Singer (1999). One of the main results underscored in this article is a complexity gap between classification learning and ranking learning. To sidestep the inherent intractability problems of ranking learning, several approaches have been suggested. One possible approach is to cast a ranking problem as a regression problem. Another is to reduce a total order into a set of preferences over pairs (Freund, Iyer, Schapire, & Singer, 2003; Herbrich, Graepel, & Obermayer, 2000). The first case imposes a metric on the set of ranking rules that might not be realistic, while the second approach is time-consuming since it requires increasing the sample size from n to O(n2 ). In this letter, we consider an alternative approach that directly maintains a totally ordered set via projections. Our starting point is similar to that of Herbrich et al. (2000) in the sense that we project each instance into the real numbers. However, our work then deviates and operates directly on rankings by associating each ranking with distinct subinterval of the real numbers and adapting the support of each subinterval while learning. The article is organized as follows. In the next section, we describe a simple and efficient online algorithm that manipulates concurrently the direction onto which we project the instances and the division into subintervals. In section 3, we prove the correctness of the algorithm and analyze its performance in the mistake-bound model on various assumptions. Section 4 contains the description and analysis of a norm-optimized version of the basic ranking algorithm. We then shift our attention to a multiplicative algorithm that is described and analyzed in section 5. We provide empirical validation of the merits of the various ranking algorithms we devise in section 6. This section describes experiments comparing the ranking algorithms to online classification and regression algorithms. Finally, we conclude with a brief discussion and mention a few open problems. Before moving on to the core of the article, we point to a few closely related articles. First, a preliminary version of this letter appeared at Neural Information Processing Systems, 2001, under the title “Pranking with Ranking” (Crammer & Singer, 2001a). This article extends the conference version in numerous directions by providing complete analysis as well as three new algorithms that were not discussed in the conference version. Second, in a recent work by Shashua and Levin (2002), an SVM-based algorithm for instance ranking was described. The algorithms of Shashua and Levin share numerous properties with the algorithms presented in this article. However, it is designed for batch settings, while our focus is on online algorithms. Last, we would like to note that Harrington (2003) demonstrated empirically an improved generalization performance of our algorithm using an “averaging” technique, while preserving the order of the thresholds. 2 The Basic PRank Algorithm This letter focuses on online algorithms for ranking instances. We are given a sequence (x¯ 1 , y1 ), . . . , (x¯ t , yt ), . . . of instance-rank pairs. For concreteness,

Online Ranking by Projecting

147

we assume that each instance x¯ t is in Rn , and its corresponding rank yt is an element from finite set Y with a total order relation. We assume without loss of generality that Y = {1, 2, . . . , k} with > as the order relation. The total order over the set Y induces a partial order over the instances in the following natural sense. We say that x¯ t is preferred over x¯ s if yt > ys . We also say that x¯ t and x¯ s are not comparable if neither yt > ys nor yt < ys . We denote this case simply as yt = ys . Note that the induced partial order is of a unique form in that the instances form k equivalence classes that are totally ordered.1 We stress that although the elements of Y are denoted by integers, we do not use the fact that the integers also belong to a metric space. We assume only that the set Y is discrete and its elements are totally ordered. A ranking rule H is a mapping from the instance space to the set of rank values, H: Rn → Y . The family of ranking rules we discuss in this article employs a vector w¯ ∈ Rn and a set of k thresholds b1 ≤ · · · ≤ bk−1 ≤ bk = ∞. For convenience, we denote by b¯ = (b1 , . . . , bk−1 ) the vector of thresholds, ¯ the ranking rule excluding bk , which is fixed to ∞. Given a new instance x, ¯ The predicted rank is then first computes the inner product between w¯ and x. ¯ x¯ < br . defined to be the index of the first (smallest) threshold br for which w· Such a ranking rule divides the space into parallel, equally ranked regions: all the instances that satisfy br−1 < w¯ · x¯ < br are assigned the same rank ¯ the predicted rank of r. Formally, given a ranking rule defined by w¯ and b, ¯ = minr∈{1,...,k} {r : w¯ · x¯ − br < 0}. Note that the above an instance x¯ is H(x) minimum is always well defined since we set bk = ∞. The analysis that we use in this letter is based on the mistake-bound model for online learning. The algorithms we describe work in rounds. On round t, the learning algorithm gets an instance x¯ t . Given x¯ t , the algorithm outputs a rank, yˆ t = minr {r : w¯ · x¯ t − br < 0}. It then receives the correct ¯ We say that rank yt and updates its ranking rule by modifying w¯ and b. our algorithm made a ranking mistake if yˆ t = yt and wish to make the predicted rank as close (in a sense described later in this article) as possible to the true rank. Formally, the goal of the learning algorithm is to minimize the ranking loss, which is defined to be the number of thresholds between the true rank and the predicted rank. Using the representation of ranks as integers in {1, . . . , k}, the ranking loss after T rounds is equal to the cumulative difference between the predicted rank values and true rank values,
T ˆ t − yt |. The algorithm we describe updates its ranking rule only on t=1 |y rounds on which the predicted rank value was incorrect. Such algorithms are called conservative. We now describe the update rule of the algorithm, which is motivated by the perceptron algorithm for classification; hence, we call it the PRank algorithm, shorthand for perceptron ranking. For simplicity, we omit the ¯ y) and index of the round when referring to an input instance rank pair (x, ¯ Since b1 ≤ b2 ≤ · · · ≤ bk−1 ≤ bk , the predicted the ranking rule w¯ and b. 1

For a discussion of this type of partial orders see Kemeny and Snell (1962).

148

K. Crammer and Y. Singer

¯ x¯ > br for r = 1, . . . , y−1 and w· ¯ x¯ < br for r = y, . . . , k−1. rank is correct if w· We represent the above inequalities by expanding the rank y into k − 1 virtual variables y1 , . . . , yk−1 . We set yr = +1 for the case w¯ · x¯ > br and yr = −1 for w¯ · x¯ < br . Put another way, a rank value y induces the vector (y1 , . . . , yk−1 ) = (+1, . . . , +1, −1, . . . , −1) where the maximal index r for which yr = +1 is y−1. Thus, the prediction of a ranking rule is correct if yr (w¯ · x¯ − br ) > 0 for all r. If the algorithm makes a mistake by ranking x¯ as yˆ instead of y, then there is at least one threshold, indexed r, for which the ¯ x¯ is on the wrong side of br , that is, yr (w· ¯ x−b ¯ r ) ≤ 0. To correct the value of w· mistake, we need to “move” the values of w¯ · x¯ and br toward each other. We ¯ x−b ¯ r ) ≤
0 and do so by modifying only the values of the br ’s for which yr (w· replace them with br − yr . We also replace the value of w¯ with w¯ + ( yr )x¯ where the sum is taken over the indices r for which there was a prediction error, that is, yr (w¯ · x¯ − br ) ≤ 0. An illustration of the update rule is given in Figure 1. In the example, we used the set Y = {1, . . . , 5}. (Note that b5 = ∞ is omitted from all the plots in Figure 1.) The correct rank of the instance is y = 4, and thus the value of w¯ · x¯ should fall in the fourth interval, between b3 and b4 . However, in the illustration, the value of w¯ · x¯ fell below b1 and the predicted rank is yˆ = 1. The threshold values b1 , b2 , and b3 are a source of the error since the ¯ To compensate for the mistake, the value of b1 , b2 , b3 is higher than w¯ · x. algorithm decreases b1 , b2 , and b3 by a unit value and replaces them with b1 − 1, b2 − 1, and b3 − 1. It also modifies w¯ to be w¯ + 3x¯ since 

yr = 3.

¯ x−b ¯ r )≤0 r:yr (w·

¯ 2 . This update is illustrated at Thus, the inner product w¯ · x¯ increases by 3x the middle plot of Figure 1. The prediction rule after the update is illustrated on the right-hand side of Figure 1. Note that after the update, the predicted rank of x¯ is yˆ = 3, which is closer to the true rank y = 4. The pseudocode of algorithm is given in Figure 2. To conclude this section, we note that PRank can be straightforwardly combined with Mercer kernels (Vapnik, 1998) and voting techniques (Freund & Schapire, 1999) often used for improving the performance of margin classifiers in batch and online settings (Cristianini & Shawe-Taylor, 2000). To do so, we assume access to an inner-product space X equipped with a kernel operator K: X × X → R. We now need to replace any innerproduct operation the algorithm performs with an implicit inner-product operation defined via the kernel operator. We thus keep inner-product operators rather than explicit vectors. For instance, the update of PRank becomes K(w¯

t+1

, ·) ← K(w¯ , ·) + t

  r

 τrt

K(x¯ t , ·),

Online Ranking by Projecting

149

P re d icte d ra n k

1

C o rrec t in te rv a l

2

3

4

5

C o rrec t in te rv a l

U pd a ted p re d icte d ra nk

1

2

C o rrec t in te rv a l

3

4

5

Figure 1: An Illustration of the update rule. The ranking rule predicts a rank of yˆ = 1 instead of y = 4 (top). The update decreases the thresholds b1 , b2 , b3 by ¯ with w ¯ + 3¯x (center), yielding that the predicted rank of one unit and replaces w x¯ after the update is yˆ = 3 (bottom).

and thus the inner product w¯ T · x¯ t becomes t−1   s=1

τrt K(x¯ s , x¯ t ).

r

3 Analysis Before we prove the mistake bound of the algorithm, we first need to show that it maintains a consistent hypothesis. That is, we need to show that PRank preserves the correct order of the thresholds; otherwise, it might be impossible to induce a rank prediction rule. We prove that the consistency of thresholds is maintained by showing inductively that for any ranking rule that can be derived by the algorithm along its run, (w¯ 1 , b¯1 ), . . . , (w¯ T+1 , b¯T+1 ) the set of inequalities btr ≤ · · · ≤ btk−1 hold for all t. Clearly, since the initialization of the thresholds is such that b11 ≤ b12 ≤ · · · ≤ b1k−1 , then it suffices

150

K. Crammer and Y. Singer

Initialize: Set w¯ 1 = 0¯ , b11 , . . . , b1k−1 = 0, b1k = ∞ Loop: For t = 1, 2, . . . , T • Receive a new instance x¯ t ∈ Rn • Predict: yˆ t = min {r : w¯ t · x¯ t − btr < 0} r∈{1,...,k}

• Receive a new rank-value yt • If yˆ t = yt update w¯ t (otherwise set w¯ t+1 = w¯ t , ∀r: bt+1 = btr ): r 1. For r = 1, . . . , k − 1 : 2. For r = 1, . . . , k − 1 :

If yt ≤ r Then ytr = −1 Else ytr = 1 If (w¯ t · x¯ t − btr )ytr ≤ 0 Then τrt = ytr Else τrt = 0

3. Update: For r = 1, . . . , k − 1 :


t  t ¯ w¯ t+1 ← w¯ t + r τr x t − τt bt+1 ← b r r r

¯ = minr∈{1,...,k} {r : w¯ T+1 · x¯ − bT+1 Output: H(x) < 0}. r Figure 2: The PRank algorithm.

to show that the claim holds inductively. For simplicity of the proof below, let us write the update rule of PRank in an alternative form. Let [[π ]] be 1 if the predicate π holds and 0 otherwise. We now rewrite the value of τrt (from Figure 2) as τrt = ytr [[(w¯ t · x¯ t − btr )ytr ≤ 0]]. Note also that the values of btr are integers for all r and t since for all r, we initialize b1r = 0 and bt+1 − btr ∈ {−1, 0, +1}. r Lemma 1 (order preservation). Let w¯ t and b¯t be the current ranking rule, where bt1 ≤ · · · ≤ btk−1 , and let (x¯ t , yt ) be an instance rank pair fed to PRank on round t. Denote by w¯ t+1 and b¯t+1 the resulting ranking rule after the update of PRank. Then bt+1 ≤ · · · ≤ bt+1 1 k−1 . Proof. In order to show that PRank preserves a nondecreasing order of the thresholds, we use the definition of the algorithm for ytr . We define ytr = +1 t+1 for r < yt and ytr = −1 for r ≥ yt . To prove that bt+1 r+1 ≥ br , we rewrite the threshold update as bt+1 = btr − ytr [[(w¯ t · x¯ t − btr )ytr ≤ 0]]. r

(3.1)

Online Ranking by Projecting

151

t+1 for all feasible r, it is sufficient to show In order to prove that bt+1 r+1 ≥ br that

btr+1 − btr ≥ ytr+1 [[(w¯ t · x¯ t − btr+1 )ytr+1 ≤ 0]] − ytr [[(w¯ t · x¯ t − btr )ytr ≤ 0]].(3.2) Since by our inductive assumption btr+1 ≤ btr and btr , btr+1 ∈ Z, we get that the value of btr+1 − btr on the left-hand side of equation 3.2 is a nonnegative integer. Recall also that ytr = 1 if yt > r and ytr = −1 otherwise, and therefore ytr+1 ≤ ytr . We now need to analyze two cases. We first consider the case where ytr+1 = ytr , which implies that ytr+1 = −1, ytr = +1. In this case, the right-hand side of equation 3.2 is at most zero, and the claim trivially holds. The second case is when ytr+1 = ytr . Here we get that the value of the right-hand side of equation 3.2 cannot exceed 1. If btr+1 > btr , then since both values are integers, the bound of 1 on the right-hand side of equation 3.2 implies that the value of btr cannot exceed the value of btr+1 . We are thus left with the case where btr = btr+1 and ytr+1 = ytr . For this case, we have that the terms ytr+1 [[(w¯ t · x¯ t − btr+1 )ytr+1 < 0]] and ytr [[(w¯ t · x¯ t − btr )ytr < 0]] attain the same value. Therefore, the right-hand side of equation 3.2 is zero and bt+1 = bt+1 r r+1 . This completes the proof. We now turn to the mistake-bound analysis of the algorithm. In order to simplify the analysis of the algorithm, we introduce the following notation. ¯ we denote by v¯ ∈ Rn+k−1 Given a hyperplane w¯ and a set of k−1 thresholds b, ¯ ¯ For brevity, ¯ ¯ b). the vector that is a concatenation of w and b, that is, v¯ = (w,  we refer to the vector v¯ as a ranking rule. Given two vectors v¯ = (w¯  , b¯ ) and ¯ 2 . Note that ¯ we have v¯  · v¯ = w¯  · w¯ + b¯ · b¯ and v ¯ 2 = w ¯ 2 + b ¯ b), v¯ = (w, n a vector v¯ induces a partial order of the vectors R . Theorem 1 (mistake bound). Let (x¯ 1 , y1 ), . . . , (x¯ T , yT ) be an input sequence to PRank where x¯ t ∈ Rn and yt ∈ {1, . . . , k}. Denote by R2 = maxt x¯ t 2 . If there exists a ranking rule v¯ ∗ = (w¯ ∗ , b¯∗ ) with b∗1 ≤ · · · ≤ b∗k−1 of a unit norm that classifies the entire sequence correctly with margin γ = minr,t {(w¯ ∗ ·x¯ t −b∗r )ytr } > 0,
the ranking loss of the algorithm Tt=1 |yˆ t − yt | is at most (k − 1)

R2 + 1 . γ2

Proof. Let us examine an example (x¯ t , yt ) that the algorithm received on round t. By definition, the algorithm ranked the example using the ranking rule v¯ t , which is composed of w¯ t and the set of thresholds b¯t . Similarly, v¯ t+1 is 
t  t ¯ the ranking rule (w¯ t+1 , b¯t+1 ) after round t. Therefore, w¯ t+1 = w¯ t + r τr x t − τ t for r = 1, 2, . . . , k − 1. Let us denote by nt = |y t − yt | the ˆ and bt+1 = b r r r

152

K. Crammer and Y. Singer

difference between the true rank and the predicted rank. Since τrt is zero for t t t t all the indices r such
thatt sign(w¯ · x¯ − br )yr > 0, then it is straightforward t to verify that n = r |τr |. Note that if there was not a ranking mistake on round t, then τrt = 0 for
r = 1, . . . , k − 1, and thus also nt = 0. To prove the theorem, we bound t nt from above by bounding v¯ t 2 from above and below. First, we derive a lower bound on v¯ t 2 by bounding v¯ ∗ · v¯ t+1 . Substituting the values of w¯ t+1 and b¯t+1 , we get that v¯ ∗ · v¯ t+1 = v¯ ∗ · v¯ t +

k−1 

τrt (w¯ ∗ · x¯ t − b∗r ).

(3.3)

r=1

We further bound the term on the right-hand side by considering two cases corresponding to whether τrt is positive or zero. Using the definition of τrt from the pseudocode in Figure 2, we first examine the case where (w¯ t · x¯ t − btr )ytr ≤ 0 and therefore τrt = ytr . From the assumption that v¯ ∗ ranks the examples correctly with a margin of at least γ , we get that τrt (w¯ ∗ · x¯ t −b∗r ) ≥ γ . The second case is when (w¯ t · x¯ t − btr )ytr > 0. In this case, we have τrt = 0 and thus τrt (w¯ ∗ · x¯ t − b∗r ) = 0. Combining the two cases and summing now over r, we get k−1 

τrt (w¯ ∗ · x¯ t − b∗r ) ≥

r=1

k−1 

|τrt |γ = nt γ .

(3.4)

r=1

Combining equations 3.3 and 3.4, we get that v¯ ∗ ·v¯ t+1 ≥ v¯ ∗ ·v¯ t +nt γ . Unfolding the sum, we get that after T rounds, the projection of the vector v¯ T+1 on v¯ ∗ satisfies v¯ ∗ · v¯ T+1 ≥



nt γ = γ



t

nt .

(3.5)

t

Recall that Cauchy-Schwartz inequality implies that v¯ T+1 2 v¯ ∗ 2 ≥ (v¯ T+1 · v¯ ∗ )2 . Using equation 3.5 with Cauchy-Schwartz inequality and the assumption that v¯ ∗ is of a unit norm, we get the following lower bound: v¯

T+1 2

 ≥

 

2 n

t

γ 2.

(3.6)

t

We next bound the norm of v¯ T+1 from above. As before, assume that an example (x¯ t , yt ) was ranked using the ranking rule v¯ t , and denote by v¯ t+1

Online Ranking by Projecting

153

the ranking rule at the end of round t. We now expand the values of w¯ t+1 and b¯t+1 whose sum is the norm of v¯ t+1 and get    τrt w¯ t · x¯ t − btr v¯ t+1 2 = w¯ t 2 + b¯t 2 + 2  +



r

2

x¯ t 2 +

τrt

 (τrt )2 .

r

r





Since τrt ∈ {−1, 0, +1}, we have that ( r τrt )2 ≤ (nt )2 and r (τrt )2 = nt , and we therefore get    v¯ t+1 2 ≤ v¯ t 2 + 2 τrt w¯ t · x¯ t − btr + (nt )2 x¯ t 2 + nt . (3.7) r

We further develop the second term using the update rule of the algorithm and get       τrt w¯ t · x¯ t − btr = [[(w¯ t · x¯ t − btr )ytr ≤ 0]] (w¯ t · x¯ t − btr )ytr ≤ 0. (3.8) r

r

Plugging equation 3.8 into equation 3.7 and using the bound x¯ t 2 ≤ R2 , we get that v¯ t+1 2 ≤ v¯ t 2 + (nt )2 R2 + nt . Thus, the ranking rule we obtain after T rounds of the algorithm is upper bounded by   (nt )2 + nt . (3.9) v¯ T+1 2 ≤ R2 t

t


Combining the lower bound v¯ T+1 2 ≥ ( t nt )2 γ 2 with the upper bound of equation 3.9, we have that  

2 t

n

γ 2 ≤ v¯ T+1 2 ≤ R2

t



(nt )2 +

t



nt .

t


Dividing the above equations by γ 2 t nt , we finally get

 R2 [ t (nt )2 ]/[ t nt ] + 1 nt ≤ . γ2 t Since by definition, nt is at most k − 1, it implies that    (nt )2 ≤ nt (k − 1) = (k − 1) nt . t

t

t

(3.10)

154

K. Crammer and Y. Singer

Plugging this inequality into equation 3.10, we get the desired bound, T 

|yˆ t − yt | =

t=1

T 

nt ≤

t=1

(k − 1)R2 + 1 R2 + 1 ≤ (k − 1) . γ2 γ2

We now turn our attention to the inseparable case in which there does not exist a positive margin value γ . To analyze the inseparable case, we use an analysis technique suggested by Freund and Schapire (1999). (This proof technique was first informally given by Vapnik, 1998.) To prove a mistake bound for the inseparable case, each example (x¯ t , yt ) is augmented with a slack variable, denoted dt . Informally, for each time step t, the variable dt designates how much the margin assumption is violated by the example. Formally, we obtain the following mistake bound for the inseparable case: Theorem 2 (mistake bound for inseparable case). Let (x¯ 1 , y1 ), . . . , (x¯ T , yT ) be an input sequence for PRank where x¯ t ∈ Rn and yt ∈ {1, . . . , k}. Denote by R2 = maxt x¯ t 2 . Let v¯ ∗ = (w¯ ∗ , b¯∗ ) be a ranking rule of a unit norm with b∗1 ≤ · · · ≤ b∗k−1 . Let γ > 0 and define dt = max{0, γ − min{(w¯ ∗ · x¯ t − b∗r )ytr }}. r

Denote by D2 = by T 




t 2 t (d ) .

(3.11)

Then the ranking loss of the algorithm is bounded above

|yˆ t − yt | ≤ (k − 1)

(D +

t=1

√ R2 + 1)2 . γ2

The proof of the theorem is based on the proof technique for theorem 1 and is given in the appendix. To conclude this section, we describe and briefly analyze a simplified version of PRank. This version shares the same algorithmic skeleton as PRank and differs only in the way it modifies its ranking rule. Rather than modifying all of the thresholds in the error set, this version chooses a single threshold to modify and update w¯ accordingly. Since a single threshold is modified on each round, we use the abbreviation Si-PRank to refer to this version. More formally, we replace step 3 in Figure 2 with the following steps, 3. Define update index: If yˆ t > yt Then r = yˆ t − 1 Else r = yˆ t 4. Update: w¯ t+1 ← w¯ t + τrt x¯ t bt+1 ← btr − τrt r bt+1 ← bts s

(s = r).

Online Ranking by Projecting

155

It is to verify that this choice of a single threshold to update preserves the overall order of the threshold. The proof is immediate consequence of the lemma 1. In addition, following exactly the same proof technique of theorem 1, we get that the mistake bound of PRank also holds for Si-PRank.2 Surprisingly, as we see later in section 6, in practice Si-PRank performs slightly better than PRank. 4 A Norm-Optimized Version of PRank The PRank algorithm presented in the previous section performs the same form of update whenever a ranking error occurs. Furthermore, even when the projection of xt onto w¯ t yields the correct rank value, the value of w¯ t · xt might lie very close to one of the thresholds btr . The difference between the projection of xt and the closest threshold plays a similar role to the notion of margin in classification problems and was used in theorem 1 to derive a mistake bound for PRank. In this section, we present a version of PRank that solves on each round a mini-optimization problem that balances between two opposing requirements. On one hand, we require that the new ranking rule v¯ t+1 be as similar as possible to the previous ranking rule v¯ t , which encompasses all our knowledge on past examples. On the other hand, we force the new ranking rule to rank the most recent example x¯ t correctly and with a large enough margin. Formally, assuming that there was a rank prediction error on round t, then we require that the new rule (w¯ t+1 , b¯t+1 ) t would satisfy, minr {(w¯ t+1 · x¯ t − bt+1 r )yr }} ≥ β, where β is a positive constant. These two requirements yield the following optimization problem: min ¯ b¯ w,

1 ¯ − (w¯ t , b¯t )2 ¯ b) (w, 2

subject to: (w¯ · x¯ t − br )ytr ≥ β

for r = 1, . . . , k − 1.

(4.1)

We call this version the Norm-Optimized PRank algorithm, or No-PRank in short. The pseudocode of the algorithm appears in Figure 3. Before analyzing the algorithm, let us first further develop the optimization problem given in equation 4.1 by expanding its Lagrangian function, k−1  1 ¯ − (w¯ t , b¯t )2 − ¯ b) L = (w, τr [(w¯ · x¯ t − br )ytr − β]. 2 r=1

(4.2)

2 In fact, an alternative bound can be given for Si-PRank, which states that the number of rounds on which an error occurs (indicated by a strictly positive value of the loss) is upper bounded by

R2 + 1 . γ2

156

K. Crammer and Y. Singer

Input: Minimal margin parameter β Initialize: Set w¯ 1 = 0¯ , b11 , . . . , b1k−1 = 0, b1k = ∞ Loop: For t = 1, 2, . . . , T • Receive a new instance x¯ t ∈ Rn . • Predict: yˆ t = min {r : w¯ t · x¯ t − btr < 0} r∈{1,...,k}

• Receive a new rank-value yt • If yt = yˆ t update w¯ t and b¯t : (otherwise set w¯ t+1 = w¯ t , b¯t+1 = b¯t ): 1. For r = 1, . . . , k − 1 :

If yt ≤ r Then ytr = −1 Else ytr = 1.

2. Update: set (w¯ t+1 , b¯t+1 ) ∈ Rn+k−1 to be the minimizer of: min ¯ b¯ w,

1 ¯ − (w¯ t , b¯t )2 ¯ b) (w, 2

subject to: (w¯ · x¯ t − br )ytr ≥ β

for r = 1, . . . , k − 1.

¯ = minr∈{1,...,k} {r : w¯ T+1 · x¯ − bT+1 < 0}. Output: H(x) r Figure 3: The Norm-Optimized PRank algorithm.

Here, τr are nonnegative Lagrange multipliers. Taking the derivative of equation 4.2 with respect to w¯ and comparing it to zero, we get  ∂ L = w¯ − w¯ t − x¯ t τr ytr = 0 ∂ w¯ r

w¯ = w¯ + t

⇒

 

 τr ytr

x¯ t . (4.3)

r

Repeating the process for br , we get ∂ L = br − btr + τr ytr = 0 ∂br

⇒

br = btr − τr ytr .

(4.4)

Plugging the value of w¯ from equation 4.3 and b from equation 4.4 into equation 4.2, we get the following dual problem:  2 1 t 2  t 1 2 minQt (τ¯ ) = x¯  τr yr + τ τ 2 2 r r r      + τr ytr w¯ t · x¯ t − btr − β r

s.t 0 ≤ τr r = 1, . . . , k − 1.

(4.5)

Online Ranking by Projecting

157

Before proceeding to discuss the formal properties of the No-PRank algorithm, it is worth noting that the new vector obtained at the end of round t, w¯ t+1 is a linear combination of w¯ t and the current instance x¯ t . Therefore, it is rather straightforward to use No-PRank in conjunction with kernel methods by maintaining w¯ t in its dual form as a weighted combination of the instances x¯ 1 , . . . , x¯ t . Note also that the optimization problem of No-PRank reduces to a simple optimization problem with a quadratic objective function and nonnegativity constraints and can be solved using standard convex optimization tools (Fletcher, 1987). Let us now discuss the formal properties of No-PRank. The following simple lemma states that No-PRank is a conservative online algorithm as it modifies its ranking rule if and only if the minimal margin requirements are not attained for the current instance. Lemma 2 (conservativeness). Let (x¯ t , yt ) denote the input to No-PRank  on round t, and let ytr be −1 if yt ≤ r and +1 otherwise. Then if ytr w¯ t · x¯ t − btr ≥ β for all r = 1, . . . , k−1 the optimum of No-PRank’s optimization problem is attained at τr = 0 for all r. Proof. Consider the dual form of No-PRank’s optimization problem given by equation 4.5. The objective function Qt (τ¯ ) is composed of three summands. The first two are clearly nonnegative and attain a value of zero iff all the multipliers τr are zero. If ytr (w¯ · x¯ − btr ) ≥ β, then the third summand is linear in τ¯ with nonnegative coefficients. Thus, its minimum is also attained when τr is zero for all r. Next we show that No-PRank preserves the order of the thresholds along its run and thus can always serve as a valid ranking rule. Lemma 3 (order preservation). Let w¯ t and b¯t be the current ranking rule, where bt1 ≤ · · · ≤ btk−1 , and let (x¯ t , yt ) be an instance-rank pair fed to No-PRank on round t. Denote by w¯ t+1 and b¯t+1 the resulting ranking rule after the update of No-PRank. Then bt+1 ≤ · · · ≤ bt+1 1 k−1 . Proof. In order to show that No-PRank maintains a correct (monotonically increasing) order of the thresholds, we use the variables employed by the algorithm along its run. Namely, we set ytr = +1 for r < yt and ytr = −1 for t+1 for all r, we expand b¯t+1 and show that r ≥ yt . To prove that bt+1 r+1 ≥ br t − ytr τrt . btr+1 − btr ≥ ytr+1 τr+1

(4.6)

We need to analyze two different settings. The first setting we analyze is when ytr+1 = ytr , which implies that ytr+1 = −1 and ytr = +1. In this case, the right-hand side of equation 4.6 is at most zero, while the left-hand side of

158

K. Crammer and Y. Singer

the equation is at least zero. Thus, the claim holds. The other case is when ytr+1 = ytr = y. In this case, equation 4.6 becomes t − τrt ). btr+1 − btr ≥ y(τr+1

(4.7)

Assume by contradiction that the optimal value of equation 4.5 does not satisfy equation 4.7 for some r. We now construct another feasible set for τ¯ that yields a lower value of the objective function Q. Let us define  τs − y τs = τs + y  τs

s=r+1 s=r otherwise,

for some value of > 0, which is determined below. Informally, the value of is set to be small enough so that the constraint τs ≥ 0 still holds. (This is possible since τr+1 > 0.) Since τ¯  and τ¯ differ only at their r and r + 1 components, we get

Q(τ¯  ) − Q(τ¯ ) =

1 2  2 ) + τr [yr (w¯ t · x¯ t − btr ) − β] (τ + τr+1 2 r  + τr+1 [yr+1 (w¯ t · x¯ t − btr+1 ) − β] 1 − (τr 2 + τr+1 2 ) − τr [yr (w¯ t · x¯ t − btr ) − β] 2 − τr+1 [yr+1 (w¯ t · x¯ t − btr+1 ) − β].

Expanding τ¯  , we now get

Q(τ¯  ) − Q(τ¯ ) =

1 ((τr + y )2 + (τr+1 − y )2 ) 2 + (τr + y )[yr (w¯ t · x¯ t − btr ) − β] + (τr+1 − y )[yr+1 (w¯ t · x¯ t − btr+1 ) − β] 1 − (τr 2 + τr+1 2 ) − τr [yr (w¯ t · x¯ t − btr ) − β] 2 − τr+1 [yr+1 (w¯ t · x¯ t − btr+1 ) − β]

= (y )2 + y (τr − τr+1 ) + y [yr (w¯ t · x¯ t − btr ) − β] − y [yr+1 (w¯ t · x¯ t − btr+1 ) − β]. Denoting both yr and yr+1 simply as y and using the fact y ∈ {−1, +1}, we get

Q(τ¯  ) − Q(τ¯ ) = (y )2 + y (τr − τr+1 ) − br + br+1 = [ − (y(τr+1 − τr ) − (br+1 − br ))].

Online Ranking by Projecting

159

Since we assumed by contradiction that y(τr+1 − τr ) − (br+1 − br ) = A > 0, we can choose 0 < ≤ A/2 and get

Q(τ¯  ) − Q(τ¯ ) = (A/2 − A) = − A/2 < 0, which contradicts the assumption that τ is the optimal solution of equation 4.5. We now turn to the analysis of the performance of No-PRank. We first bound the sum of the weight employed by the dual program as they accu

t mulate along the run of No-PRank t r τr . Based on the bound on the weights, we then prove a mistake bounds on the number of rounds on which an error occurred, that is, yt = yˆ t . As we see shortly, the bound depends on both the value of the attainable margin γ and the margin parameter, β, employed by the algorithm. Theorem 3 (bound on weights). Let (x¯ 1 , y1 ), . . . , (x¯ T , yT ) be an input sequence to No-PRank where x¯ t ∈ Rn and yt ∈ {1,
. . . , k}. Let v¯ ∗ = (w¯ ∗ , b¯∗ ) be ∗ ∗ ∗ 2 ¯ a ranking rule with b1 ≤ · · · ≤ bk−1 and w  + r (b∗r )2 = 1. Assume that v¯ ∗ classifies the entire sequence correctly with a margin value γ = minr,t {(w¯ ∗ · x¯ t − b∗r )ytr } > 0. Then the total sum of the weights generated by No-PRank is bounded by T   t=1

τrt ≤ 2

r

β , γ2

where β is a predefined parameter of the algorithm (see Figure 3). Proof. Let us concentrate on an example (x¯ t , yt ) that the algorithm received on round t. By construction, the algorithm ranked the example using the ranking rule v¯ t , which is composed of w¯ t and the thresholds b¯t . Similarly, we denote by v¯ t+1 the updated rule (w¯ t+1 , b¯t+1 ) after round t, that is, w¯

t+1

= w¯ + t

 

 ytr τrt

= btr − ytr τrt for r = 1, 2, . . . , k − 1. x¯ t and bt+1 r

r



To bound t r τrt from above, we derive bounds on v¯ t 2 from both above and below. First, we derive a lower bound on v¯ t 2 by bounding v¯ ∗ · v¯ t+1 . Substituting the values of w¯ t+1 and b¯t+1 , we get v¯ ∗ · v¯ t+1 = v¯ ∗ · v¯ t +

k−1  r=1

τrt ytr (w¯ ∗ · x¯ t − b∗r ).

160

K. Crammer and Y. Singer

Using the assumption that v¯ ∗ ranks the data correctly with a margin of at least γ , we get that ytr (w¯ ∗ · x¯ t − b∗r ) ≥ γ : v¯ ∗ · v¯ t+1 ≥ v¯ ∗ · v¯ t +

k−1 

τrt γ = v¯ ∗ · v¯ t + γ

r=1

k−1 

τrt .

r=1

Unfolding the sum, we get that after T rounds, the algorithm satisfies v¯ ∗ · v¯ T+1 ≥ γ



τrt .

t,r

Plugging this result into Cauchy-Schwartz inequality, v¯ T+1 2 v¯ ∗ 2 ≥ (v¯ T+1 · v¯ ∗ )2 , and using the assumption that v¯ ∗ is of a unit norm, we get the lower bound, v¯

T+1 2

 ≥

 

2 τrt

γ 2.

(4.8)

t,r

We next bound the norm of v¯ T+1 from above. As before, assume that an example (x¯ t , yt ) was ranked using the ranking rule v¯ t , and denote by v¯ t+1 the ranking rule after the round. We now expand the values of w¯ t+1 and b¯t+1 in the norm of v¯ t+1 and get v¯ t+1 2 = w¯ t 2 + b¯t 2 + 2  + x¯ t 2



τrt ytr (w¯ t · x¯ t − btr )

r



2 τrt ytr

+

 (ytr τrt )2 .

r

r

We add and subtract the term 2β equation and get v¯ t+1 2 = w¯ t 2 + b¯t 2 + 2  + x¯ 

t 2

 r




t r τr



τrt [ytr (w¯ t · x¯ t − btr ) − β]

r 2

τrt ytr

on the right-hand side of the above

+



ytr τrt

r

= w¯ t 2 + b¯t 2 + 2Q(τ¯ t ) + 2β

2



+ 2β



τrt

r

τrt ,

(4.9)

r

where we used the definition of Q from equation 4.5 to obtain the last equality.

Online Ranking by Projecting

161

Note that τ¯ = 0 is a feasible solution for the optimization problem posed in equation 4.5. The value attained by Q for this particular choice of τ¯ is zero. Since τ¯ t is the minimizer Q(τ¯ ), we must have that

Q(τ¯ t ) ≤ 0.

(4.10)

Substituting equation 4.10 in equation 4.9, we obtain the following bound on the norm of v¯ t+1 in terms of the norm of v¯ t :  τrt . (4.11) v¯ t+1 2 ≤ v¯ t 2 + 2β r

Thus, the ranking rule we obtain after T rounds of the algorithm satisfies the upper bound: v¯ T+1 2 ≤ 2β



τrt .

(4.12)

t,r

Combining the lower bound of equation 4.8 with the upper bound of equation 4.12, we have that  

2 τrt

γ 2 ≤ v¯ T+1 2 ≤ 2β

t,r

t,r

τrt ≤

τrt .

t,r

Dividing both sides by γ 2 



2β . γ2




t t,r τr ,

we finally get (4.13)

We now discuss some properties of the algorithm and its corresponding loss bound. As mentioned above, the algorithm updates its ranking rule only on rounds on which the margin is less than β, which reflects a minimal margin requirement. Informally, β can be viewed as a minimal difference requirement between the projection of the example x¯ t onto w¯ t and any of the thresholds btr (see Figure 4). Thus, the larger β is, the better is the separation between the rank levels. Formally, the algorithm attempts to enclose all the examples in subintervals of the real numbers defined by the thresholds. As stated above, based on theorem 3, we can derive a mistake bound for NoPRank. Surprisingly, the dependency on the margin parameter β cancels out, and the end result is a mistake bound that depends on only the geometrical properties of the problem: the normalized margin. Corollary 1 (mistake bound). Assume the conditions of theorem 3 hold, and denote by R = maxt x¯ t . Then the number of rounds for which No-PRank made

162

K. Crammer and Y. Singer

β

b1

β



WX 

b2

b3

b4

Figure 4: An illustration of the margin required by the No-PRank algorithm.

a mistake is upper bounded by

2

R2 + 1 . γ2

Proof. To prove the corollary, we use theorem 3 and show
that whenever an error occurred on round t, the sum of the dual weights r τrt is at least β/(R2 + 1). Since w¯ t+1 must satisfy the constraints of equation 4.1, we now show that for r = 1, . . . , k − 1, t (w¯ t+1 · x¯ t − bt+1 r )yr ≥ β.


from equation 4.3 and bt+1 = Substituting w¯ t+1 = w¯ t + x¯ t r τrt ytr and bt+1 r r t t t br − τr yr from equation 4.4, we get that the dual variables τ1t , . . . , τk−1 satisfy  β≤

w¯ + x¯ t

t





· x¯ −

τst yts

t

btr

+

τr ytr

ytr .

s

Rearranging terms in the above equation, we get  β≤

w¯ + x¯ t

t



 τst yts

· x¯ − t

s

= (w¯ t · x¯ t − btr )ytr + ytr x¯ t 2

btr



+

τr ytr

ytr

τst yts + τrt (ytr )2 .

(4.14)

s

Since a rank prediction error occurred on round t, there exists r such
that t·x ¯ t − btr )ytr ≤ 0. In addition, since ytr ∈ {−1, +1}, we have that ytr s τst yts (w¯
≤ s τst . Using these two inequalities in equation 4.14 in conjunction with

Online Ranking by Projecting

163

the fact that τrt ≥ 0, we get β ≤ 0 + x¯ t 2



τrt +



r

≤ (R2 + 1)



τrt

r

τrt .

r

We have thus shown that if there was a rank prediction error on round t, then  β τrt . ≤ R2 + 1 r

(4.15)

To prove the corollary, we combine theorem 3 with equation 4.15. Denote by M the number of rounds with errors. We now show that on each
prediction 2 + 1) ≤ t , and for the rest of the rounds, we simply such round, β/(R τ r r
bound the sum r τrt from below by 0. We thus get that M

R2

 β τrt . ≤ +1 t,r

Applying theorem 3, we get M

β R2 + 1 β ⇒ M ≤ 2 . ≤ 2 R2 + 1 γ2 γ2

Note that the mistake bounds of theorem 1 and corollary 1 are identical up to a multiplicative (k − 1)/2 factor. Furthermore, if we employ the t t trivial
t factt that |y − yˆ | ≤ k − 1, we can immediately obtain a bound on ˆ |y − y | from corollary 1 that is a 2 factor of the bound given in theot rem 1. However, the bound for No-PRank is more refined, as in addition
to the mistake bound, we can also bound the total sum of the weights t,r τrt . Unfortunately, since τrt may be arbitrarily close to zero, the more refined analysis does not yield a better mistake bound. One possible direction for improving the mistake bound itself may be obtained by modifying the minimal margin requirement from β to β|yt − yˆ t |. Another viable approach is to replace the fixed margin constraint (w¯ t+1 · x¯ t − brt+1 )ytr ≥ β with a margin requirement that is dependent on the difference between the correct label yt and the specific threshold r. Specifically, we define a new set of constraints (w¯ t+1 · x¯ t − brt+1 )ytr ≥ βar , where ar = yt − r for r < yt and ar = r + 1 − yt for r ≥ yt . Therefore, the further the threshold r from the interval containing the projection of the instance on the hyperplane, the larger the margin we require. We leave these possible extensions to future research.

164

K. Crammer and Y. Singer

5 A Multiplicative Version of PRank In this section, we give a multiplicative version of PRank that we term MuPRank. This version is analogous to the basic PRank update, but it modifies w¯ and b¯ in a multiplicative manner. That is, on each round with rank prediction error, the current weight vector w¯ t and thresholds b¯t are multiplied by factors that depend on x¯ t . The motivation for this version is the multiplicative updates employed by the work of Warmuth and colleagues on online prediction (see, e.g., Kivinen & Warmuth, 1997). Mu-PRank maintains a normalized ranking rule, that is, (w¯ t , b¯t )1 = 1 for all t. As in PRank, on round t, Mu-PRank computes the vector τ¯ t , which determines the update of w¯ t and b¯t . It then takes the exponent of the updated ranking rule and normalizes it so that the 1 norm of (w¯ t+1 , b¯t+1 ) will be 1. The pseudocode of the algorithm is given in Figure 5. Examining the logarithm of b¯t on each round, it is rather simple to verInput: Learning rate η 1 Initialize: Set w1i = n+k−1 Loop: For t = 1, 2, . . . , T

i = 1, . . . , n , b11 , . . . , b1k−1 =

• Receive a new instance x¯ t ∈ Rn ,

1 n+k−1 ,

b1k = ∞

x¯ t ∞ ≤ 1

• Predict: yˆ t = min {r : w¯ t · x¯ t − btr < 0} r∈{1,...,k}

• Receive a new rank-value yt • If yt = yˆ t update w¯ t and b¯t (otherwise set w¯ t+1 = w¯ t , b¯t+1 = b¯t ): 1. For r = 1, . . . , k − 1 :

If yt ≤ r Then ytr = −1 Else ytr = 1 If (w¯ t · x¯ t − btr )ytr ≤ 0 Then τrt = ytr Else τrt = 0

2. For r = 1, . . . , k − 1 : 3. Define:

Zt =

n 

wti eηxi

t




τt r r

i=1

+

k−1 

btr e−ητr

t

r=1

4. Update:


t t For i = 1, . . . , n : wt+1 ← wti eηxi r τr /Z t i t For r = 1, . . . , k − 1 : bt+1 ← btr e−ητr /Z t r

¯ = minr∈{1,...,k} {r : w¯ T+1 · x¯ − bT+1 Output: H(x) < 0} r Figure 5: The multiplicative algorithm.

Online Ranking by Projecting

165

ify that, like PRank, Mu-PRank preserves the order of the thresholds. The additional step Mu-PRank employs in its update stage is the normalization of its ranking rule. However, since this normalization is applied to all the thresholds, it clearly keeps the order of the thresholds intact. More formally, log(btr ) can be written as ηutr + log(Ct ), where utr is an integer and Ct is independent of r. Applying exactly the same proof technique used in lemma 1 to utr gives the following corollary: Lemma 4 (order preservation). Let w¯ t and b¯t denote the current ranking rule and assume that bt1 ≤ · · · ≤ btk−1 . Then, after (x¯ t , yt ) is fed to Mu-PRank, the new rule (w¯ t+1 , b¯t+1 ) preserves the order of the thresholds, bt+1 ≤ · · · ≤ bt+1 1 k−1 . Next, we analyze the mistake bound of Mu-PRank. Note that Mu-PRank employs a learning rate parameter, η. The mistake bound of Mu-PRank given in the following theorem depends on this value. If an priori lower bound on the margin γ is known, we can fix the value of η so as to minimize the loss bound of Mu-PRank. The resulting bound is described in corollary 2, which follows theorem 4. Theorem 4 (mistake bound). Let (x¯ 1 , y1 ), . . . , (x¯ T , yT ) be an input sequence to Mu-PRank, where x¯ t ∈ Rn , x¯ t ∞ ≤ 1 , and yt ∈ {1, . . . , k}. Assume that there exists a ranking rule v¯ ∗ = (w¯ ∗ , b¯∗ ) with b∗1 ≤ · · · ≤ b∗k−1 and w¯ ∗ , b¯∗ 1 = 1, which classifies the entire sequence correctly with margin γ = minr,t {(w¯ ∗ ·x¯ t −b∗r )ytr } > 0.
Then the ranking loss of Mu-PRank, Tt=1 |yˆ t − yt | is, at most,   log k + n − 1   . 2 log eη(k−1) +e + ηγ −η(k−1) Proof. As before, let v¯ t = (w¯ t , b¯t ) and v¯ t+1 = (w¯ t+1 , b¯t+1 ) denote the ranking rules at the beginning and end of round t, respectively. To prove the theorem, we analyze the decrease in the Kullback-Leibler (KL) divergence (Cover & Thomas, 1991) between v¯ t and v¯ ∗ . The KL
divergence of two dis q is DKL (p  q) = i pi log(pi /qi ). To prove crete probability distributions p, the theorem, we examine the change of the KL divergence in two consecutive rounds: t = DKL (v¯ ∗ v¯ t+1 ) − DKL (v¯ ∗ v¯ t ). We bound




t t from above and below. We first bound this sum from above.

166

K. Crammer and Y. Singer

As a reminder, we use nt to denote |yˆ t − yt |. Using this notation, we get n 

t =

w∗i log



i=1 n 

=

w∗i log



=

w∗i

+

i=1

 +

Zt
t

k−1 

k−1 

b∗r log

r=1

eηxi

i=1

n 

w∗i wti

b∗r

 +

τt r r

k−1 

b∗r btr



b∗r log

r=1





Zt e−ητr

t

k−1    τrt (w¯ ∗ · xt − b∗r ) log Z t − η

r=1

≤ log(Z t ) − ηγ



r=1

k−1 

|τrt | = log(Z t ) − ηγ nt ,

(5.1)

r=1

where we used the definition of nt and τrt in conjunction with the fact that v¯ ∗ achieves a margin of γ to obtain the inequality above. We now bound log(Z t ). We need the following inequality, a + x ηa a − x −ηa e + e , 2a 2a

eηx ≤

(5.2)

which holds for a, η > 0 and x ∈ [−a, a]. (The proof of this inequality is an immediate application of the convexity of the exponent function.) Now recall that

Zt =

n 

wti eηxi

t

i=1




τt r r

+

k−1 

btr e−ητr . t

(5.3)

r=1

We bound the left-hand side and the right-hand side of the
above sum sep-

arately. For the left-hand side, we bound each term eηxi
r τr . Using equation 5.2 in conjunction with the fact that |xti | ≤ 1 and | r τrt | ≤ k − 1, we get t

eηxi

t




τt r r

≤

t



k − 1 + xti r τrt η(k−1) k − 1 − xti r τrt −η(k−1) + . e e 2(k − 1) 2(k − 1)

(5.4)

Similarly, |τrt | ≤ 1 ≤ k − 1, and thus eητr ≤ t

k − 1 + τrt η(k−1) k − 1 − τrt −η(k−1) + . e e 2(k − 1) 2(k − 1)

Using the bounds from equations 5.4 and 5.5 in 5.3, we get

Z ≤ t

 i

 wti



 k − 1 + xti r τrt η(k−1) k − 1 − xti r τrt −η(k−1) + e e 2(k − 1) 2(k − 1)

(5.5)

Online Ranking by Projecting

+

 r

=

 i

+

 btr

167

k − 1 + τrt η(k−1) k − 1 − τrt −η(k−1) + e e 2(k − 1) 2(k − 1)



 1 1 wti (eη(k−1) + e−η(k−1) ) + btr (eη(k−1) + e−η(k−1) ) 2 2 r



τrt (w¯ t · xt − btr )

r

eη(k−1) + e−η(k−1) . 2(k − 1)

(5.6)

The definition of τrt implies that τrt (w¯ t ·xt −btr ) ≤ 0. (Either there was a ranking error or τrt = 0). In addition, since Mu-PRank normalizes its ranking rule at

the end of each round, we know that (w¯ t , b¯t )1 = i wti + r btr = 1. Using these facts in equation 5.6, we get the following bound on Z t : 1 Z t ≤ (eη(k−1) + e−η(k−1) ). 2

(5.7)

Using equation 5.7 in equation 5.1, we obtain an upper bound on t : 

 1 η(k−1) −η(k−1) +e ) − ηγ nt . t ≤ log (e 2 Since a ranking error occurred, we know that nt ≥ 1. In addition, the argument of the logarithm is at least 1; we can rearrange terms and write     1 η(k−1) t −η(k−1) t ≤ n log (e +e ) − ηγ . (5.8) 2 Summing over t, we get       1 η(k−1) t ≤ log + e−η(k−1) − ηγ nt . e 2 t t

(5.9)


To bound t from below, we unravel the sum and get   t = (DKL (v¯ ∗ v¯ t+1 ) − DKL (v¯ ∗ v¯ t )) t

t

= DKL (v¯ ∗ v¯ T+1 ) − DKL (v¯ ∗ v¯ 1 ) ≥ −DKL (v¯ ∗ v¯ 1 ), where the inequality is due to the fact that the KL divergence is always ¯ 1 , we get that DKL (v¯ ∗ v¯ 1 ) = log(k − 1 + n). nonnegative. Using the value of w
Combining the lower bound on t t with equation 5.9, we get 



2 log(k − 1 + n) ≥ log η(k−1) e + e−η(k−1)

 + ηγ

 t

nt .

168

K. Crammer and Y. Singer

Rearranging terms, we finally get the desired bound:  t

  log k − 1 + n   . n ≤ 2 + ηγ log eη(k−1) +e −η(k−1) t

As discussed above, the bound of theorem 4 depends on the learning rate η. If γ or a lower bound on γ is known, we can set η to be   1 k−1+γ η= log . 2(k − 1) k−1−γ For this choice of η, we get the following corollary: Corollary 2. η=

If we run Mu-PRank with

  k−1+γ 1 log , 2(k − 1) k−1−γ

then under the assumptions of theorem 4, the cumulative ranking loss obtained by Mu-PRank is bounded above by 

t

2 log

n ≤ (k − 1)

t

  n+k−1 . γ2

This corollary implies that the cumulative ranking loss of Mu-PRank is inversely proportional to square of the margin γ . Note that a direct comparison of this bound to the mistake bound of PRank (see theorem 1) is not possible as the notion of margin here is different. (For PRank, we used the 2 norm of the instances and the ranking rule to define the margin, while for MuPRank, we tacitly use the 1 norm of the ranking rule in conjunction with the ∞ of the instances.) This relation between the bounds also holds between additive and multiplicative online algorithms for classification (Rosenblatt, 1958; Littlestone, 1987, 1988, 1989). Putting these differences in the notion of margin aside, the two bounds exhibit the same quadratic dependency on the margin. 6 Experiments In this section, we describe experiments we performed that compared the variants of PRank discussed in the previous section with two other online learning algorithms applied to ranking: a multiclass generalization of the Perceptron algorithm (Crammer & Singer, 2001b), denoted MCP, and the Widrow-Hoff algorithm (Widrow & Hoff, 1960) for online regression learning, which we denote by WH. For WH we fixed its learning rate to a

Online Ranking by Projecting

169

constant value. The hypotheses that the variants of PRank and the two other algorithms maintain share similarities but are different in their complexity: PRank maintains a vector w¯ of dimension n and a vector of k − 1 modifiable ¯ totaling n + k − 1 parameters; MCP maintains k prototypes, thresholds b, which are vectors of dimension n, yielding kn parameters; WH maintains a single vector w¯ of size n. Therefore, MCP builds the most complex hypothesis, while WH builds the simplest. We describe two sets of experiments with two different data sets. The data set used in the first experiment is synthetic and was generated in a similar way to the data set used by Herbrich et al. (2000). We first generated points x¯ = (x1 , x2 ) uniformly at random from the unit square [0, 1]2 . Each point was assigned a rank y from the set {1, . . . , 5} according to the following ranking rule, y = maxr {r : 10((x1 − 0.5)(x2 − 0.5)) + ξ > br } where b¯ = (−∞, −1, −0.1, 0.25, 1) and ξ is a normally distributed noise with a zero mean and a standard deviation of 0.125. We generated sequences of instance rank pairs, each of length 8000. We fed the sequences to PRank, SiPRank, MCP, and WH. We then obtained predictions for each instance. We converted the real-valued predictions of WH into ranks by rounding each prediction to its closest rank value. As in Herbrich et al. (2000), we used a nonhomogeneous polynomial of degree 2, K(x¯ 1 , x¯ 2 ) = ((x¯ 1 · x¯ 2 ) + 1)2 as the inner product operation between each input instance and the hyperplanes that the four additive algorithms maintain. At each time step, we computed for each algorithm the accumulative ranking loss normalized by the instantaneous sequence length. Formally, the time-averaged loss after T rounds is
(1/T) Tt |yˆ t − yt |. We computed the loss for each algorithm we tested for T = 1, . . . , 8000. To increase the statistical significance of the results, we repeated the process 100 times, picking a new random instance rank sequence of length 8000 each time, and then averaged the instantaneous losses across the 100 runs. The results are depicted on the left-hand side of Figure 6. The size of the symbols in the plot is larger than 95% confidence intervals for each result depicted in the figure. In this experiment, the performance of MCP is constantly worse than the performance of WH and PRank. WH initially suffers the smallest instantaneous loss, but after about 500 rounds, both PRank and Si-PRank start to outperform MCP and WH. Eventually the ranking loss that PRank and Si-PRank suffer is significantly lower than both WH and MCP. In the second set of experiments, we used the EachMovie data set (McJones, 1997). This data set is used for collaborative filtering tasks and contains ratings of movies provided by 61,265 people. Each person in the data set viewed a subset of movies from a collection of 1623 titles. Each viewer rated each movie that she saw using one of six possible ratings: 0, 0.2, 0.4, 0.6, 0.8, 1. We chose subsets of people who viewed a significant number of movies, extracting for evaluation people who have rated at least 100 movies. There were 7542 such viewers. We chose at random one per-

170

K. Crammer and Y. Singer

1.1

PRank Si−PRank WH MC−Perceptron

1 0.9

Rank Loss

0.8 0.7 0.6 0.5 0.4 0.3 0

1000

2000

3000

4000

5000

6000

7000

8000

Round

Figure 6: Comparison of the time-averaged ranking loss of PRank, Si-PRank, WH, and MCP on synthetic data.

son among these viewers and set the person’s ratings to be the target rank. We used the ratings of the other viewers as features. Thus, the goal is to learn to predict the “taste” of a random user with the user’s past ratings serving as a feedback and the ratings of fellow viewers as features. The prediction rule associates a weight with each fellow viewer and therefore can be seen as learning correlations between the tastes of different viewers. Next, we subtracted 0.5 from each rating; therefore the possible rating values are −0.5, −0.3, −0.1, 0.1, 0.3, 0.5. This linear transformation enabled us to assign a value of zero to movies that have not been rated. We fed each pair of feature and rank level in an online fashion. Since we chose viewers who rated at least 100 movies, we were able to perform at least 100 rounds of online predictions and updates. We repeated this experiment 500 times, each time choosing a random viewer as the target rank. The results are given in the left-hand plots of Figure 7. The error bars in the plot indicate 95% confidence levels. We repeated the experiment using viewers who have seen at least 200 movies. (There were 1802 such viewers.) The results of this experiment are given on the right-hand plots of Figure 7. The plots at the top of the figure compare the additive algorithms: PRank, Si-PRank, MCP, and WH. Along the entire run of the algorithms, PRank and Si-PRank are significantly

Online Ranking by Projecting 2.4

171 2.2

PRank Si−PRank WH MC−Perceptron

2.2

1.8 Rank Loss

Rank Loss

2 1.8 1.6

1.6 1.4

1.4

1.2

1.2

1

1 0

PRank Si−PRank WH MC−Perceptron

2

20

40

60

80

0.8 0

100

50

Round

1.9

1.7

200

Si−PRank η 0.50 η 1.00 η 4.00 η 16.00

1.7 1.6 1.5

1.6

Rank Loss

Rank Loss

150

1.8

Si−PRank η 0.50 η 1.00 η 4.00 η 16.00

1.8

1.5 1.4

1.4 1.3 1.2

1.3

1.1

1.2 1.1 0

100 Round

1

20

40

60 Round

80

100

0.9 0

50

100

150

200

Round

Figure 7: Comparison of the time-averaged ranking loss of the variants of PRank, WH, and MCP on the EachMovie data set using viewers who rated at least 200 movies (top) and at least 100 movies (bottom).

better than WH and consistently better than the multiclass perceptron algorithm, although the last employs a hypothesis that is substantially bigger. Comparing the two additive variants of PRank, we see that the ranking loss of Si-PRank is slightly lower than that of PRank. The plots at the bottom of the figure compare the best of the additive algorithms, Si-PRank, with the Mu-PRank run with different learning rates (η = 0.5, 1, 4, 16). One of the goals of this experiment is to check the dependency of the performance of Mu-PRank on its learning rate. It is clear from the two bottom plots of Figure 7 that the performance of Mu-PRank is sensitive to the choice of the learning rate. Note, though, that the best learning rate is problem dependent: the best learning rate is η = 4 for the first partition of EachMovie, while η = 0.5 results in the smallest ranking loss in the case of the second partition. This type of behavior is also exhibited by multiplicative algorithms in other problems such as binary classification (Kivinen & Warmuth, 1997). Nonetheless, Mu-PRank outperforms most of the other algorithms we compared for a broad range

172

K. Crammer and Y. Singer

of learning rates. Focusing first on the left plot, we observe that after about 60 rounds, Si-PRank achieves the best performance, while at the start of the training process, its ranking loss is almost inferior to most of the copies of Mu-PRank. Summing up, both the additive and multiplicative versions of PRank outperform regression and classification algorithms when evaluated with respect to the ranking loss. While the superior performance of PRank is not surprising, it provides an empirical validation of the formal analysis presented in the previous sections. 7 Conclusion In this article, we described a family of algorithms for instance ranking. The roots of the algorithms go back the perceptron algorithm (Rosenblatt, 1958). One of the major results in the article is the description of a new approach for solving ranking problems. While most of the previous approaches reduce the problem of ranking to a classification of pairs, we presented an alternative approach that builds a connection between rank levels and subintervals of the reals. An open problem that arises from both the theoretical analysis and the empirical results is deciding what version of PRank to use. In particular, PRank and Si-PRank share the same mistake bound, while in our experiments, Si-PRank performed better than PRank. All of the versions of PRank presented in this article are online algorithms, and for their analysis we used the mistake-bound model. An interesting research direction is the design and analysis of algorithms for batch settings in which all of the training examples are given at once. As mentioned in section 1, Shashua and Levin (2002) have described a batch algorithm for ranking. An interesting question is how the different variants relate to the algorithm of Shashua and Levin. In addition to continuing our research on online algorithms for ranking problems, an interesting research direction is the design and generalization analysis of batch algorithms for various ranking losses. Appendix: Technical Proofs A.1 Proof of Theorem 2. To prove the theorem, we introduce a slight modification of theorem 1 by relaxing the assumption that v¯ ∗ is of a unit norm and adding the norm of v¯ ∗ to the mistake bound of the basic PRank algorithm. We thus write the mistake bound of theorem 1 as T  t=1

|yˆ t − yt | ≤ (k − 1)

(R2 + 1)v¯ ∗ 2 . γ2

Since the case D = 0 reduces to setting of theorem 1, we can assume that D > 0. We prove the theorem by transforming the inseparable problem into a separable one. We do so by expanding each original instance x¯ t ∈ Rn into

Online Ranking by Projecting

173

a vector z¯ t ∈ Rn+T as follows. The first n coordinates of z¯ t are set to x¯ t . The n + t coordinate of z¯ t is set to , which is a positive real number whose value is set below. The rest of the coordinates of z¯ t are set to zero. We similarly extend the ranking rule (w¯ ∗ , b¯∗ ) to (u¯ ∗ , c¯∗ ) ∈ R(n+T) × Rk−1 as follows. We rewrite the value of dt from equation 3.11 as   dt = max γ − min{(w¯ ∗ · x¯ t − b∗r )ytr }, 0, γ − min{(w¯ ∗ · x¯ t − b∗r )ytr } , (A.1) r≥yt

r
and define st to be the following indicator function:  −1 0 st =  +1

dt = γ − minr≥yt {(w¯ ∗ · x¯ t − b∗r )ytr } dt = 0 . dt = γ − minr
(A.2)

Note that if st = −1, then dt = (w¯ ∗ · x¯ t − b∗r )ytr for some r ≥ yt , and thus ytr = −1. Similarly, if st = +1, then dt = (w¯ ∗ · x¯ t − b∗r )ytr for some r < yt and ytr = +1. We set the first n columns u¯ ∗ to be w¯ ∗ , and the n + t coordinate of t t u¯ ∗ is set to be sd and the rest of the coordinates of u¯ ∗ are set to zero. We now show that (u¯ ∗ , c¯∗ ) achieves a margin value γ on the expanded sequence. Using the definition of st and dt , we get st dt  − b∗r )ytr  = (w¯ ∗ · x¯ t − b∗r )ytr + st dt ytr

(u¯ ∗ · z¯ t − c∗r )ytr = (w¯ ∗ · x¯ t +

= (w¯ ∗ · x¯ t − b∗r )ytr + dt

≥ (w¯ ∗ · x¯ t − b∗r )ytr + γ − (w¯ ∗ · x¯ t − b∗r )ytr

= γ.

(A.3)

Note that by construction, ¯zt 2 ≤ R2 + 2 . Since the norm of (w¯ ∗ , b¯∗ ) is 1, we have u¯  + ¯c  = w¯  + b¯∗ 2 + ∗ 2

∗ 2

∗ 2

  st dt 2 t



=1+

D2 . 2

We now apply the bound of theorem 1 in the form discussed above and get that T  t=1

|yˆ t − yt | ≤ (k − 1)

(R2 + 2 + 1)(1 + γ2

D2 ) 2

.

(A.4)

√ Setting 2 = D R2 + 1 in the equation above yields the desired bound.

174

K. Crammer and Y. Singer

It remains to show that the predictions of the algorithm on each element of the original sequence and on the expanded sequences are identical. This part follows exactly the same line of proof used in theorem 1 from Freund and Schapire (1998) and is thus omitted. Acknowledgments Thanks to Sanjoy Dasgupta and Rob Schapire for numerous discussions on ranking problems and algorithms. Thanks also to Eleazar Eskin and Uri Maoz for carefully reading the manuscript. This work was supported in part by the IST Programme of the European Community, under the PASCAL Network of Excellence, IST-2002-506778. This publication reflects only the authors’ views. References Cohen, W., Schapire, R. E., & Singer, Y. (1999). Learning to order things. Journal of Artificial Intelligence Research, 10, 243–270. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley. Crammer, K., & Singer, Y. (2001a). Pranking with ranking. In T. G. Dietterich, S. Becker, & Z. Ghahramani (Eds.), Advances in neural information processing systems, 14. Cambridge, MA: MIT Press. Crammer, K., & Singer, Y. (2001b). Ultraconservative online algorithms for multiclass problems. In Proceedings of the Fourteenth Annual Conference on Computational Learning Theory. Amsterdam: Springer. Cristianini, N., & Shawe-Taylor, J. (2000). An introduction to support vector machines. Cambridge: Cambridge University Press. Fletcher, R. (1987). Practical methods of optimization (2nd ed.). New York: Wiley. Freund, Y., Iyer, R., Schapire, R. E., & Singer, Y. (2003). An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4, 933–969. Freund, Y., & Schapire, R. E. (1999). Large margin classification using the perceptron algorithm. Machine Learning, 37, 277–296. Harrington, E. F. (2003). Online ranking/collaborative filtering using the perceptron algorithm. In Proceedings of the Twentieth International Conference on Machine Learning. Washington, DC: AIII Press. Herbrich, R., Graepel, T., & Obermayer, K. (2000). Large marging rank boundaries for ordinal regression. In A. Smola, B. Scholkopf, ¨ & D. Schuurmans (Eds.), Advances in large margin classifiers. Cambridge, MA: MIT Press. Kemeny, J. G., & Snell, J. L. (1962). Mathematical models in the social sciences. Cambridge, MA: MIT Press. Kivinen, J., & Warmuth, M. K. (1997). Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132(1), 1–64. Littlestone, N. (1987). Learning when irrelevant attributes abound. In 28th Annual Symposium on Foundations of Computer Science (pp. 68–77). Los Angeles: IEEE.

Online Ranking by Projecting

175

Littlestone, N. (1988). Learning when irrelevant attributes abound: A new linearthreshold algorithm. Machine Learning, 2, 285–318. Littlestone, N. (1989). Mistake bounds and logarithmic linear-threshold learning algorithms. Unpublished doctoral dissertation, University of California, Santa Cruz. McJones, P. (1997). Eachmovie collaborative filtering data set. DEC Systems Research Center. Available online at: http://www.research.digital.com/SRC/ eachmovie/. Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65, 386–407. Shashua, A., & Levin, A. (2002). Ranking with large margin principle: Two approaches. In S. Becker, S. Thrun, ¨ & K. Obermayer (Eds.), Advances in neural information processing systems, 15. Cambridge, MA: MIT Press. Vapnik, V. N. (1998). Statistical learning theory. New York: Wiley. Widrow, B., & Hoff, M. E. (1960). Adaptive switching circuits. 1960 IRE WESCON Convention Record, 96–104. Received October 31, 2003; accepted June 1, 2004.

LETTER

Communicated by John Shawe-Taylor

On Learning Vector-Valued Functions Charles A. Micchelli charles [email protected] Department of Mathematics and Statistics, State University of New York, University at Albany, Albany, NY, 12222, U.S.A.

Massimiliano Pontil [email protected] Department of Computer Sciences, University College London, London WC1E, England, UK

In this letter, we provide a study of learning in a Hilbert space of vectorvalued functions. We motivate the need for extending learning theory of scalar-valued functions by practical considerations and establish some basic results for learning vector-valued functions that should prove useful in applications. Specifically, we allow an output space Y to be a Hilbert space, and we consider a reproducing kernel Hilbert space of functions whose values lie in Y . In this setting, we derive the form of the minimal norm interpolant to a finite set of data and apply it to study some regularization functionals that are important in learning theory. We consider specific examples of such functionals corresponding to multiple-output regularization networks and support vector machines, for both regression and classification. Finally, we provide classes of operator-valued kernels of the dot product and translation-invariant type. 1 Introduction The problem of computing a function from empirical data is addressed in several areas of mathematics and engineering. Depending on the context, this problem goes under the name of function estimation (statistics), function learning (machine learning theory), and function approximation and interpolation (approximation theory), among others. The type of functions typically studied are real-valued functions (in learning theory, the related binary classification problem is often treated as a special case). There is a large literature on the subject. We recommend Cherkassky and Mulier (1998), Cristianini and Shawe-Taylor (2000), Evgeniou, Pontil, and Poggio (2000), Vapnik (1998), Wahba (1990). In this work, we address the problem of computing functions whose range is in a Hilbert space Y , discuss ideas within the perspective of learnNeural Computation 17, 177–204 (2005)

c 2004 Massachusetts Institute of Technology


178

C. Micchelli and M. Pontil

ing theory, and elaborate on their connections to interpolation and optimal estimation. Despite its importance, learning vector-values functions has been only marginally studied within the learning theory community, and this article is a first attempt to set down a framework to study this problem. We focus on Hilbert spaces of vector-valued functions that admit a reproducing kernel (Aronszajn, 1950). In the scalar case, these spaces have received considerable attention over the past few years in machine learning theory due to the successful application of kernel-based learning methods to complex data, including images, text data, speech data, and biological data (see, e.g., Cristianini & Shawe-Taylor, 2000; Scholkopf ¨ & Smola, 2002; Vapnik, 1998). We briefly motivate learning vector-valued functions when Y is an ndimensional Euclidean space. One straightforward approach consists in separately representing each component of a vector-valued function f = ( f1 , . . . , fn ) by a linear space of smooth functions and then learn these components independently, for example, by minimizing some regularized error functional. This approach does not capture relations between components of f and so will be suboptimal when these relations occur. For example, if the components of f represent similar classification tasks, we can benefit by learning the tasks simultaneously (see, e.g., Evgeniou & Pontil, 2004). Similarly, if f maps to a space of images, the components of f are associated with pixels that may be related. In such cases, we should introduce a measure of smoothness of f that models relations among its components. We propose to do this by using a matrix-valued kernel K: X × X → Rn×n that reflects the interaction among the components of f . This letter provides a foundation for this approach. Vector-valued learning offers both practical and theoretical challenges. We establish some basic insights into kernel-based vector-valued learning as well as indications where this may be valuable for applications. In section 2, we outline the theory of reproducing kernel Hilbert spaces (RKHS) of vector-valued functions. These RKHSs admit a kernel with values that are bounded linear operators on Y . They have been studied by Burbea and Masani (1994), but only in the context of complex analysis and used recently for the solution of partial differential equations by Amodei (1997). Section 3 treats the problem of minimal norm interpolation (MNI) in the context of RKHS. MNI plays a central role in many approaches to function estimation, and so we highlight its relation to learning. In particular, in section 4, we use MNI to resolve the form of the minimizer of regularization functionals in the context of vector-valued functions. In section 5, we discuss the form of operator-valued kernels that are either of the dot product or translationinvariant form as they are often used in learning. Finally, in section 6, we describe examples where we feel there is practical need for vector-valued learning as well as report on numerical experiments that highlight the advantages of the proposed methods.

On Learning Vector-Valued Functions

179

2 Reproducing Kernel Hilbert Spaces of Vector-Valued Functions Let Y be a real Hilbert space with inner product (·, ·), X a set, and H a linear space of functions on X with values in Y . We assume that H is also a Hilbert space with inner product ·, ·. Definition 1. We say that H is a reproducing kernel Hilbert space (RKHS) when for any y ∈ Y and x ∈ X the linear functional that maps f ∈ H to (y, f (x)) is continuous. In this case, according to the Riesz lemma (see, e.g., Akhiezer & Glazman, 1993), there is, for every x ∈ X and y ∈ Y , a function K(x|y) ∈ H such that for all f ∈ H, (y, f (x)) = K(x|y), f . Since K(x|y) is linear in y, we write K(x|y) = Kx y where Kx : Y → H is a linear operator. The above equation can be now rewritten as (y, f (x)) = Kx y, f .

(2.1)

For every x, t ∈ X , we also introduce the linear operator K(x, t): Y → Y defined, for every y ∈ Y , by K(x, t)y := (Kt y)(x).

(2.2)

We say that H is normal provided there does not exist (x, y) ∈ X × (Y \{0}) such that the linear functional (y, f (x)) = 0 for all f ∈ H. In proposition 1, we state the main properties of the function K. To this end, we let L(Y ) be the set of all bounded linear operators from Y into itself, and for every A ∈ L(Y ), we denote by A∗ its adjoint. We also use L+ (Y ) to denote the cone of nonnegative bounded linear operators, that is, A ∈ L+ (Y ) provided that, for every y ∈ Y , (y, Ay) ≥ 0. When this inequality is strict for all y = 0, we say A is positive definite. Finally, we denote by Nm the set of positive integers up to and including m. Proposition 1. If K(x, t) is defined, for every x, t ∈ X, by equation 2.2 and Kx is given by equation 2.1, the kernel K satisfies, for every x, t ∈ X , the following properties: a. For every y, z ∈ Y , we have that (y, K(x, t)z) = Kt z, Kx y

(2.3)

b. K(x, t) ∈ L(Y ), K(x, t) = K(t, x)∗ , and K(x, x) ∈ L+ (Y ). Moreover, K(x, x) is positive definite for all x ∈ X if and only if H is normal.

180

C. Micchelli and M. Pontil

c. For any m ∈ N, {xj : j ∈ Nm } ⊆ X , {yj : j ∈ Nm } ⊆ Y we have that


(yj , K(xj , x )y ) ≥ 0.

(2.4)

j,∈Nm 1

d. Kx  = K(x, x) 2 . 1

1

e. K(x, t) ≤ K(x, x) 2 K(t, t) 2 . f. For every f ∈ H and x ∈ X , we have that 1

 f (x) ≤  f K(x, x) 2 . Proof. We prove property a in proposition 1 by merely choosing f = Kt z in equation 2.1 to obtain that Kx y, Kt z = (y, (Kt z)(x)) = (y, K(x, t)z). Consequently, from this equation, we conclude that K(x, t) admits an algebraic adjoint K(t, x) defined everywhere on Y , and so the uniform boundness principle (Akhiezer & Glazman, 1993) implies that K(x, t) ∈ L(Y ), and K(x, t) = K(t, x)∗ . Moreover, choosing t = x in equation 2.3 proves that K(x, x) ∈ L+ (Y ). As for the positive definiteness of K(x, x), merely use equations 2.1 and 2.3. These remarks prove property b. As for property c, we again use equation 2.3 to get
j,∈Nm

(yj , K(xj , x )y ) =




Kxj yj , Kx y  = 

j,∈Nm




Kxj yj 2 ≥ 0.

j∈Nm

For the proof of property d, we choose y ∈ Y and observe that Kx y2 = (y, K(x, x)y) ≤ yK(x, x)y ≤ y2 K(x, x), 1

which implies that Kx  ≤ K(x, x) 2 . Similarly, we have that K(x, x)y2 = (K(x, x)y, K(x, x)y) = Kx K(x, x)y, Kx y ≤ Kx yKx K(x, x)y ≤ yKx 2 K(x, x)y, thereby implying that Kx 2 ≥ K(x, x), which proves property d. For the claim e, we compute K(x, t)y2 = (K(x, t)y, K(x, t)y) = Kx K(x, t)y, Kt y ≤ Kx K(x, t)yKt y ≤ yKt Kx K(x, t)y,

On Learning Vector-Valued Functions

181

which gives K(x, t)y ≤ Kx Kt y and establishes property e. For our final assertion, we observe, for all y ∈ Y , that (y, f (x)) = Kx y, f  ≤  f Kx y ≤  f yKx , which implies the desired result:  f (x) ≤  f Kx . Definition 2. We say that K: X × X → L(Y ) is a kernel if it satisfies properties b and c in proposition 1. So far we have seen that if H is a RKHS of vector-valued functions, there exists a kernel. In the spirit of Moore-Aronszajn’s theorem for RKHS of scalar functions (Aronszajn, 1950), it can be shown that a kernel determines an RKHS of vector-valued functions. We state the theorem below; however, as the proof parallels the scalar case, we do not elaborate on the details. Theorem 1. If K: X × X → L(Y ) is a kernel, then there exists a unique (up to an isometry) RKHS, which admits K as the reproducing kernel. Let us observe that in the case Y = Rn , the kernel K is an n × n matrix of scalar-valued functions. The elements of this matrix can be identified by appealing to equation 2.3. Indeed, choosing y = ek , and z = e , k,  ∈ Nn , these being the standard coordinate bases for Rn , yields the formula (K(x, t))k = Kx ek , Kt e .

(2.5)

In particular, when n = 1, equation 2.1 becomes f (x) = Kx , f , which is the standard reproducing kernel property, while equation 2.3 reads K(x, t) = Kx , Kt . The case Y = Rn serves also to illustrate some features of the operatorvalued kernels that are not present in the scalar case. In particular, let H ,  ∈ Nn be RKHS of scalar-valued functions on X with kernels K ,  ∈ Nn and define the kernel whose values are n × n matrices by the formula D = diag(K1 , . . . , Kn ). Clearly, if { f :  ∈ Nn }, {g :  ∈ Nn } ⊆ H, we have that
 f, g =  f , g  , ∈Nn

182

C. Micchelli and M. Pontil

where ·, · is the inner product in the RKHS of scalar functions with kernel K . Diagonal kernels can be effectively used to generate a wide variety of operator-valued kernels that have the flexibility needed for learning. We have in mind the following construction. For every set {Aj : j ∈ Nm } of r × n matrices and {Dj : j ∈ Nm } of diagonal kernels, the operator-valued function K(x, t) =




Aj∗ Dj (x, t)Aj , x, t ∈ X

(2.6)

j∈Nm

is a kernel. We conjecture that all operator-valued kernels are limits of kernels of this type. Generally, the kernel in equation 2.6 cannot be diagonalized, that is, it cannot be rewritten in the form A∗ DA, unless all the matrices Aj , j ∈ Nm can be transformed into a diagonal matrix by the same matrix. For r much smaller than n, equation 2.6 results in low-rank kernels, which should be an effective tool for learning. In particular, in many practical situations, the components of f may be linearly related, that is, for every x ∈ X , f (x) lies on a linear subspace M ⊆ Y . In this case, it is desirable to use a kernel that has the property that f (x) ∈ M, x ∈ X for all f ∈ H. An elegant solution to this problem is to use a low-rank kernel modeled by the output examples themselves, namely,
K(x, t) = λj yj Kj (x, t)yj∗ , j∈Nm

where λj are nonnegative constants and Kj , j ∈ Nm are some prescribed scalar-valued kernels. Property c of proposition 1 has an interesting interpretation concerning the RKHS of scalar-valued functions. Every f ∈ H determines a function F on X × Y defined by F(x, y) := (y, f (x)), x ∈ X , y ∈ Y .

(2.7)

We let H1 be the linear space of all such functions. Thus, H1 consists of functions that are linear in their second variable. We make H1 into a Hilbert space by choosing F =  f . It then follows that H1 is an RKHS with a reproducing scalar-valued kernel defined, for all (x, y), (t, z) ∈ X × Y , by the formula K1 ((x, y), (t, z)) = (y, K(x, t)z). This idea is known in the statistical context (e.g., Cressie, 1993), and can be extended in the following manner. We define, for any p ∈ Nn , the family of functions Kp ((x, y), (t, z)) := (y, K(x, t)z)p , (x, y), (t, z) ∈ X × Y .

On Learning Vector-Valued Functions

183

The lemma of Schur (see, e.g., Aronszajn, 1950) implies that Kp is a scalar kernel on X × Y . The functions in the associated RKHS consist of scalar-valued functions that are homogeneous polynomials of degree p in their second argument. These spaces may be of practical value for learning polynomial functions of projections of vector-valued functions. A kernel K can be realized by a mapping : X → L(W , Y ) where W is a Hilbert space by the formula K(x, t) = (x)∗ (t), x, t ∈ X , and functions f in H can be represented as f = w for some w ∈ W , so that  f H = wW . Moreover, when H is separable, we can choose W to be the separable Hilbert space of square summable sequences (Akhiezer & Glazman, 1993). In the scalar case, Y = R,  is referred to in learning theory as the feature map, and it is central in developing kernel-based learning algorithms (see, e.g., Cristianini & Shawe-Taylor, 2000; Scholkopf ¨ & Smola, 2002). 3 Minimal Norm Interpolation In this section, we turn to the problem of minimal norm interpolation of vector-valued functions within the RKHS framework already developed. This problem consists in finding, among all functions in H that interpolate a given set of points, a function with minimum norm. We will see later that the minimal norm interpolation problem plays a central role in characterizing regularization approaches to learning. Definition 3. For distinct points {xj : j ∈ Nm }, we say that the linear functionals defined for f ∈ H as Lxj f := f (xj ), j ∈ Nm are linearly independent if and only if there does not exist {cj : j ∈ Nm } ⊆ Y (not all zero) such that for all f ∈ H,
(cj , f (xj )) = 0. (3.1) j∈Nm

Lemma 1. The functionals {Lxj : j ∈ Nm } are linearly independent if and only if for any {yj : j ∈ Nm } ⊆ Y , there are unique {cj : j ∈ Nm } ⊆ Y such that
K(xj , x )c = yj , j ∈ Nm . (3.2) ∈Nm

Proof. We denote by Y m the mth Cartesian product of Y . We make Y m into a Hilbert space by defining for every c = (cj : j∈ Nm ) ∈ Y m and d = (dj : j ∈ Nm ) ∈ Y m their inner product c, d := j∈Nm (cj , dj ). Let us consider the bounded linear operator A: H → Y m , defined for f ∈ H by Af = ( f (xj ) : j ∈ Nm ).

184

C. Micchelli and M. Pontil

Therefore, by construction, c satisfies equation 3.1 when c ∈ Ker(A∗ ) and, hence, the set of linear functionals {Lxj : j ∈ Nm } is linearly independent if and only if Ker(A∗ ) = {0}. Since Ker(A∗ ) = Ran(A)⊥ (see Akhiezer & Glazman, 1993), this is equivalent to the condition that Ran(A) = Y m . From the reproducing kernel property, we can identify A∗ , by computing for c ∈ Y m and f ∈ H: c, Af  =




(cj , f (xj )) =

j∈Nm




Kxj cj , f .

j∈Nm

 Thus, we have established that A∗ c = j∈Nm Kxj cj . We now consider the symmetric bounded linear operators B := AA∗ : Y m → Y m , which we identify for c = (cj : j ∈ Nm ) ∈ Y m as  Bc = 




 K(x , xj )cj :  ∈ Nm  .

j∈Nm

Consequently, equation 3.2 can be equivalently written as Bc = y, and so this equation means that y ∈ Ran(B) = Ker(A∗ )⊥ . Since it can be verified that Ker(A∗ ) = Ker(B), we have shown that Ker(A∗ ) = {0} if and only if the linear functionals {Lxj : j ∈ Nm } are linearly independent and the result follows. Theorem 2. If the linear functionals Lxj f = f (xj ), f ∈ H, j ∈ Nm are linearly independent, then the unique solution to the variational problem min{ f 2 : f (xj ) = yj , j ∈ Nm }

(3.3)

is given by fˆ =




Kxj cj ,

j∈Nm

where {cj : j ∈ Nm } ⊆ Y is the unique solution of the linear system of equations


K(xj , x )c = yj , j ∈ Nm .

(3.4)

∈Nm

Proof. Let f be any element of H such that f (xj ) = yj , j ∈ Nm . This function always exists since we have shown that the operator A maps onto Y m . We set g := f − fˆ and observe that  f 2 = g + fˆ2 = g2 + 2 fˆ, g +  fˆ2 .

On Learning Vector-Valued Functions

185

However, since g(xj ) = 0, j ∈ Nm , we obtain that


 fˆ, g =

j∈Nm

Kxj cj , g =




(cj , g(xj )) = 0.

j∈Nm

It follows that  f 2 = g2 +  fˆ2 ≥  fˆ2 , and we conclude that fˆ is the unique solution to equation 3.3. When the set of linear functionals {Lxj : j ∈ Nm } is dependent, generally data y ∈ Y m may not admit an interpolant. Thus, the variational problem in theorem 2 requires that y ∈ Ran(A). Since Ran(A) = Ker(A∗ )⊥ = Ker(B)⊥ = Ran(B), we see that if y ∈ Ran(A), that is, when the data admit an interpolant in H, then the system of equations 3.2 has a (generally not unique) solution, and so the function in equation 3.4 is still the solution of the extremal problem. Hence, we have proved the following result. Theorem 3. If y ∈ Ran(A) the minimum of problem 3.3 is unique and admits  the form fˆ = j∈Nm Kxj cj , where the coefficients {cj : j ∈ Nm } solve the linear system of equations,


K(xj , x )c = yj , j ∈ Nm .

∈Nm

An alternative approach to proving the above result is to trim the set of linear functionals {Lxj : j ∈ Nm } to a maximally linearly independent set and then apply theorem 2 to this subset of linear functionals. This approach, of course, requires that y ∈ Ran(A) as well. 4 Regularization We begin this section with the approximation scheme that arises from the minimization of the functional
E( f ) := yj − f (xj )2 + µ f 2 , (4.1) j∈Nm

where µ is a fixed positive constant and {(xj , yj ) : j ∈ Nm } ⊆ X × Y . Our initial remarks shall prepare for the general case 4.5 treated later. Problem 4.1 is a special form of regularization functionals introduced by Tikhonov, Ivanov,

186

C. Micchelli and M. Pontil

and others to solve ill-posed problems (Tikhonov & Arsenin, 1977). Its application to learning is discussed, for example, in Evgeniou et al., (2000). A justification, from the theory of optimal estimation, that regularization is the optimal algorithm to learn a function from noisy data is given in Melkman and Micchelli (1979). Theorem 4. fˆ =




If fˆ minimizes E in H, it is unique and has the form Kxj cj

(4.2)

j∈Nm

where the coefficients {cj : j ∈ Nm } ⊆ Y are the unique solution of the linear equations


(K(xj , x ) + µδj )c = yj , j ∈ Nm .

(4.3)

∈Nm

Proof. that

The proof is similar to that of theorem 2. We set g = f − fˆ and note

E( f ) = E( fˆ) +




g(xj )2 − 2

j∈Nm




(yj − fˆ(xj ), g(xj ))

j∈Nm

+ 2µ fˆ, g + µg2 . Using equations 2.1, 4.2, and 4.3 gives the equations  fˆ, g =





(cj , g(xj ))

j∈Nm

(yj − fˆ(xj ), g(xj )) = µ

j∈Nm




(cj , g(xj )),

j∈Nm

and so it follows that E( f ) = E( fˆ) +




g(xj )2 + µg2 ,

j∈Nm

from which we conclude that fˆ is the unique minimizer of E. The representation theorem above embodies the fact that regularization is also an MNI problem in the space H × Y m . In fact, we can make this space into a Hilbert space by setting, for every f ∈ H and ξ = (ξj : j ∈ Nm ) ∈ Y m ,
( f, ξ )2 := ξj 2 + µ f 2 j∈Nm

On Learning Vector-Valued Functions

187

and note that the regularization procedure 4.1 is equivalent to the MNI problem defined in theorem 3 with linear functionals defined at ( f, ξ ) ∈ H × Y m by the equation Lxj ( f, ξ ) := f (xj ) + ξj , j ∈ Nm corresponding to data y = (yj : j ∈ Nm ). Let us consider again the case that Y = Rn . The linear system of equations 4.3 reads (G + µI)c = y,

(4.4)

and we view G as an m × m block matrix, where each block is an n × n matrix (so G is an mn×mn scalar matrix), and c = (cj : j ∈ Nm ), y = (yj : j ∈ Nm ) are vectors in Rmn . Specifically, the jth, kth block of G is Gjk = K(xj , xk ), j, k ∈ Nm . Proposition 1 ensures that G is symmetric and positive semidefinite. Moreover, the diagonal elements of G are positive semidefinite. There is a wide variety of circumstances where the linear systems of equations 4.4 can be effectively solved. Specifically, as before, we choose the kernel K := AT DA, where D := diag(K1 , . . . , Kn ) and each K ,  ∈ Nn is a prescribed scalarvalued kernel, and A is a nonsingular n × n matrix. In this case, the linear system of equations 4.4 becomes ˜ = y, A˜ T D˜ Ac where A˜ is the m × m block diagonal matrix whose n × n diagonal blocks are formed by the matrix A and D˜ is the m × m matrix whose jth, kth block is D˜ jk := D(xj , xk ), j, k ∈ Nm . When A is upper triangular, this system of ˜ = y and equations can be efficiently solved by first solving the system A˜T Dz ˜ then solving the system Ac = z. Both of these steps can be implemented by solving only m × m systems coupled with vector substitution. We can also reformulate the solution of the regularization problem 4.1 in terms of the feature map. Indeed, if  : X → L(W , Y ) is any such map, where W is some Hilbert space, the solution that minimizes equation 4.1 has the form f (x) = (x)w where w ∈ W is given by the formula −1 

∗ (xj )(xj ) + µI ∗ (xj )yj . w= j∈Nm

j∈Nm

Ym × R

Let V: + → R be a prescribed function and consider the problem of minimizing the functional E( f ) := V(( f (xj ) : j ∈ Nm ),  f )

(4.5)

over all functions f ∈ H. A special case is covered by the functional of the form
E( f ) := Q(yj , f (xj )) + h( f ), j∈Nm

188

C. Micchelli and M. Pontil

where h: R+ → R+ is a strictly increasing function and Q: Y × Y → R+ is some prescribed loss function. In particular, functional 4.1 corresponds to the choice h(t) := µt2 , t ∈ R+ , and Q(y, f (x)) = y − f (x)2 . However, even for this choice of h, other loss functions are important in applications (see Vapnik, 1998). Within this general setting, we provide a representation theorem for any function that minimizes the functional in equation 4.5. This result is wellknown in the scalar case (see, e.g., Scholkopf ¨ & Smola, 2002). The proof below uses the representation for minimal norm interpolation presented above. This method of proof has the advantage that it can be extended to Banach spaces (see Micchelli & Pontil, 2004, for discussion). Theorem 5. If for every y ∈ Y m , the function h: R+ → R+ defined for t ∈ R+ by h(t) := V(y,  t) is strictly increasing and f0 ∈ H minimizes the functional 4.5, then f0 = j∈Nm Kxj cj for some {cj : j ∈ Nm } ⊆ Y . In addition, if V is strictly convex, the minimizer is unique. Proof. Let f be any function such that f (xj ) = f0 (xj ), j ∈ Nm , and define y0 := ( f0 (xj ) : j ∈ Nm ). By the definition of f0 , we have that     V y0 ,  f0  ≤ V y0 ,  f  , and so  f0  = min{ f  : f (xj ) = f0 (xj ), j ∈ Nm , f ∈ H}.

(4.6)

Therefore, by theorem 3, the result follows. When V is strictly convex, the uniqueness of a global minimum of E is immediate. There are examples of functions V above for which the functional E given by equation 4.5 may have more than one local minimum. The question arises to what extent a local minimum of E has the form described in theorem 5. Our next result establishes that the form of any local minimizer has the same form as in theorem 5. This result seems new even in the scalar case. First, let us explain what we mean by a local minimum of E. A function f0 ∈ H is a local minimum for E provided that there is a positive number  such that whenever f ∈ H satisfies  f0 − f  ≤ , then E( f0 ) ≤ E( f ). Our first observation shows that the conclusion of theorem 5 remains valid for local minima of E. Theorem 6. If V satisfies  the hypotheses of theorem 5 and f0 ∈ H is a local minimum of E, then f0 = j∈Nm Kxj cj for some {cj : j ∈ Nm } ⊆ Y .

On Learning Vector-Valued Functions

189

Proof. If g is any function in H such that g(xj ) = 0, j ∈ Nm and t a real number such that |t|g ≤ , then     V y0 ,  f0  ≤ V y0 ,  f0 + tg . Consequently, we have that  f0  ≤  f0 + tg, from which it follows that ( f0 , g) = 0. Thus, f0 satisfies equation 4.6, and the result follows. So far, we have obtained a representation theorem for both global or local minima of E when V is strictly increasing in its last argument. Our second observation does not require this hypothesis. Instead, we shall see that if V is merely differentiable at a local minima and its partial derivative relative to its last coordinate is nonzero, the conclusion of theorems 5 and 6 remains valid. To state this observation, we explain precisely what we require of V. There exist c = (cj : j ∈ Nm ) ∈ Y m and a nonzero constant a ∈ R such that for any g ∈ H, the derivative of the univariate function h defined for t ∈ R by the equation h(t) := V(( f0 (xj ) + tg(xj ) : j ∈ Nm ),  f0 + tg) is given by h (0) =




(cj , g(xj )) + a f0 , g.

j∈Nm

For example, the regularization functional in equation 4.1 has this property. Theorem 7. If V is differentiable at a local minimum f0 ∈  H of E as defined above, then there exists c = (cj : j ∈ Nm ) ∈ Y m such that f0 = j∈Nm Kxj cj . Proof. By the definition of f0 , we have that h(0) ≤ h(t) for all t ∈ R such that |t|g ≤  for some  > 0. Hence, h (0) = 0, and so we conclude that

1
1
(cj , g(xj )) = − Kxj cj , g  f0 , g = − a j∈N a j∈N m

m

and since g was arbitrary, the result follows. We now discuss specific examples of loss functions that lead to quadratic programming (QP) problems. These problems are a generalization of the support vector machine (SVM) regression algorithm for scalar functions (see Vapnik, 1998).

190

C. Micchelli and M. Pontil

Example 1. We chooseY = Rn , y = (yj : j ∈ Nm ), f = ( f :  ∈ Nn ): X → Y , and Q(y, f (x)) := ∈Nn max(0, |y − f (x)| − ), where  is a positive parameter and consider the problem

min

 


j∈Nm

 

Q(yj , f (xj )) + µ f 2 : f ∈ H . 

(4.7)

Using theorem 2, we transform the above problem into the quadratic programming problem,   

min (ξj + ξj∗ ) · 1 + µ (cj , K(xj , x )c ) ,   j,∈Nm j∈Nm

subject, for all j ∈ Nm , to the constraints on the vectors cj , ξj , ξj∗ ∈ Rn , j ∈ Nm that yj −
∈Nm




K(xj , x )c ≤  + ξj

∈Nm

K(xj , x )c − yj ≤  + ξj∗ ξj , ξj∗ ≥ 0,

where the inequalities are meant to hold component-wise, the symbol 1 is used for the vector in Rn whose all of components are equal to one, and “·” stands for the standard inner product in Rm . Using results from quadratic programming (see Mangasarian, 1994), this primal problem can be transformed to its dual problem, 

1
(αj − αj∗ , K(xj , x )(α − α∗ )) 2 j,∈N m 
∗ ∗ (αj − αj ) · yj − (αj + αj ) · 1 , +

max −

j∈Nm

subject, for all j ∈ Nm , to the constraints on the vectors αj , αj∗ ∈ Rn , j ∈ Nm that 0 ≤ αj , αj∗ ≤ (2µ)−1 . If {(αˆ j , −αˆ j∗ ) : j ∈ Nn } is a solution to the dual problem, then {ˆcj := αˆ j − αˆ j∗ : j ∈ Nm } is a solution to the primal problem. The optimal parameters {(ξj , ξj∗ ) : j ∈ Nm } can be obtained, for example, by setting c = cˆ in the primal

On Learning Vector-Valued Functions

191

problem above and minimizing the resulting function only with respect to ξ, ξ ∗ . Example 2. The setting is as in example 1, but now we choose the loss function Q(y, f (x)) := max{max(0, |y − f (x)| − ) :  ∈ Nm }. In this case, problem 4.7 is equivalent to the following QP problem,   

min (ξj + ξj∗ ) + (cj , K(xj , x )c ) ,   j∈Nm

j,∈Nm

subject, for all j ∈ Nm , to the constraints on the vectors cj ∈ Rn , ξj , ξj∗ ∈ R, j ∈ Nm that
K(xj , x )c ≤  + ξj yj −
∈Nm

∈Nm

K(xj , x )c − yj ≤  + ξj∗ ξj , ξj∗ ≥ 0.

The corresponding dual problem is 

1
(αj − αj∗ , K(xj , x )(α − α∗ )) 2 j,∈N m 
∗ ∗ (αj − αj ) · yj − (αj + αj ) · 1 , +

max −

j∈Nm

subject, for all j ∈ Nm , to the constraints on the vectors αj , αj∗ ∈ Rn , j ∈ Nm that αj · 1, αj∗ · 1 ≤ (2µ)−1 , αj , αj∗ ≥ 0, where the symbol 1 is used for the vector in Rn whose components are all equal to one, and · stands for the standard inner product in Rm . We note that when Y = R, the loss functions used in examples 1 and 2 are the same as the loss function used in the scalar SVM regression problem— the loss Q(y, f (x)) := max(0, |y − f (x)| − ) (see Vapnik, 1998). In particular, the dual problems above coincide. Likewise, we can also consider squared versions of the loss functions in the above examples and show that the dual problems are still quadratic programming problems. Those problems reduce to the SVM regression problem with the loss max(0, |y − f (x)| − )2 .

192

C. Micchelli and M. Pontil

Examples 1 and 2 share some similarities with the multiclass classification problem considered in the context of SVM. This problem consists in learning a function from an input space X to the index set Nn . To every index  ∈ Nn , n > 2, we associate a th class or category denoted by C . A common approach to solve such problem is by learning a vector-valued function f = ( f :  ∈ Nn ): X → Rn and classify x in the class Cq such that q = argmax∈Nn f (x). Given a training set {(xj , kj ) : j ∈ Nm } ⊆ X × Nn with m > n if, for every j ∈ Nm , xj belongs to class Ckj , the output yj ∈ Rn is a binary vector with all components equal to −1 except for the k − jth component, which equals 1. Thus, the function f should separate as best as possible examples in the class C (the positive examples) from examples in the remaining  classes (the negative examples).1 In particular, if we choose Q(yj , f (xj )) := ∈Nn max(0, 1 − fkj (xj ) + f (xj )), the minimization problem 4.7 leads to the multiclass formulation studied in Vapnik (1998) and Weston and Watkins (1998), while the choice Q(yj , f (xj )) := max{max(0, 1 − fkj (xj ) + f (xj )) :  ∈ Nn ,  = kj } leads to the multiclass formulation of Cramer and Singer (2001) (see also Bennett & Mangasarian, 1993, for related work in the context of linear programming). (For further information on the quadratic programming problem formulation and discussion of algorithms for the solution of these optimization problems, see Bennett & Mangasarian, 1993; Cramer & Singer, 2001; Vapnik, 1998; and Weston & Watkins, 1998). Finally, we wish to emphasize that from a Bayesian perspective, kernels can be seen as covariances of a gaussian process in the RKHS. This relation is discussed Wahba (1990) and has been further developed in machine learning (see, e.g., Williams and Barber, 1998; Cornford, Csato, Evans, & Opper, 2004). 5 Kernels In this section, we consider the problem of characterizing a wide variety of kernels of a form that have been found to be useful in learning theory (see Vapnik, 1998). We begin with a discussion of dot product kernels. To recall n this idea, we let our space  X to be R , and on X we put the usual Euclidean inner product x · y = j∈Nn xj yj , x = (xj : j ∈ Nn ), y = (yj : j ∈ Nn ). A dot product kernel is any function of x · y, and a typical example is to choose Kp (x, y) = (x · y)p , where p is a positive integer. Our first result provides a substantial generalization. To this end, we let Nm be the set of vectors in Rm whose components are nonnegative integers, and if α = (αj : j ∈ Nm ) ∈    α Nm , z ∈ Rm , we define zα := j∈Nm zj j , α! := j∈Nm αj !, and |α| := j∈Nm αj . Finally, we let C be the set of complex numbers. 1 The case n = 2 (binary classification) is not included here because it merely reduces to learning a scalar-valued function.

On Learning Vector-Valued Functions

193

Definition 4. We say that a function h: Cm → L(Y ) is entire whenever there is a sequence of bounded operators {Aα : α ∈ Nm } ⊆ L(Y ), such that for every z ∈ Cm and any c ∈ Y , the function


(c, Aα c)zα

α∈Nm

is an entire function on Cm , and for every c ∈ Y and z ∈ Cm , it equals (c, h(z)c).2 In this case we write h(z) =




Aα zα , z ∈ Cm .

(5.1)

α∈Nm

Let B1 , . . . , Bm be any n × n complex matrices and h an entire function with values in L(Y ). We let h((B1 ), . . . , (Bm )) be the n × n matrix whose j,  element is the operator h((B1 )j , . . . , (Bm )j ) ∈ L(Y ) Proposition 2. If {Kj : j ∈ Nm } is a set of kernels on X × X with values in R, and h: Rm → L(Y ) an entire function of the type 5.1 where {Aα : α ∈ Nm } ⊆ L+ (Y ), then K = h(K1 , . . . , Km ) is a kernel on X × X with values in L(Y ). Proof. that

For any {cj : j ∈ Nm } ⊆ Y , and {xj : j ∈ Nm } ⊆ X , we must show


(cj , h(K1 (xj , x ), . . . , Km (xj , x ))c ) ≥ 0.

j,∈Nm

To this end, we write the sum in the form

α∈Nm

αm K1α1 (xj , x ) · · · Km (xj , x )(cj , Aα c ).

j,∈Nm

Since Aα ∈ L+ (Y ), it follows that the matrix ((cj , Aα c )), j,  ∈ Nm is positive semidefinite. Therefore, by the lemma of Schur (Aronszajn, 1950), we see that the matrix whose jth, th element appears in the sum above is positive semidefinite for each α ∈ Nm . From this fact, the result follows. 2 A function f : Cm → C is called entire or analytic if it has a convergent power series everywhere in Cm .

194

C. Micchelli and M. Pontil

In our next observation, we show that the function h with the property described in proposition 2 must have the form 5.1 with {Aα : α ∈ Nm } ⊆ L+ (Y ) provided it and the kernels K1 , . . . , Km satisfy some additional conditions. This issue is resolved in FitzGerald, Micchelli, and Pinkus (1995) in the scalar case. To apply those results, we find the following definition useful: Definition 5. We say that the set of kernels Kj : X × X → C, j ∈ Nm , is full if for every n and any n × n positive semidefinite matrices Bj , j ∈ Nm , there exists a sequence of sets of distinct points {{xs :  ∈ Nn } : s ∈ N} ⊆ X such that for every j ∈ Nm , Bj = lims→∞ (Kj (xs , xsk )),k∈Nn . For example, when m = 1 and X is an infinite-dimensional Hilbert space with inner product (·, ·), the kernel (·, ·) is full. Proposition 3. If {Kj : j ∈ Nm } is a set of full kernels on X × X , hCm → L(Y ), there exists some r > 0 such that the constant γ = sup{h(z) : z ≤ r} is finite, and if h(K1 , . . . , Km ) is a kernel with values in L(Y ), then h is entire of the form h(z) =




Aα z α , z ∈ C m

α∈Nm

for some {Aα : α ∈ Nm } ⊆ L+ (Y ). For the proof, see Micchelli and Pontil (2003). Let us now comment on translation-invariant kernels on Rn with values in L(Y ). This case is covered in great generality in Berberian (1966) and Fillmore (1970), where the notion of operator-valued Borel measures is described and used to generalize the theorem of Bochner (see Fillmore, 1970). For the purpose of applications, we point out the following sufficient condition on a function h: Rn → L(Y ) to give rise to a translation-invariant kernel. Specifically, for each x ∈ Rn , we let W(x) ∈ L+ (Y ), and define for x ∈ Rn the function  eix·t W(t) dt. h(x) = Rn

Consequently, for every m ∈ N, {xj : j ∈ Nm } ⊆ Rn , and {cj : j ∈ Nm } ⊆ Y , we have that 
(cj , h(xj − x )c ) = (cj eixj ·t , W(t)(c eix ·t )), j,∈Nm

Rn

On Learning Vector-Valued Functions

195

which implies that K(x, y) := h(x − t) is a kernel on Rn × Rn . In Berberian (1966) and Fillmore (1970), all translation-invariant kernels are characterized in the form  h(x) = eix·t dµ(t), Rn

where dµ is an operator-valued Borel measure relative to L+ (Rn ). In particular, this says, for example, that a function
ei(x−t)·tj Bj h(x − t) = j∈Nm

where {Bj : j ∈ Nm } ⊆ L+ (Rn ) gives a translation-invariant kernel too. We also refer to Berberian (1966) and Fillmore (1970) to assert the claim that the function g: R+ → R+ such that K(x, t) := g(x − t2 ) is a radial kernel on X × X into L(Y ) for any Hilbert space X has the form  ∞ g(s) = e−σ s dµ(σ ), 0

where again dµ is an operator-valued measure with values in L+ (Y ). When the measure is discrete and Y = Rn , we conclude that
2 e−σj x−t Bj K(x, t) = j∈Nm

is a kernel for any {Bj : j ∈ Nm } ⊆ L+ (Rn ) and {σj : j ∈ Nm } ⊂ R+ . This is the form of the kernel we use in our numerical simulations. 6 Practical Considerations In this final section, we describe several issues of a practical nature. They include a description of some numerical simulations we have performed, a list of several problems for which we feel learning vector-valued functions is valuable, and a brief review of previous literature on the subject. 6.1 Numerical Simulations. In order to investigate the practical advantages offered by learning within the proposed function spaces, we have carried out a series of three preliminary experiments where we tried to learn a target function f0 by minimizing the regularization functional in equation 4.1 under different conditions. In all experiments, set f0 to be of the form f0 (x) = K(x, x1 )c1 + K(x, x2 )c2 , c1 , c2 ∈ Rn , x ∈ [−1, 1].

(6.1)

In the first experiment, the kernel K was given by a mixture of two gaussian functions, K(x, t) = Se−6x−t + Te−30x−t , x, t ∈ [−1, 1], 2

2

(6.2)

196

C. Micchelli and M. Pontil

where S, T ∈ L+ (Rn ). Here we compared the solution obtained by minimizing equation 4.1 either in the RKHS of kernel K above (we call this method 1) or in the RKHS of the diagonal kernel,  I

 tr(S) −6x−t2 tr(T) −30x−t2 + e e , x, t ∈ [−1, 1], n n

(6.3)

where tr denotes the trace of a squared matrix (we call this method 2). We performed different series of simulations, each identified by the number of the outputs n ∈ {2, 3, 5, 10, 20} and an output noise parameter a ∈ {0.1, 0.5, 1}. Specifically, for each pair (n, a), we computed the matrices S = A∗ A and T = B∗ B where the elements of matrices A and B were uniformly sampled in the interval [0, 1]. The centers x1 , x2 ∈ [−1, 1] and the vector parameters c1 , c2 ∈ [−1, 1]n defining the target function f0 were also uniformly distributed. The training set {(xj , yj ) : j ∈ Nm } ⊂ [−1, 1] × Rn was generated by sampling the function f0 with noise: xj was uniformly distributed in [−1, 1] and yj = f0 (xj ) + j with j also uniformly sampled in the interval [−a, a]. In all simulations, we used a set of m = 20 points for training and a separate validation set of 100 data to select the optimal value of the regularization parameter. Then we computed on a separate test set of 200 samples the mean squared error between the target function f0 and the function learned from the training set. This simulation was repeated 100 times; that is, for each pair (n, a), 100 test error measures were produced for method 1 and method 2. As a final comparison measure of method 1 and method 2, we computed the average difference between the test error of method 2 and the test error of method 1 and its standard deviation, so, for example, a positive value of this average means that method 1 performs better than method 2. These results are shown in Table 1, where each cell contains the average difference of the test squared errors and, below it, its standard deviation. Not surprisingly, the results indicate that the use of the nondiagonal kernel always on the average improves performance, especially when the number of outputs increases. In the second experiment, the target function in equation 6.1 was modeled by the diagonal kernel,   2 2 K(x, t) = I ae−6x−t + be−30x−t , x, t ∈ [−1, 1], where a, b ∈ [0, 1], a + b = 1. In this case, we compared the optimal solution produced by minimizing equation 4.1 in either the RKHS of the above kernel (we call this method 3) or the RKHS of the kernel an bn 2 2 Se−6x−t + Te−30x−t , x, t ∈ [−1, 1] tr(S) tr(T) (we call this method 4). The results of this experiment are shown in Table 2, where it is evident that the nondiagonal kernel very often does worse

On Learning Vector-Valued Functions

197

Table 1: Results of Experiment 1. a\n

2

3

5

10

20

0.1

2.01e−4 (1.34e−4) 1.83−3 (1.17e−3) .0167 (.0104)

9.28e−4 (5.07e−4) .0276 (.0123) .0238 (.0097)

.0101 (.0203) .0197 (.126) .0620 (.0434)

.0178 (.0101) .0279 (.0131) .3397 .3608

.0331 (.0238) .5508 (.3551) .5908 (0.3801)

0.5 1

Notes: The average difference between the test error of method 1 and the test error of method 2 with its standard deviation (bottom row in each cell) for different values of the number of outputs n and the noise level a. The average error of method 1 (not reported here) ranged between 2.80e−3 and 1.423.

Table 2: Results of Experiment 2. a\n

2

3

5

10

20

0.1

8.01e−4 (6.76e−4) 1.57e−3 (1.89e−3) .0113 (.0086)

7.72e−4 (8.05e−4) .0133 (.0152) .0149 (.0245)

-6.85e−4 (5.00e−4) 7.30−3 (4.51e−3) 9.17e−3 (5.06e−3)

6.72e−4 5.76e−4 -7.30e−3 (4.51e−3) .0242 (.0103)

7.42e−4 3.99e−3 8.64e−3 (4.63e−3) .0233 (.0210)

0.5 1

Notes: The average difference between the test error of method 3 and the test error of method 4 with its standard deviation for different values of the number of outputs n and the noise level a. The average error of method 3 (not reported here) ranged between 3.01e−3 and 6.95e−2.

than the diagonal kernel. Comparing these results with those in Table 1, we conclude that the nondiagonal kernel does not offer any advantage when n = 2, 3. But this type of kernel helps dramatically when the number of outputs is larger than or equal to 5 (it does little damage in experiment 2 and great benefit in experiment 1). There is no current explanation for this phenomenon. Thus, it appears to us that such a kernel and its generalization, as we discussed above, may be important for applications where the number of outputs is large and complex relations among them are likely to occur. The last experiment addressed hyperparameters tuning. Specifically, we again choose a target function as in equation 6.1 and K as in equation 6.2, but now, S = S0 where S0q = 1 if  = q, −.4 if  = q + 1 (with periodic conditions, i.e., n + 1 = 1), and zero otherwise, and T = T0 is a sparse outof-diagonal positive definite binary matrix. Thus, S0 models correlations between adjacent components of f (it acts as a difference operator), whereas T0 accounts for possible “far-away” component correlations. We learn this

198

C. Micchelli and M. Pontil

target function by means of three kernel models. Model 1 consists of kernels as in equation 6.2 with S = S0 + λI and T = ρT0 , λ, ρ ≥ 0. Model 2 is as model 1 but with ρ = 0. Finally, model 3 consists of diagonal kernels as in equation 6.3, with S = S0 and T = νT0 , ν > 0. The data generation setting was as in the above experiments except that the matrices S0 and T0 were kept fixed in each simulation (this explains the small variance in the plots below) and the output noise parameter was fixed to 0.5. Hyperparameters of each model, as well as the regularization parameter in equation 4.1, were found by minimizing the squared error computed on a separate validation set containing 100 independently generated points. Figure 1 depicts the mean square test error with its standard deviation as a function of the number of training points (5,10,20,30) for n = 10, 20, 40 output components. The result clearly indicated the advantage offered by model 1. Although the above experiments are preliminary and consider a simple data setting, they enlighten the value offered by matrix-valued kernels. 6.2 Application Scenarios. We outline some practical problems arising in the context of learning vector-valued functions where the above theory can be of value. Although we do not treat the issues that would arise in a detailed study of these problems, we hope that our discussion will motive substantial studies of them. A first class of problems deals with finite-dimensional vector spaces. For example, an interesting problem arises in reinforcement learning or control when we need to learn a map between a set of sensors placed in a robot or autonomous vehicle and a set of actions taken by the robot in response to the surrounding environment (see, e.g., Franke et al., 1998). Another instance of this class of problems deals with the space of n × m matrices over a field whose choice depends on the specific application. Such spaces can be equipped with the standard Frobenius inner product. One of many problems that come to mind is to compute a map that transforms an ×k image x into an n × m image y. Here both images are normalized to the unit interval, so that X = [0, 1]×k and Y = [0, 1]n×m . There are many specific instances of this problem. In image reconstruction, the input x is an incomplete (occluded) image obtained by setting to 0 the pixel values of a fixed subset in an underlying image y, which forms our target output. A generalization of this problem is image denoising, where x is an n × m image corrupted by a fixed but unknown noise process and the output y is the underlying “clean” image. A different application is image morphing, which consists in computing the pointwise correspondence between a pair of images depicting two similar objects, for example, the faces of two people. The first image, x0 , is meant to be fixed (a reference image), whereas the second one, x, is sampled from a set of possible images in X = [0, 1]n×m . The output y ∈ R2(n×m) is a vector field in the input image plane that associates each pixel of the input image to the vector of coordinates of the corresponding pixel in the reference image (see, e.g., Beymer & Poggio, 1996, for more information on these issues).

On Learning Vector-Valued Functions

199

1.4

n=10

1.2 1 0.8 0.6 0.4 0.2

5

10

15

20

25

1.2

30

n=20

1 0.8 0.6 0.4 0.2

5

10

15

20

25

30

1.6

n=40 1.4 1.2 1 0.8 0.6 0.4 0.2 5

10

15

20

25

30

Figure 1: Experiment 3 (n = 10, 20, 40). Mean square test error as a function of the training set size for model 1 (solid line), model 2 (dashed line), and model 3 (dotted line). See the text for descriptions of these models.

200

C. Micchelli and M. Pontil

A second class of problems deals with spaces of strings, such as text, speech, or biological sequences. In this case, Y is a space of finite sequences whose elements are in a (usually finite) set A. A famous problem asks to compute a text-to-speech map that transforms a word from a certain fixed language into its sound as encoded by a string of phonemes and stresses. This problem was studied by Sejnowski and Rosenberg (1987) in their NETtalk system. Their approach consists of learning several Boolean functions that subsequently are concatenated to obtain the desired text-to-speech map. Another approach is to learn directly a vector-valued function. In this case, Y could be described by means of an appropriate RKHS (see below). Another important problem is protein structure prediction, which consists in predicting the 3D structure of a protein immersed in a aqueous solution (see, e.g., Baldi, Pollastri, Frasconi, & Vullo, 2002). Here x = (x j ∈ A : j ∈ N ) is the primary sequence of a protein, that is, a sequence over an alphabet of 20 possible symbols representing the amino acids present in nature. The length  of the sequence varies typically between a few tens of and a few thousand elements. The output y = (yj ∈ R3 : j ∈ N ) is a vector with the same length as x and yj in the position of the jth amino acid of the input sequence. Toward the solution of this difficult problem, an intermediate step consists of predicting a neighborhood relation among the amino acids of a protein. In particular, recent work has focused on predicting a squared binary matrix C, called the contact map matrix, where C(j, k) = 1 if the amino acids j and k are at a distance smaller or of the order of 10A, and zero otherwise (see Baldi et al., 2002). Note that in the last class of problems, the output space is not a Hilbert space, since the sum of two vectors of different length is not defined. An approach to overcome this difficulty is to embed Y into a RKHS. This requires choosing a scalar-valued kernel G: Y × Y → R, which provides a RKHS space HG . We associate every element y ∈ Y with the function G(y, ·) ∈ HG and denote this map by G: Y → HG . If we wish to learn a function f : X → Y , we instead learn the composition mapping f˜ := G ◦ f . We then can obtain f (x) for every x ∈ X as f (x) = argminy∈Y G(y, ·) − f˜(x)G . If G is chosen to be bijective, the minimum is unique. This idea is also described in Weston, Chapelle, Elisseeff, Scholkopf, ¨ and Vapnik, (2003), where some applications of it are presented. As a third and final class of problems that illustrate the potential of vector-valued learning, we point to the possibility of learning a curve, or even a manifold. In the first case, the usual paradigm for the computer generation of a curve in Rn is to start with scalar-valued basis functions {Mj : j ∈ Nm } and a set of control points, {cj : j ∈ Nm } ⊆ Rn , and consider

On Learning Vector-Valued Functions

201

the vector-valued function
cj Mj f = j∈Nm

(see, e.g., Micchelli, 1995). We are given data {yj : j ∈ Nm } ⊆ Rn , and we want to learn the control points. However the input data, which belong to a unit interval, say X = [0, 1], are unknown. We then fix {xj : j ∈ Nm } ⊂ [0, 1], and we learn a parameterization τ : [0, 1] → [0, 1] such that f (τ (xi )) = yi . So the problem is to learn both τ and the control points. Similarly, if {Mj : j ∈ Nm } are functions in Rk , k < n, we face the problem of learning a manifold M obtained by embedding Rk in Rn . It seems that to learn the manifold M, a kernel of the form
Mj (x)Mj (t)Aj K(x, t) = j∈Nm

where {Aj : j ∈ Nm } ⊂ L+ (Rn ) would be appropriate because the functions generated by such kernel lie in the manifold generated by the basis functions, namely, K(x, t)c ∈ M for every x, t ∈ Rk , c ∈ Rn . 6.3 Previous Works on Vector-Valued Learning. The problem of learning vector-valued functions has been addressed in the statistical literature under the name of multiple response estimation or multiple output regression (see Hastie, Tibshirani, & Friedman, 2002). Here we briefly discuss two methods that we find interesting. A well-studied technique is the Wold partial least-squares (PLS) approach. This method is similar to principal component analysis but with the important difference that the principal directions are computed simultaneously in the input and output spaces, both being finite-dimensional Euclidean spaces (see, e.g., Hastie et al., 2002). Once the principal directions have been computed, the data are projected along the first n directions and a least-square fit is computed in the projected space where the optimal value of the parameter n can be estimated by means of cross-validation. We note that like principal component analysis, partial least squares are linear models, that is, they can model only vector-valued functions that depend linearly on the input variables. Recent work by Rosipal and Trejo (2001) and Bennett and Embrechts (2003) has reconsidered PLS in the context of scalar RKHS. Another statistically motivated method is the Breiman and Friedman (1997). curd & whey procedure. This method consists of two main steps. First, the coordinates of a vector-valued function are separately estimated by means of a least-squares fit or by ridge regression. Then they are combined in a way that exploits possible correlations among the responses (output

202

C. Micchelli and M. Pontil

coordinates). The authors show experiments where their method is capable of reducing prediction error when the outputs are correlated and not increasing the error when the outputs are uncorrelated. The curd & whey method is also primarily restricted to model linear relations of possibly nonlinear functions. However, it should be possible to “kernelize” this method following the same lines as in Bennett and Embrechts (2003). Acknowledgments We are grateful to Dennis Creamer, head of the Computational Science Department at National University of Singapore, for providing both of us with the opportunity to complete this work in a scientifically stimulating and friendly environment. Kristin Bennett of the Department of Mathematical Sciences at RPI and Wai Shing Tang of the Mathematics Department at NUS provided us with several helpful references and remarks. Bernard Buxton of the Department of Computer Science at UCL read a preliminary version of the manuscript and made useful suggestions. Finally, we are grateful to Phil Long of the Genome Institute of NUS for discussions that led to theorem 6, as well as the referees for their useful comments. This work was partially supported by NSF grant ITR-0312113. References Akhiezer, N.I., & Glazman, I.M. (1993). Theory of linear operators in Hilbert spaces (Vol. 1). New York: Dover. Amodei, L. (1997). Reproducing kernels of vector-valued function spaces. In A. Le Meehaute, C. Rabut, & L. L. Schumaker (Eds.), Curves and Surfaces: Proceedings of Chamonix 1996. Nashville, TN: Vonderbilt University Press. Aronszajn, N. (1950). Theory of reproducing kernels. Trans. Amer. Math. Soc., 686, 337–404. Baldi, P., Pollastri, G., Frasconi, P., & Vullo, A. (2002). New machine learning methods for the prediction of protein topologies. In P. Frasconi & R. Shamir (Eds.), Artificial intelligence and heuristic methods for bioinformatic. Amsterdam: IOS Press. Bennett, K.P., & Embrechts, M.J. (2003). An optimization perspective on partial least squares. In J. Suykens, G. Horrath, S. Basu, C. Micchelli, & J. Vandewalle (Eds.), Advances in kearning theory: Methods, models, and applications. (pp. 227– 250). Amsterdam: IOS Press. Bennett, K.P., & Mangasarian, O.L. (1993). Multicategory discrimination via linear programming. Optimization Methods and Software, 3, 722–734. Berberian, S.K., (1966). Notes on spectral theory. New York: Van Nostrand. Beymer, D., & Poggio, T. (1996). Image representations for visual learning. Science, 272(5270), 1905–1909. Breiman, L., & Friedman, J. (1997). Predicting multivariate responses in multiple linear regression (with discussion). J. Roy. Statist. Soc. B., 59, 3–37.

On Learning Vector-Valued Functions

203

Burbea, J., & Masani, P. (1984). Banach and Hilbert spaces of vector-valued functions. New York: Pitman. Cherkassky, V., & Mulier, F.(1998). Learning from data: Concepts, theory, and methods. New York: Wiley. Cornford, D., Csato, L., Evans, D., & Opper, M. (2004). Bayesian analysis of the scatterometer wind retrieval inverse Problem: Some new approaches. Journal of the Royal Statistical Society B, 66, 1–17. Cramer, K., & Singer, Y. (2001). On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learning Research, 2, 265– 292. Cressie, N.A.C. (1993). Statistics for spatial data. New York: Wiley. Cristianini N., & Shawe-Taylor, J. (2000). An introduction to support vector machines. Cambridge: Cambridge University Press. Evgeniou, T., & Pontil, M. (2004). Regularized multi-task learning. Proc. of the 10th ACM SIGKOD Int. Conf. on Knowledge Discovery and Data Mining, Seattle, WA. Evgeniou, T., Pontil, M., & Poggio, T. (2000). Regularization networks and support vector machines. Advances in Computational Mathematics, 13, 1–50. Fillmore, P.A. (1970). Notes on operator theory. New York: Van Nostrand. FitzGerald, C.H., Micchelli, C. A., & Pinkus, A. M. (1995). Functions that preserve families of positive definite functions. Linear Algebra and Its Applications, 221, 83–102. Franke, U., Gavrila, D., Goerzig, S., Lindner, F., Paetzold, F., & Woehler, C. (1998). Autonomous driving goes downtown. IEEE Intelligent Systems 13, 40–48. Hastie, T., Tibshirani, R., & Friedman, J. (2002). The elements of statistical learning: Data mining, inference, and prediction. New York: Springer-Verlag. Mangasarian, O.L. (1994). Nonlinear programming. Philadelphia: SIAM. Melkman, A.A., & Micchelli, C.A., (1979). Optimal estimation of linear operators in Hilbert spaces from inaccurate data. SIAM Journal of Numerical Analysis, 16(1), 87–105. Micchelli, C.A. (1995). Mathematical aspects of geometric modeling. Philadelphia: SIAM. Micchelli, C.A., & Pontil, M. (2003). On learning vector-valued functions (Research Note No. RN/03/08). London: Department of Computer Science, University College London. Micchelli, C.A., & Pontil, M. (2004). A function representation for learning in Banach spaces. In J. Shawe-Taylor & Y. Singer (Eds), Proceedings of the Seventeenth Annual Conference on Learning Theory. New York: Springer-Verlag. Rosipal, R., & Trejo, L. J. (2001). Kernel partial least squares regression in reproducing kernel Hilbert spaces. Journal of Machine Learning Research, 2, 97–123. Scholkopf, ¨ B., & Smola, A.J. (2002). Learning with kernels. Cambridge, MA: MIT Press. Sejnowski, T.J., & Rosenberg, C.R. (1987). Parallel networks which learn to pronounce English text. Complex Systems, 1, 145–163. Tikhonov, A. N., & Arsenin, V. Y. (1997). Solutions of ill-posed problems. Washington, DC: Winston. Vapnik, V. N. (1998). Statistical learning theory. New York: Wiley.

204

C. Micchelli and M. Pontil

Wahba, G. (1990). Splines models for observational data. Philadelphia: SIAM. Weston, J., Chapelle, O., Elisseeff, A., Scholkopf, ¨ B., & Vapnik, V.N. (2003). Kernel dependency estimation. In Advances in neural information processing Systems, 15, Cambridge, MA: MIT Press. Weston, J. & Watkins, C. (1998). Multi-class support vector machines. (Tech. Rep. No. CSD-TR-98-04). London: Department of Computer Science, Royal Holloway, University of London. Williams, C.K.I. & Barber, D. (1998). Bayesian classification with gaussian processes. IEEE Trans. on Patt. Anal. Mach. Intel., 20, 1342–1351. Received November 5, 2003; accepted May 27, 2004.

LETTER

Communicated by Stephen Jose Hanson

RSPOP: Rough Set–Based Pseudo Outer-Product Fuzzy Rule Identification Algorithm Kai Keng Ang [email protected]

Chai Quek [email protected] Centre for Computational Intelligence, Nanyang Technological University, School of Computer Engineering, Singapore 639798

System modeling with neuro-fuzzy systems involves two contradictory requirements: interpretability verses accuracy. The pseudo outer-product (POP) rule identification algorithm used in the family of pseudo outerproduct-based fuzzy neural networks (POPFNN) suffered from an exponential increase in the number of identified fuzzy rules and computational complexity arising from high-dimensional data. This decreases the interpretability of the POPFNN in linguistic fuzzy modeling. This article proposes a novel rough set–based pseudo outer-product (RSPOP) algorithm that integrates the sound concept of knowledge reduction from rough set theory with the POP algorithm. The proposed algorithm not only performs feature selection through the reduction of attributes but also extends the reduction to rules without redundant attributes. As many possible reducts exist in a given rule set, an objective measure is developed for POPFNN to correctly identify the reducts that improve the inferred consequence. Experimental results are presented using published data sets and real-world application involving highway traffic flow prediction to evaluate the effectiveness of using the proposed algorithm to identify fuzzy rules in the POPFNN using compositional rule of inference and singleton fuzzifier (POPFNN-CRI(S)) architecture. Results showed that the proposed rough set–based pseudo outer-product algorithm reduces computational complexity, improves the interpretability of neuro-fuzzy systems by identifying significantly fewer fuzzy rules, and improves the accuracy of the POPFNN. 1 Introduction Neural networks and fuzzy systems are very popular techniques in soft computing (Zadeh, 1994). Neuro-fuzzy hybridization synergizes these two techniques by combining the human-like reasoning style of fuzzy systems with the learning and connectionist structure of neural networks. Neuro-fuzzy hybridization is widely termed fuzzy neural networks (FNN) or neuroNeural Computation 17, 205–243 (2005)

c 2004 Massachusetts Institute of Technology


206

K. Ang and C. Quek

fuzzy systems (NFS) in the literature (Mitra & Hayashi, 2000; Buckley & Hayashi, 1994, 1995; Nauck, Klawonn, & Kruse, 1997; Lin & Lee, 1996). Neuro-fuzzy systems (the more popular term is used henceforth) incorporates the human-like reasoning style of fuzzy systems through the use of fuzzy sets and a linguistic model consisting of a set of if-then fuzzy rules. Thus, the main strength of neuro-fuzzy systems is that they are universal approximators (Castro, 1995; Ying, Ding, Li, & Shao, 1999; Tikk, Koczy, ´ & Gedeon, 2003) with the ability to solicit interpretable if-then rules (Guillaume, 2001). The identification of interpretable fuzzy rules thus forms the most important aspect in the design of neuro-fuzzy systems. A number of articles on research on the learning of fuzzy rules from numerical data have been published (Lin & Lee, 1991; Hayashi, Nomura, Yamasaki, & Wakami, 1992; Ishibuchi, Tanaka, & Okada, 1994; Yager, 1994; Shann & Fu, 1995; Lozowski, Cholewo, & Zurada, 1996; Quek & Zhou, 2001; Tung & Quek, 2002b), and a survey is given in Mitra and Hayashi (2000). The strength of neuro-fuzzy systems involves two contradictory requirements in fuzzy modeling: interpretability versus accuracy. In practice, one of the two properties prevails. The fuzzy modeling research field is divided into two areas: linguistic fuzzy modeling, which is focused on interpretability, mainly the Mamdani model (Mamdani & Assilian, 1975) given in equation 1.1, and precise fuzzy modeling, which is focused on accuracy, mainly the Takagi-Sugeno-Kang (TSK) model (Takagi & Sugeno, 1985; Sugeno & Kang, 1988) given in equations 1.2 and 1.3 (Guillaume, 2001; Casillas, Cordon, ´ Herrera, & Magdalena, 2003). A recent comprehensive coverage on interpretability issues of fuzzy modeling is given in Casillas et al. (2003). Rk : IF x1 is Ak,1 AND · · · AND xn1 is Ak,n1 THEN y is Bk

(1.1)

Rk : IF x1 is Ak,1 AND · · · AND xn1 is Ak,n1 THEN yk = ak x + bk n3
y= wk (x)yk ,

(1.2) (1.3)

k=1

where x, y are the input vector x = [x1 , . . . , xn1 ] and output value, respectively, Ak,i , Bk are the linguistic labels with fuzzy sets associated defining their meaning, n1 is the number of inputs, and n3 is the number of rules. The consequents in equation 1.2 are linear functions of the inputs, whereas the consequents in equation 1.1 are simply linguistic labels. Therefore, the TSK model has decreased interpretability but increased representative power compared to the Mamdani model (Casillas et al., 2003). Recently, a number of articles on research addressing the issues of the former have been reported (Johansen & Babuˇska, 2003; Jin, 2000; Yen, Wang, & Gillespie, 1998). In contrast, an increased number of fuzzy rules are needed in the latter to yield the same representative power of the former. However, if the number of fuzzy rules is large in any of these models, then any semantic

RSPOP

207

interpretability is lost (Tikk & Baranyi, 2003; Nauck, 2003). Moreover, if the number of rules is bounded as a practical limitation, then the universal approximation property of the models does not hold anymore (Moser, 1999). Thus, the interpretability issue that arises from the large number of fuzzy rules motivates the complexity reduction of rule-based models; a recent survey is given in Kaynak, Jezernik, and Szeghegyi (2002). This issue is similar to the problems encountered by numerical data-driven techniques in data mining (Han & Kamber, 2001) due to ultra-large data and redundant data. These techniques rely on heuristics to guide their search or reduce their search space horizontally or vertically (Lin & Cercone, 1997). Horizontal reduction is realized by the merging of identical data tuples or the quantization of continuous numerical values. Vertical reduction is realized by the application of feature selection methods. The former corresponds to fuzzy sets and the latter to fuzzy rule pruning and reduction in neuro-fuzzy systems. In the literature, fuzzy rule identification algorithms apply fuzzy rule pruning based on certainty factors (Chakraborty & Pal, 2004; Shann & Fu, 1995), or identify fuzzy rules based on certain heuristic threshold (Quek & Zhou, 2001; Tung & Quek, 2002b). Recently, rough set methods (Pawlak, 1991) have been shown to reduce pattern dimensionality significantly and proven to be viable data mining techniques (Swiniarski & Skowron, 2003). Shen and Chouchoulas (2002) presented a rough-fuzzy approach in fuzzy rule identification using the rough set attribute reduction algorithm (RSAR) (Shen & Chouchoulas, 1999) to reduce the attributes of fuzzy rules identified from the rule induction algorithm (RIA) (Lozowski et al., 1996). This approach not only reduced the sensitivity of the fuzzy rules to the data dimension but also outperformed the fuzzy rules before reduction. This motivates the investigation on the rough-fuzzy approach of fuzzy rule identification in this work. This letter proposes a novel rough set–based pseudo outer-product (RSPOP) fuzzy rule identification algorithm for the pseudo outer-product– based fuzzy neural networks (POPFNN) (Ang, Quek, & Pasquier, 2003; Zhou & Quek, 1996; Quek & Zhou, 1999). The pseudo outer-product (POP) fuzzy rule identification algorithm (Quek & Zhou, 2001) used in POPFNN is a simple one-pass rule identification algorithm that is fast, reliable, efficient, and easy to understand. However, it has suffered from an exponential increase in the number of fuzzy rules identified, as well as an increased computational complexity that arose with high-dimensional data. This decreased the interpretability of POPFNN in linguistic fuzzy modeling as well as accuracy in high-dimensional modeling. Therefore, instead of pruning fuzzy rules based on certainty factors that introduce inconsistencies, as described in Pal and Pal (1999), or the use of a heuristic threshold to govern the derivation of fuzzy rules, the proposed RSPOP algorithm integrates the sound concept of knowledge reduction from rough set theory (Pawlak, 1991) with the POP algorithm (Quek & Zhou, 2001). Compared with rough set attribute reduction (RSAR) (Shen & Chouchoulas, 1999), the proposed

208

K. Ang and C. Quek

RSPOP algorithm not only performs reduction of redundant attributes, but also extends the reduction to the fuzzy rules without redundant attributes. A brief background of POPFNN and rough set (Pawlak, 1991) is provided in sections 2 and 3. Section 4 gives a detailed description of the proposed RSPOP algorithm. Experimental results and analysis are provided in section 5. Finally conclusions are presented in section 6. 2 Pseudo Outer-Product-Based Fuzzy Neural Network POPFNN is a family of neuro-fuzzy systems based on the linguistic fuzzy model (Quek & Zhou, 2001). Three members of POPFNN exist in the literature: POPFNN-CRI(S) (Ang et al., 2003), which is based on the commonly accepted fuzzy compositional rule of inference; POPFNN-TVR (Zhou & Quek, 1996), which is based on truth value restriction; and POPFNN-AARS(S) (Quek & Zhou, 1999), which is based on the approximate analogical reasoning scheme. The POPFNN architecture is a five-layer neural network, as shown in Figure 1. For simplicity, only the interconnections for the output ym are shown in the figure. Each layer performs a specific fuzzy operation, and the nodes and operations of each layer are noted with a superscript of I to V for clarity. The inputs and outputs are represented as nonfuzzy vector X = [x1 , x2 , . . . , xi , . . . , xn1 ] and nonfuzzy vector Y = [y1 , y2 , . . . , yl , . . . , yn5 ], respectively. The fuzzification of the inputs and the defuzzification of the outputs are, respectively, performed by the input linguistic and output linguistic layers, while the fuzzy inference is collectively performed by the rule, condition, and consequence layers. The numbers of neurons in the condition, rule, and consequence layers are defined in equations 2.1 to 2.3, respectively: n2 =

n1


Ji

(2.1)

i=1

  n1  n3 = Ji n4 =

i=1 n5


Lm ,

(2.2) (2.3)

m=1

where Ji is the number of linguistic labels for the ith input; Lm is the number of linguistic labels for the mth output; n1 is the number of inputs; n2 is the number of neurons in the condition layer; n3 is the number of rules or rulebased neurons; n4 is the number of linguistic labels for the output; and n5 is the number of outputs. Neurons in the input linguistic layer are called input nodes. Each input node Ii1 represents an input linguistic variable (Zadeh, 1975a, 1975b, 1975c)

RSPOP

209 y1

ym

yn5

O1V

OmV

OnV5

IV IV OL1,1 OL1,2

Consequent

OL1,IVL

1

OLIV m,1

I Antecedent ci ∈Ck

II IL1,2

IV OLIV n ,1 OLn ,2

OLIV m, Lm

5

wk ,m,1 wk ,m ,l wk ,m, Lm

d m ∈DVk

R1III

II IL1,1

OLIV m,l

c1

IL1,IIJ

1

RkIII

Weights

ILiII, j

Consequence Layer IV 5 (Output-Label nodes)

OLIV n ,Ln 5

wkIII,m Rule Layer III (Rule nodes)

RnIII3

cni

ci

ILiII,1

5

Output Linguistic Layer V (Output nodes)

ILiII, J

i

ILIIn ,1 ILIIn ,2 1

1

I1I

IiI

I nI1

x1

xi

xn1

ILIIn , J n 1

1

Condition Layer II (Input-Label nodes)

Input Linguistic Layer I (Input nodes)

Figure 1: Structure of pseudo outer-product-based fuzzy neural networks (POPFNN).

of the corresponding input xi . Neurons in the condition layer are called input-label nodes. Each input-label node ILII i,j represents the jth linguistic label of the ith input node from the input layer. The input label nodes constitute the antecedent of the fuzzy rules. Each input label node is represented by a membership function µi,j (x). Two commonly used fuzzy membership function are gaussian membership function, described by two parameters (vi,j , wi,j ), and trapezoidal membership function, described by a fuzzy interval formed by four parameters (αi,j , βi,j , γi,j , δi,j ). Neurons in the rule-based layer are called rule nodes. Each rule node RIII k represents an if-then fuzzy rule. Neurons in the consequence layer are called output label nodes. The output label node represents the lth linguistic label of the output ym . The output

210

K. Ang and C. Quek

label nodes constitute the conclusions of the if-then fuzzy rules. Each output label node is represented by a membership function µm,l (x) similar to the input label node where gaussian or trapezoidal membership functions can be used. The neurons in the output linguistic layer are called output nodes. The output node OV m represents the output linguistic variable of the output ym . The function of an output node is to perform defuzzification, where each conclusion expressed in terms of fuzzy sets in the consequence layer is converted to a single real number. 2.1 Learning Process of POPFNN. The learning process of POPFNN consists of three phases: fuzzy membership generation, fuzzy rule identification, and supervised fine-tuning. Various fuzzy membership generation algorithms can be used: learning vector quantization (LVQ; Kohonen, 1989), fuzzy Kohonen partitioning (FKP; Ang et al., 2003), or discrete incremental clustering (DIC; Tung & Quek, 2002a). Generally, the POP algorithm and its variant, LazyPOP, are used to identify the fuzzy rules (Quek & Zhou, 2001). Figure 2 shows the learning process of the POP algorithm in the identification of the output label nodes for a rule node RIII k . is linked to only one input label node from each Each rule node RIII k input node and only one output label node from each output node. The links of rule nodes to the antecedent in the condition layer and the consequent in the consequence layer are mathematically denoted as sets (CI , DV ) in this article. The antecedent of rule nodes is represented by a set CI = {c1 , c2 , . . . , ci , . . . , cn1 } where ci | ci ∈ ILII i is a condition variable. The set of condition labels that ci can assume is semantically represented by the set of II II II II input label nodes ILII i = {ILi,1 , ILi,2 , . . . , ILi,j , . . . , ILi,Ji }, but a computational notation of ILII i = {1, 2, . . . , j, . . . , Ji } is used in this article. Similarly, the consequent of rule nodes is represented by a set DV = {d1 , d2 , . . . , dm , . . . , dn5 }, where dm | dm ∈ OLIV m is a consequent variable. The set of consequent labels that dm can assume is semantically represented by the set of output label IV IV IV IV nodes OLIV m = {OLm,1 , OLm,2 , . . . , OLm,l , . . . , OLm,Lm }, but a computational notation of OLIV m = {1, 2, . . . , l, . . . , Lm } is used in this article. The specific links of the kth rule node RIII k to the antecedent in the condition layer and the consequent in the consequence layer are denoted as (CIk , DV k ). During are computationally repPOP learning, the consequents of rule node RIII k . A description of POP algorithm (Quek & Zhou, resented as weights wIII k 2001) follows: POP Algorithm • Step 1: Rule initialization. Initialize the links CIk of rule nodes to input label nodes for k = 1 · · · n3 such that all possible combinations of rules are created using equation 2.7. Initialize wIII k,m,l = 0 for k = 1 · · · n3 , m = 1 · · · n5 , l = 1 · · · Lm .

RSPOP

211 Output Linguistic Layer V (Output nodes)

OmV

OLIV m,1

Consequent

R1III

I Antecedent ci ∈Ck

II IL1,2

Consequence Layer IV (Output-Label nodes)

OLIV m, Lm

wk ,m,1 wk ,m,l wk ,m, Lm

d m ∈DVk

POP Learning

II IL1,1

OLIV m,l

c1

IL1,IIJ

1

RkIII

Weights

Rule Layer III (Rule nodes)

RnIII3 cni

ci

ILiII,1

ILiII, j

ILiII, J

IiI

I1I

wkIII,m

i

ILIIn ,1 ILIIn ,2 1

1

I nI1

ILIIn , J n 1

1

Condition Layer II (Input-Label nodes)

Input Linguistic Layer I (Input nodes)

Figure 2: Pseudo outer-product in the identification of output label nodes for kth rule node

• Step 2: Hebbian learning. Given the training data tuples {{X1 , Y1 }, {X2 , Y2 , . . . , {Xp , Yp }, . . . , {Xn , Yn }}, determine weights wIII k,m,l using pseudo other-product Hebbian learning (Quek & Zhou, 2001; Hebb, 1949) given in equation 2.4. For p = 1 · · · n, m = 1 · · · n5 , wIII k,m,l =

n3


III fk,p µm,l (ym,p )

(2.4)

p=1 IV III III where wIII k,m,l is the weight of the link between Rk and OLm,l ; fk,p is the firing strength of the pth instant of training data; µm,l is the membership value of the lth output label node for the mth output; and ym,p is the the mth element of the pth instant of Y. End for.

• Step 3: Rule formulation. Determine the link of each rule node RIII k to out= {d , d , . . . , d , . . . , d }, where d∈ put label nodes, denoted as DV 1 2 m n5 k IV OLIV and OL = {1, 2, . . . , l, . . . , L }. For k = 1 · · · n , m = 1 · · · n m 3 5: m m

212

K. Ang and C. Quek

a. Compute the consequence link with the highest weight using equation 2.5: III wIII k,m,lmax = max wk,m,l . l

(2.5)

b. Select or delete the link to the consequence of mth output using equation 2.6:  dm =

lmax ∅

if wIII k,m,lmax > 0 . if wIII k,m,lmax = 0

(2.6)

V V c. Delete rule node RIII k if Dk = ∅, that is, ∀dm ∈ Dk , dm = ∅. End for. d. Update n3 with the total number of remaining rules. III in step The initialization of rules in step 1 and the computation of fk,r I 2 involve an integer rule index k. The computation of k given Ck is given in equation 2.7. This equation is constructed such that each condition variable ci ∈ CIk construes a digit in a positional number system where c1 is the least significant digit and cn1 is the most significant digit. As the set of each i−1 condition labels ci can assume is ILII i = {1, 2, . . . , Ji }, the term p=0 Jp computes the associated weight of each digit, and the addition and subtraction of unity in the equation ensures that all possible combinations of CIk fall within the contiguous integer range of k = 1 · · · n3 ,

k=

n1
i=1

 (ci − 1) ×

i−1 

 Jp  + 1,

J0 = 1,

(2.7)

p=0

where Ji is the number of linguistic labels for the ith input; ci is the condition I variable of the ith input of rule Rk s. t. ci ∈ CIk , ci ∈ ILII i ; Ck is the set of condition variables of rule Rk , CIk = {c1 , c2 , . . . , ci , . . . , cn1 }; and ILII i is the set II of condition labels ci can assume, ILi = {1, 2, . . . , Ji }. 3 Preliminaries on Rough Set Theory Pawlak (1991) introduced rough set theory to deal with imprecise or vague concepts. A recent development of rough set theory and artificial intelligence is available in Curry (2003). A rough set is an approximation of a vague concept by a pair of precise concepts called the lower and upper approximations. The lower approximation is a description of the domain objects that are known with certainty to belong to the subset of interest, whereas the upper approximation is a description of the objects that possibly belong

RSPOP

213

to the subset. In addition, rough set theory provides rigorous mathematical techniques for discovering regularities in data. The approach in this letter applies two fundamental concepts of rough set knowledge reduction in the derivation of fuzzy rules: a reduct and the core. A reduct of knowledge is its essential part, which suffices to define all basic concepts occurring in the considered knowledge, whereas the core is the most important part of the knowledge. Central to this rough set approach is the concept of attribute dispensability in terms of indiscernability relations (Pawlak, 1991). An attribute R ∈ R is dispensable if it satisfies IND(R) = IND(R − {R}),

(3.1)

where IND(P) is the indiscernibility relation over P, which is the intersection of all equivalence relations belonging to P. The rough set knowledge reduction uses the value attribute pair to represent the knowledge of decision rules (Pawlak, 1991). When decision rules are represented as such, rough set theory then supplies the logical methods employing attribute dispensability and decision rule consistency for knowledge reduction and analysis. Because decision rules do not have the attributes of truth or falsity as opposed to formulas, consistency and inconsistency are used instead to describe decision rules in computational terms. The following sections elaborate on the knowledge representation system (KRS), attribute dispensability, and consistency from rough set theory. 3.1 Knowledge Representation System. To express mathematically how rough set theory is applied in knowledge reduction, Tarski’s style semantics of the decision logic (DL) language is used in Pawlak (1991), where the notions of a model and satisfiability are employed. Knowledge is represented as a model S = (U, A) where U is the universe of discourse and A is the set of attributes. The formula S is satisfied by an element x in U if and only if an extraction of attribute value from x belongs to the set of attribute A. In other words, an  object x ∈ U satisfies S if and only if a(x) = v, where a ∈ A and v ∈ V, V = Va and Va being the set of attribute value for a ∈ A. An atomic formula is represented as (a, v) in DL language for any a ∈ A and v ∈ V. A formula is expressed as (a1 , v1 ) ∧ (a2 , v2 ) ∧ · · · ∧ (ai , vi ) · · · ∧ (an , vn ),

(3.2)

where vi is the attribute constant value, vi ∈ Vai ; ai is the attribute constant, ai ∈ A; and ∧ is the conjunction connectives. In a decision rule system, A = {C ∪ D}, where C is the set of conditional attributes and D is the set of decision attributes. If P = {a1 , a2 , . . . , ai , . . . , an } and P ⊆ A, then the formula in equation 3.2 is called a P-basic formula (in short, P-formula). The satisfied in the system disjunction of all P-formulas S is represented as S P. If P = A, then S A is called the characteris-

214

K. Ang and C. Quek

Table 1: Example of the Knowledge Representation System Using Attribute Value Table Adopted from Pawlak (1991). U

a1

a2

a3

1 2 3 4 5 6

1 2 1 1 2 1

0 0 1 1 1 0

2 3 1 1 3 3

tic formula of S. The example in Table 1 illustrates how the knowledge is represented. Expressing the atomic formula (ai , v) in short as ai,v and omitting the conjunction symbol ∧, the characteristic formula of the knowledge in Table 1 is
A = a1,1 a2,0 a3,2 ∧ a1,2 a2,0 a3,3 ∧ a1,1 a2,1 a3,1 ∧ a1,2 a2,1 a3,3 ∧ a1,1 a2,0 a3,3 . (3.3) S

Now, given a decision rule φ → ψ where φ and ψ are the C-basic and Dbasic formula, respectively, then the decision rule φ → ψ is called a CD basic decision rule, or in short CD rule denoted as (C, D). Considering the knowledge in Table 1, assume that C = {a1 , a2 } and D = {a3 }, sets C and D associate the following CD decision rules: a1,1 a2,0 → a3,2 a1,2 a2,0 → a3,3 a1,1 a2,1 → a3,1 a1,2 a2,1 → a3,3 a1,1 a2,0 → a3,3

(3.4)

3.2 Rule Consistency. When a decision rule φ → ψ satisfies S, the decision rule is consistent in S if and only if for any decision rule φ  → ψ  in S, φ = φ  implies ψ = ψ  (Pawlak, 1991). Considering the knowledge in Table 1 with C = {a1 , a2 } and D = {a3 }, rule a1,1 a2,0 → a3,2 is inconsistent with rule a1,1 a2,0 → a3,3 . Thus, the decision rule base (C, D) is inconsistent. If either rule is removed, then (C, D) is consistent. 3.3 Attribute Reduction. Considering a consistent CD decision rule base (C, D), an attribute ai ∈ C is dispensable if and only if ((C − {ai }), D) is consistent (Pawlak, 1991). If all attributes ai ∈ C are indispensable in (C, D), then (C, D) is independent. The concept of a reduct of knowledge in rough set theory is the subset of attributes R ⊆ C, where (R, D) is independent

RSPOP

215

and consistent. The concept of the core of knowledge in rough set theory is the set of all indispensable attributes of knowledge (C, D), denoted as CORE(C, D). 3.4 Decision Rule Reduction. The concept of a reduct can also be applied to simplify dispensable conditions of decision rules similar to attribute reduction performed on the decision rule base (Pawlak, 1991). Before proceeding further, an addition denotation, /, is required. Considering a CD rule φ → ψ and Q ⊆ C, the notation φ/Q means the Q basic decision rule obtained from only those consistent in Q; others are removed. An attribute ai is dispensable in the decision rule φ → ψ if the removal of ai from C is also consistent. In other words, an attribute ai ∈ C is dispensable in φ → ψ that is consistent in S if and only if φ/(C − {ai }) → ψ is also consistent in S. 4 Rough Set–Based Pseudo Outer-Product Although the POP algorithm (Quek & Zhou, 2001) is a simple, easy-tounderstand one-pass learning algorithm that performs fast, reliable, and efficient consequence identification for fuzzy rules, it has to compute all possible rules during the learning process in equation 2.4 (Quek & Zhou, 1999). As a result, the fuzzy rules identified using POP usually consist of a large number of rules. The total number of rules identified in equation 2.3 also increases rapidly with an increase in the dimension of the input space or the number of input labels. As pointed out in Tikk and Baranyi (2003) and Nauck (2003), if the number of fuzzy rules identified is large, then any semantic interpretability of the fuzzy rules is lost. To resolve the problem of identifying a large number of rules, the lazy pseudo outer-product (LazyPOP) learning algorithm is proposed in Quek and Zhou (1999). Instead of considering all possible rules, this algorithm uses three user-defined threshold criteria—noise criterion (NC), ambiguity criterion (AC), and focus criterion (FC)—in the selection and pruning of fuzzy rules. Similarly, the RULEMAP algorithm in the generic self-organizing fuzzy neural network (GenSoFNN) uses two user-defined thresholds, ThresISP and ThresOSP , to govern the identification of fuzzy rules (Tung & Quek, 2002b). The introduction of heuristic thresholds in the identification and pruning of fuzzy rules serves the purpose of reducing the number of identified fuzzy rules. However, the accuracy of the neuro-fuzzy systems and the interpretability of the fuzzy rules identified are usually heavily dependent on the selection of these threshold values. Therefore, this motivates work to enhance POP using the rough set approach instead of using heuristic thresholds to identify a reduced number of fuzzy rules. The proposed rough set– based pseudo outer-product (RSPOP) learning algorithm consists of three parts: RSPOP rule identification, RSPOP attribute reduction, and RSPOP rule reduction. A description of the RSPOP learning algorithm follows with a detailed explanation of the rationale of the algorithm:

216

K. Ang and C. Quek

RSPOP Rule Identification • Step 1: Initialization. Initialize CIk and wIII k,m,l as in step 1 of POP. • Step 2: Hebbian learning. Given the training data tuples {{X1 , Y1 }, {X2 , Y2 }, . . . , {Xp , Yp }, . . . , {Xn , Yn }}, for p = 1 · · · n: a. Determine the input labels CIk,p = {c1 , c2 , . . . , ci , . . . , cn1 } for the pth instant of X with the maximum output for each input using II ci = jmax | max(oII i,jmax ,p ) = max(oi,j,p )∀ i = 1 · · · n1 j

ci ∈

CIk,p ,

(4.1)

where k is the index to rule nodes to be computed with CIk,p II using equation 2.7; oII i,j,p is the output of ILi,j for the pth instant of X; and jmax is the label index j = 1 · · · Ji for a particular ith input from ILII i,j that yields the highest output for the pth instant of X. b. Compute the firing strength of the most influential rule node RIII k using III fk,p = min(oII i,ci ),

(4.2)

i

I II where oII i,ci is the output of ILi,ci , ci ∈ Ck,p for the pth instant of X. c. Compute the output labels DV k,p = {d1 , d2 , . . . , dm , . . . , dn5 } for the pth instant of Y with the maximum membership value using

dm = lmax | µm,Jmax (ym,p ) = max µm,l (ym,p )∀ m = 1 · · · n5 , l

dm ∈ DV k,p ,

(4.3)

where µm,l is the membership function of OLIV m,l for the pth instant of Y; ym,p is the mth element of the pth instant of Y; and Imax is the label index l = 1 · · · Lm for a particular mth output from OLIV m,l that yields the maximum membership value for the pth instant of Y. d. Update the weights wIII k,m,dm of the most influential rule node III Rk using a Hebbian learning rule (Hebb, 1949) given in III V wIII k,m,dm = fk,p × µm,dm (ym,p )∀ m = 1 · · · n5 , dm ∈ Dk,p , (4.4) IV III where wIII k,m,dm is the weights of Rk to OLm,dm with the highest III membership value; fk,p is the firing strength of the most influ-

ential rule node RIII k for the pth instant of X from equations 4.1

RSPOP

217

to 4.3; and µm,dm is the maximum membership value of the mth output for the pth instant of Y from equation 4.3. End for. • Step 3: Rule formulation. Determine the link of each rule node RIII k to output label nodes and update n3 with the total number of rules as in step 3 of POP. RSPOP Attribute Reduction • Step 1: Baseline. Given the training data tuples {{X1 , Y1 }, {X2 , Y2 }, . . . , {Xp , Yp }, . . . , {Xn , Yn }} and the rules in POPFNN before reduction as CI , DV ), compute the baseline objective measure of CD rule base CI , DV ). For p = 1 · · · n: Compute the POPFNN consequence output for pth instant of X using COm,l,p = oIV m,l,p ∀ m = 1 · · · n5 , l = 1 · · · Lm .

(4.5)

End for. • Step 2: Reduction. Perform input attribute reduction for each attribute ci of the rule base. For i = 1 · · · n1 : a. Compute the consistency of the CD rule base ((CI − {ci }), DV ). ((CI − {ci }), DV ) is inconsistent if the following equation is true when rules with the same conditions ignoring attribute ci infer different consequents: I V ∀Rk : ((CIk − {ci }), DV k ), ∃Rk : ((Ck − {ci }), Dk ) | V ((CIk − {ci }) = (CIk − {ci })) ∧ (DV k = Dk ) ∀ k,

k  = 1 · · · n3 .

(4.6)

b. If ((CI − {ci }), DV ) is consistent, compute the objective measure of the CD rule base ((CI − {ci }), DV ) using IV COreduct m,l,p = om,l,p ∀ p = 1 · · · n, m = 1 · · · n5 , l = 1 · · · Lm . (4.7)

c. Detect the deterioration of the objective measure by comparing the objective measure computed in step 2b with the baseline objective measure. The objective measure is deteriorated if the following equation is true when one of the correct consequent outputs (COm,l,p , l = dm ) of using reduct is decreased. The object measure is also deteriorated if the incorrect consequent output (COm,l,p , l = dm ) is greater than that of a correct consequent

218

K. Ang and C. Quek

output: V ∃p, ∃m s.t. ∀l((COreduct m,l,p < COm,l,p ), l = dm ∈ Dk,p ) ∨ reduct V ((COreduct m,l,p > COm,l,p ) ∧ (COm,l,p > COm,dm ,p ), l = dm ∈ Dk,p )

∀p = 1 · · · n, m = 1 · · · n5 , l = 1 · · · Lm .

(4.8)

d. If there is no deterioration of objective measure, then allow the reduction of attribute ci using the two following equations, remove duplicated rules, and recompute n3 : (CI , DV ) = ((CI − {ci }), DV ) COm,l,p =

COreduct m,l,p

(4.9)

∀ p = 1 · · · n, m = 1 · · · n5 ,

l = 1 · · · Lm .

(4.10)

End for. RSPOP Rule Reduction • Step 1: Baseline. Given the training data tuples {{X1 , Y1 }, {X2 , Y2 }, . . . , {Xp , Yp }, . . . , {Xn , Yn }} and the rules in POPFNN before reduction as (CI , DV ), compute the baseline objective measure COm,l,p of CD rule base (CI , DV ) as in step 1 of RSPOP Attribute reduction. • Step 2: Reduction. Perform rule reduction for each attribute ci of each RIII k . For k = 1 · · · n3 , i = 1 · · · n1 : V I a. Based on the picked rule RIII k : (Ck , Dk ) and the picked condition attribute ci , denoting CD rule as φ → ψ, compute the consistency of all CD rules φ/(CIk − ci ) → ψ using

∃Rk : ((CIk − {ci }), DV k ) | V ((CIk − {ci }) = (CIk − {ci })) ∧ (DV k = Dk )

∀ k  = 1 · · · n3 .

(4.11)

b. If the CD rules φ/(CIk −ci ) → ψ are consistent, remove attribute ci from these CD rules and compute the objective measure of POPFNN using these reduced rules with equation 4.7. c. Detect the deterioration of objective measures of using reduced rules if equation 4.8 is true. d. If there is no deterioration of objective measures, allow the reduction of rules, and remove duplicated rules. End for. The first part, RSPOP rule identification, addresses the inherent problem of the POP algorithm in considering all possible rules. In equation 2.4 of the

RSPOP

219

III POP algorithm, the weight wIII k,m,l is dependent on the firing strength fk,p for

the pth instant of training data {X, Y}. In the computation of oIV m,l , the highest IV III from RIII results in the largest o . This implies that the kth rule with fk,p m,l k III the highest firing strength fk,p is the most influential rule for the pth instant of training data {X, Y}. However, it is not necessary to consider all possible III rules to determine the rule index k to the rule node RIII k with the highest fk,p . III The firing strength fk,p for the pth instant of training data {X, Y} is computed from the same equation given in III fk,p = min(oII i,j ),

(4.12)

i

II where oII i,j is the output of the input label node ILi,j that forms the antecedent conditions for the ith input to the kth fuzzy rule for the pth instant of X. Expanding equation 4.12 with the objective of getting a maximum fkIII gives III II II II = min(max(oII fk,p 1,c1 ), max(o2,c2 ), . . . , max(oi,ci ), . . . , max(on1 ,cn )), (4.13) 1

where ci is the label index j = 1 · · · Ji for a particular ith input from ILII i,j that yields the highest output for the pth instant of X. Therefore, the maximum III firing strength fk,p for the pth instant of training data tuple {X, Y} can be computed from the minimum of all the maximum output of the condition layer for each input. This allows the identification of the most influential rule I node RIII k with the conditions Ck,p = {c1 , c2 , . . . , ci , . . . , cn1 } where ci ∈ ILi for the pth instant of training data tuple {X, Y} in equation 4.1 of RSPOP rule identification step 2a. The time complexity of the reduction step 2a of both RSPOP attribute reduction and RSPOP rule reduction is O(n23 ), which is dependent on the number of fuzzy rules. If POP is used, then the reduction step inherits the problem from POP in considering a large number of rules given in equation 2.2. RSPOP rule identification improves on POP such that given the pth instant of training data tuple {X, Y}, RSPOP identifies only one most influential rule. Because this rule may already have been identified from the previous instant of the training data, the total number of fuzzy rules identified using RSPOP rule identification is thus bounded from above as given in n3 ≤ n,

(4.14)

where n is the number of training tuples and n3 is the number of rules identified. Thus, RSPOP rule identification improves on the time complexity of POP by reducing the number of rules considered by POP, which is exponentially

220

K. Ang and C. Quek

dependent on the dimension of the input space in equation 2.2, to an upper bound as given in equation 4.14. Comparing RSPOP against LazyPOP, the former identifies only one influential rule for a given training data tuple, but the latter identifies a number of legitimate rules based on a heuristic noise criterion. This gives RSPOP a stronger advantage in identifying a significantly reduced number of fuzzy rules for high-dimensional modeling, which improves the interpretability of the fuzzy rules identified. To allow the reduction of attributes and rules in POPFNN, the firing strength of rules of POPFNN has to accommodate the removal of condition attributes. Therefore, the firing strength of a rule RIII k in the rule layer of POPFNN for handling reduced attributes and rules is modified to I fkIII = min(oII i,ci ), ci ∈ Ck , ci = ∅, ∀ i = 1 · · · n1 . i

(4.15)

In the reduction of identified fuzzy rules, the effectiveness of a reduced rule base must not deteriorate compared to the rule base prior to reduction. Therefore, an objective measure is vital in assessing the effectiveness of the neuro-fuzzy system before and after reduction. In rough set theory, there can be only one core of a knowledge, but there can be many possible reducts (Pawlak, 1991). This objective measure thus serves the purpose of identifying the reducts that improve but do not deteriorate the performance of the neuro-fuzzy system. The fundamental objective of a fuzzy rule identification algorithm is to identify rules that infer correct consequents based on the conditions and consequents learned from training data. Thus, an objective measure can be construed based on the correct inference of consequents from training data. Given the training data {{X1 , Y1 }, {X2 , Y2 }, . . . , {Xp , Yp }, . . . , {Xn , Yn }}, the lth consequent of the mth output for the pth instant of training data {X, Y} is oIV m,l , which is computed in the consequence layer. The correct consequent of the mth output is computed from the maximum membership of output labels OLIV m,l given ym,p . Therefore, an improvement in the objective measure is an increase in oIV m,lmax | µm,lmax (ym,p ) = max µm,l (ym,p ). A deterioration in the l

IV IV objective measure is an increase in oIV m,l | l = lmax , om,l > om,lmax . This objective measure is used in RSPOP attribute reduction and RSPOP rule reduction to assess the performance of POPFNN before and after reduction. The application of attribute reduction and rule reduction in RSPOP is as described in section 3. Section 5 gives a more detailed explanation of the operation of RSPOP in the process of identifying dispensible attributes and interpretable rules. The next section presents experimental results using published data sets and a real-world application involving highway traffic flow prediction to evaluate the effectiveness of using RSPOP to identify the fuzzy rules in pseudo outer-product–based fuzzy neural networks using compositional rule of inference and singleton fuzzifier (POPFNN-CRI(S)) architecture.

RSPOP

221

5 Numerical Experiments Two sets of experiments are performed to evaluate the performance of the proposed RSPOP algorithm in the identification of fuzzy rules in the POPFNN-CRI(S) architecture (Ang et al., 2003): (1) three sets of Nakanishi data set extracted from published papers (Nakanishi, Turksen, & Sugeno, 1993; Sugeno & Yasukawa, 1993) and (2) highway traffic modeling and prediction.

5.1 Nakanishi Data Set. This section outlines the experiments performed to evaluate the effectiveness of fuzzy rules identified using the proposed RSPOP algorithm in the POPFNN-CRI(S) architecture (abbreviated as RSPOP-CRI) on three data sets: a nonlinear system, the human operation of a chemical plant, and the daily price of a stock in a stock market. These three data sets were used by Nakanishi et al. (1993) to evaluate six reasoning methods: Mamdani version of Zadeh’s CRI, Turksen’s version of interval-valued CRI, Turksen’s versions of point-valued and interval-valued AARS, and Sugeno’s version of position and gradient type of reasoning. They are also used in the classic paper by Sugeno and Yasukawa (1993), where qualitative modeling based on fuzzy logic is proposed. A qualitative model of a system is one built with numerical input-output data without a prior knowledge of a system. The use of fuzzy-logic-based qualitative modeling instead of nonfuzzy modeling stems from the fact that system behavior can be qualitatively described using natural language where a precise mathematical model of these complex systems cannot be obtained (Sugeno & Yasukawa, 1993). Each of these three data sets is split into three groups: A, B, and C (Nakanishi et al., 1993). In the following experiments, groups A and B of all data sets are used as the training data sets, and group C is used as the ex ante test data sets. The experimental results are compared against POPFNN-CRI(S) using the same gaussian membership functions generated in RSPOP-CRI but with fuzzy rules identified using POP (abbreviated as POP-CRI) and other neuro-fuzzy systems: adaptive-network–based fuzzy inference systems (Jang, 1993) with subtractive clustering (Chiu, 1994; denoted as ANFIS), evolving fuzzy neural networks (denoted as EFuNN; Kasabov, 2001), and dynamic evolving neural-fuzzy inference system (denoted as DENFIS; Kasabov & Song, 2002). ANFIS is configured with a range of influence of 0.5 for subtractive clustering, while EFuNN and DENFIS are both configured with default parameters.

5.1.1 Example 1: A Nonlinear System. In this experiment, POPFNNCRI(S) is used to model the nonlinear system. Table 2 shows the centroid v and width w of the gaussian membership function generated using a modified variant of the learning vector quantization (Kohonen, 1989) al-

222

K. Ang and C. Quek

Table 2: Gaussian Membership Function of Example 1 Generated for POPFNNCRI(S). ILII

Attribute

OLIV

Label j

vi,j

wi,j

v2,j

w2,j

v3,j

w3,j

v4,j

w4,j

v1,j

w1,j

1 2 3 4

1.599 2.535 3.527 4.472

0.216 0.561 0.567 0.209

1.293 2.206 4.412

0.140 0.548 0.329

1.459 2.615 4.069 4.870

0.209 0.693 0.480 0.054

1.185 2.261 3.599 4.340

0.105 0.646 0.445 0.102

0.498 0.498 0.718 0.806

1.499 2.330 3.526 4.869

gorithm (abbreviated as MLVQ) from the training data set.1 A total of 4, 3, 4, 4 clusters from the input data X1 to X4 , respectively, and a total of 4 clusters from the output data Y1 are generated. The gaussian membership functions generated form the input label and output label nodes of POPFNN-CRI(S). Table 3 shows the initial and reduced rules identified using RSPOP with the same training data set. The feature selection method in Nakanishi et al. (1993) discards the inputs X3 and X4 , whereas RSPOP discards the input X3 and only partially discards X4 in some of the rules. A detailed explanation on the attribute and rule reduction process of RSPOP is provided here in Table 3. From the identified input and output membership functions, the sets CI = {c1 , c2 , c3 , c4 } and DV = {d1 } are used to represent the condition and consequent attributes, respectively. The attribute values c1 ∈ {1, 2, 3, 4}, c2 ∈ {1, 2, 3}, c3 ∈ {1, 2, 3, 4}, c4 ∈ {1, 2, 3, 4}, and d1 ∈ {1, 2, 3, 4} are used to represent the 4, 3, 4, 4, 4 gaussian membership functions for condition variables c1 to c4 and consequent variable d1 , respectively. RSPOP identified an initial 22 rules before attribute and rule reduction. In the process of attribute reduction, one reduct ((CI − {c3 }), DV ) that results in consistent rules is identified. Using this reduct, the computation of objective measure resulted in no deterioration, and thus this reduct is used and duplicated fuzzy rules are removed. An example of duplicated fuzzy rules using this reduct from the initial rules in Table 3 is R9 and R10 . After attribute reduction, the attribute c3 is redundant, and the number of rules is reduced from 22 to 20 after removing duplicated rules. In the process of rule reduction, several groups of rules are identified with dispensable condition. An example of a pair of rules with dispensable condition c1 from the attribute reduced fuzzy rules in Table 3 is R5 and R9 . However, the computation of the objective measure using this pair of reduced rules resulted in deterioration; thus, this

1 The authors’ work on the generation of interpretable fuzzy membership functions will be communicated in a separate article. It suffices here to consider the algorithm as a modified variant of learning vector quantization algorithm as used in POPFNN-TVR (Zhou & Quek, 1996).

RSPOP

223

Table 3: Fuzzy Rules of Example 1 Identified Using RSPOP. Rules From RSPOP Rule Identification

Rules After RSPOP Attribute Reduction DV k

CIk

Rules After RSPOP Rule Reduction DV k

CIk

DV k

CIk

k

c1

c2

c3

c4

d1

k

c1

c2

c3

c4

d1

k

c1

c2

c3

c4

d1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

2 2 1 4 3 3 1 3 2 2 4 3 1 4 1 3 3 1 2 2 1 4

3 1 2 2 2 3 1 1 2 2 3 2 3 3 2 2 3 1 1 2 2 3

1 2 3 4 1 1 2 2 2 3 3 4 1 1 2 2 2 3 4 1 2 2

1 1 1 1 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 4 4 4

1 3 2 1 2 1 2 2 2 2 1 2 2 1 3 1 1 3 2 2 3 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

2 2 1 4 3 3 1 3 2 4 1 4 1 3 3 1 2 2 1 4

3 1 2 2 2 3 1 1 2 3 3 3 2 2 3 1 1 2 2 3

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 3 4 4 4

1 3 2 1 2 1 4 2 2 1 2 1 3 1 1 3 2 2 3 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

2 2 1 4 3 3 1 3 2 4 1 1 3 1 2 2 1

3 1 2 2 2 3 1 1 2 3 3 2 2 1 1 2 2

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1 1 1 1 2 0 2 2 2 0 3 3 3 3 3 4 4

1 3 2 1 2 1 4 2 2 1 2 3 1 3 2 2 3

pair of rules is not reduced. As the rule reduction process continues, groups of reducible rules that resulted in no deterioration of objective measures are used with the dispensable conditions set to 0, and the number of rules is reduced from 20 to 17. The numbers 0–4 that represent condition and consequent labels in Table 3 do not show that the fuzzy rules identified are interpretable. To illustrate the intuitiveness of the fuzzy rules identified, a mapping of semantic labels to each condition and consequent labels is required. As formulated in Zadeh (1975a, 1975b, 1975c), a linguistic variable is characterized by a quintuple (L, T(L), U, G, M), where L is the name of the variable; T(L) is the linguistic term set of L; U is a universe of discourse; G is a syntactic rule that generates T(L); and M is a semantic rule that associates each T(L) with its meaning. Here, the names L of the input and output linguistic variables of the data set are {X1 , X2 , X3 , X4 } and {Y}, which are represented by condition and consequent variables {c1 , c2 , c3 , c4 } and {d1 }, respectively. The linguistic terms are represented by the attribute values, which are {1, 2, 3, 4} for attribute variables c1 , c2 , c3 , d1 and {1, 2, 3} for attribute variable c2 . The meaning of each linguistic label M is represented by its gaussian membership function in Table 2. A mapping of semantic labels such as T(·) = {Low,Medium,High,Very High} reveals the intuitiveness of the rules identified in Table 3, as shown in Figure 3 and Table 4.

membership function µ4,j(x4)

K. Ang and C. Quek

1

T(2) Medium

T(1) Low

T(3) High

T(4) Very High

0.8 0.6 0.4 0.2 0 1

1

2

3

4

input value x1

T(1) T(2) Low Medium

5

T(3) High

0.8 0.6

⇒

0.4 0.2 0 0

1

1

2

3

4

5

input value x2

T(1) Low

6

membership function µ1,l(y1)

membership function µ2,j(x2)

membership function µ1,j(x1)

224

1

T(1) T(2) T(3) T(4) Low Medium High Very High

0.8 0.6 0.4 0.2 0 0

2

4

output value y1

6

8

T(3) T(4) High Very High

T(2) Medium

0.8 0.6 0.4 0.2 0 0

1

2

3

input value x4

4

5

Figure 3: Semantic interpretation of the fuzzy sets in Table 2.

After the fuzzy rules are identified, the output using the test data set from example 1 is then computed. Figure 4 shows the experimental results obtained from RSPOP-CRI compared to POP-CRI and other neuro-fuzzy systems. Consolidated experimental results on the three data sets in terms of square of the Pearson product-moment correlation value (R2 ) and mean square error (MSE) are given in Table 9. 5.1.2 Example 2: Human Operation of a Chemical Plant. In this experiment, the POPFNN-CRI(S) is used to model the human operation of a chemical. Table 5 shows the centroid v and width w of the gaussian membership functions generated using MLVQ from the training data set. A total of 4, 5,

RSPOP

225

Table 4: Semantic Interpretation of the Fuzzy Rules in Table 3. R1 : R4 : R6 : R10 : R13 :

IF X1 is medium IF X1 is very high IF X1 is high IF X1 is very high IF X1 is high THEN Y is low IF X1 is low IF X1 is high IF X1 is high IF X1 is medium IF X1 is low IF X1 is medium IF X1 is medium THEN Y is medium IF X1 is medium IF X1 is low IF X1 is low IF X1 is low THEN Y is high IF X1 is low

R3 : R5 : R8 : R9 : R11 : R15 : R16 : R2 : R12 : R14 : R17 : R7 :

AND X2 AND X2 AND X2 AND X2 AND X2

is high is medium is high is high is medium

AND X4 is low AND X4 is low

AND X2 AND X2 AND X2 AND X2 AND X2 AND X2 AND X2

is medium is medium is low is medium is high is low is medium

AND X4 AND X4 AND X4 AND X4 AND X4 AND X4 AND X4

is low is medium is medium is medium is high is high is very high

AND X2 AND X2 AND X2 AND X2

is low is medium is low is medium

AND X4 AND X4 AND X4 AND X4

is low is high is high is very high

AND X2 is low

AND X4 is high

AND X4 is medium

5

Data Value

4

Actual RSPOP-CRI POP-CRI Sugeno (R-G) Mamdani Turksen (IVCRI) ANFIS

3

2

1

0 0

5

10

15

20

25

Data Instance Figure 4: Experimental results of RSPOP-CRI on test data of example 1 compared against POP-CRI and other neuro-fuzzy systems.

226

K. Ang and C. Quek

Table 5: Gaussian Membership Function of Example 2 Generated for POPFNNCRI(S). Attribute Label j 1 2 3 4 5 6

ILII vi,j 4.615 5.519 5.950 6.522

wi,j

v2,j

w2,j

v3,j

w3,j

OLIV v4,j

w4,j

v5,j

w5,j

v1,j

0.542 −0.244 0.055 693.5 1036.5 −0.389 0.111 −0.100 0.060 909.47 0.258 −0.152 0.037 2421.0 809.9 −0.204 0.062 0.000 0.060 2579.7 0.258 −0.090 0.037 3770.9 809.9 −0.100 0.060 0.100 0.056 3870 0.344 0.001 0.054 6742.3 1782.8 0.000 0.060 0.193 0.056 6757.8 0.154 0.092 0.100 0.056 0.194 0.056

w1,j 1002.2 774.14 774.14 1732.7

4, 6, 4 clusters from the input data X1 to X5 , respectively, and a total of 4 clusters from the output data Y1 are generated. Table 6 shows the initial and reduced rules identified using RSPOP. RSPOP identified an initial 24 rules, which reduced to 14 rules after attribute reduction and remained at 14 rules after rule reduction. The feature selection method in Nakanishi et al. (1993) discards the inputs X2 , X4 , and X5 , whereas RSPOP discards the inputs X1 , X2 , and X5 . Figure 5 shows the experimental results obtained from RSPOP-CRI compared against POP-CRI and other neuro-fuzzy systems. Consolidated experimental results on the three data sets are given in Table 9. Similar to example 1, a term set T(·) has to be used to map the attribute values semantically. The semantic interpretation of the fuzzy rules identified is omitted henceforth since translating the gaussian membership functions generated in Table 5 and the fuzzy rules identified in Table 6 is a rather intuitive process. 5.1.3 Example 3: Daily Data of a Stock in a Stock Market. In this experiment, the POPFNN-CRI(S) is used to model the daily data of a stock market. Table 7 shows the centroid v and width w of the gaussian membership function generated using MLVQ from the training data set. A total of three to five clusters from the input data X1 to X10 and a total of four clusters from the output data Y1 are generated. Table 8 shows the initial and reduced rules identified using RSPOP. RSPOP identified an initial 50 rules and reduced to 23 rules after attribute and rule reduction. The feature selection method in Nakanishi et al. (1993) discards the inputs X1 , X2 , X3 , X6 , X7 , X9 , and X10 , whereas RSPOP discards the inputs X1 , X2 , X3 , X6 , X10 and partially discards X5 , X7 , X8 , and X9 in some of the rules. Figure 6 shows the experimental results obtained from POPFNN-CRI(S) using the proposed RSPOP compared against POPFNNCRI(S) using POP and other neuro-fuzzy systems. Consolidated experimental results on the three data sets are given in Table 9. 5.1.4 Discussion. Table 9 shows the consolidated experimental results of examples 1 to 3 in terms of square of the Pearson product-moment cor-

RSPOP

227

Table 6: Fuzzy Rules of Example 2 Identified Using RSPOP. Rules from RSPOP Rule Identification

Rules After RSPOP Attribute and Rule Reduction DV k

CIk

DV k

CIk

k

c1

c2

c3

c4

c5

d1

k

c1

c2

c3

c4

c5

d1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

1 2 1 1 1 1 3 2 2 1 1 1 2 2 3 1 4 4 1 3 4 1 4 1

4 4 4 1 4 4 4 1 2 4 3 4 1 5 5 2 1 4 4 2 3 4 3 4

4 2 4 3 4 4 3 2 3 4 4 4 3 3 3 4 1 1 4 1 1 4 1 4

2 3 3 4 4 5 6 3 3 3 4 4 5 1 2 2 3 3 3 4 4 4 5 3

1 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4

4 2 4 4 4 4 3 2 3 4 4 4 3 3 3 4 1 1 4 1 1 4 1 4

1 2 3 4 5 6 7 8 9 10 11 12 13 14

0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0

4 2 4 3 4 4 3 3 3 3 3 1 1 1

2 3 3 4 4 5 6 3 5 1 2 3 4 5

0 0 0 0 0 0 0 0 0 0 0 0 0 0

4 2 4 4 4 4 3 3 3 3 3 1 1 1

Table 7: Gaussian Membership Function of Example 3 Generated for POPFNNCRI(S). ILII

Attribute Label j

vi,j

wi,j

v2,j

w2,j

v3,j

w3,j

v4,j

w4,j

v5,j

w5,j

1 2 3 4 5

−0.052 0.065 0.181 0.296 0.412

0.070 0.069 0.069 0.069 0.069

−0.029 0.108 0.280 0.378

0.082 0.082 0.059 0.059

−12.231 −4.405 10.460 19.760 26.719

4.696 4.696 5.580 4.175 4.175

−6.541 10.980 25.491

10.512 8.707 8.707

−3.429 −2.388 −0.898 −0.127 0.922

0.625 0.625 0.462 0.462 0.629

Attribute Label j 1 2 3 4 5

ILII v6,j

w6,j

v7,j

−13.671 −4.518 0.623 7.815

5.492 3.085 3.085 4.315

−8.332 −4.692 −1.130 3.489

w7,j

v8,j

OLIV w8,j

v9,j

w9,j

v10,j

w10,j

v1,j

2.184 −17.637 4.161 −18.024 4.585 −19.222 4.658 −19.122 2.137 −10.702 4.161 −10.382 3.441 −11.460 4.658 2.972 2.137 −3.322 3.301 −4.647 2.481 −2.962 3.131 23.592 2.771 2.180 3.301 −0.512 2.481 2.256 2.671 32.693 7.818 3.382 5.248 3.456 6.708 2.671

w1,j 13.256 12.372 5.461 5.461

228

K. Ang and C. Quek

10000 8000

Data Value

6000

4000 Actual RSPOP-CRI POP-CRI Sugeno (R-G) Mamdani Turksen (IVCRI) DENFIS

2000 0

-2000 0

5

10

15

20

25

30

35

Data Instance Figure 5: Experimental results of RSPOP-CRI on test data of example 2 compared against POP-CRI and other neuro-fuzzy systems.

50 40

Data Value

30 20 10 0 Actual RSPOP-CRI POP-CRI Sugeno (R-G) Mamdani Turksen (IVCRI) ANFIS

-10 -20 -30 -40 0

10

20

30

40

50

Data Instance Figure 6: Experimental results of RSPOP-CRI on test data of example 3 compared against POP-CRI and other neuro-fuzzy systems.

RSPOP

229

Table 8: Fuzzy Rules of Example 3 Identified Using RSPOP. Rules from RSPOP Rule Identification DV k

CIk k 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 .. . 41 42 43 44 45 46 47 48 49 50

c1 4 4 4 4 4 3 1 3 5 2 5 1 4 1 2 1 1

c2 1 2 2 1 1 1 1 1 2 1 1 1 3 1 1 1 1

c3 3 3 3 3 4 2 1 3 4 1 5 2 3 1 2 1 1

c4 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1

c5 2 2 3 1 2 4 3 4 4 4 5 3 4 3 4 3 5

4 5 5 4 5 4 5 1 5 5

4 2 2 4 4 4 4 2 4 3

3 5 5 3 5 3 5 2 3 4

3 1 1 2 3 2 3 1 3 2

5 5 5 5 5 5 5 5 5 5

.. .

c6 1 2 2 2 4 3 3 2 2 3 3 3 3 3 3 3 3

c7 3 2 3 1 4 2 3 2 3 3 4 2 2 3 3 4 3

c8 3 1 3 3 2 3 3 4 4 4 4 3 3 3 3 3 4

c9 1 2 2 3 3 3 3 3 3 3 3 4 4 4 4 4 4

c10 1 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 4 3 3 3 3 3 4

3 4 3 3 4 3 3 4 4 4

4 4 4 4 4 5 5 5 5 5

5 5 5 5 5 5 5 5 5 5

4 4 5 5 5 5 5 5 5 5

d1 2 2 1 2 2 2 2 2 2 2 2 2 4 2 2 2 2 .. . 1 2 2 1 1 1 1 2 1 1

relation value (R2 ) and MSE. Comparing the modeling accuracy in terms of MSE, POP-CRI outperformed the rest on example 1, DENFIS outperformed the rest on example 2, and RSPOP-CRI outperformed the rest on example 3. From Table 9, ANFIS and DENFIS yielded significantly accurate performance on all three data sets. As these two neuro-fuzzy systems are based on the TSK model (Sugeno & Kang, 1988; Takagi & Sugeno, 1985) used for precise fuzzy modeling, these models generally yield decreased interpretability but increased accuracy (Casillas et al., 2003). In contrast, POPFNN-CRI(S) is based on the Mamdani model (Mamdani & Assilian, 1975) used for linguistic fuzzy modeling, which is focused on interpretability. The objective of the comparison with the recent TSK-based models is to obtain a quality check on the performance of RSPOP-CRI since interpretability is not directly comparable between these two models.

230

K. Ang and C. Quek

Table 8, continued. Rules After RSPOP Attribute and Rule Reduction DV k

CIk k 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

c1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

c2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

c3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

c4 1 1 2 1 1 1 1 1 1 1 1 1 1 1 3 3 1 3 1 2 1 1 2 2 2 2 3 1 2

c5 2 2 3 1 2 4 3 4 5 0 4 4 2 4 5 5 4 4 5 5 4 4 4 4 5 5 5 5 5

c6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

c7 3 2 3 1 4 2 3 3 0 0 2 3 4 4 4 0 4 4 2 4 2 3 3 3 3 3 3 4 4

c8 3 1 3 3 2 0 3 4 4 3 3 0 1 5 5 4 4 4 5 5 2 3 3 4 4 5 5 5 5

c9 1 2 2 3 3 3 3 3 0 4 4 4 5 3 0 0 4 4 4 4 5 5 5 5 5 5 5 5 5

c10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

d1 2 2 1 2 2 2 2 2 2 2 4 2 2 2 1 1 2 1 2 1 1 2 3 2 1 1 1 2 1

Table 9: Consolidated Experimental Results on Nakanishi Data Sets. Example 1 2 3

R2 MSE R2 MSE R2 MSE

POP-CRI

RSPOP-CRI

Sugeno (P-G)

Mandani

Turksen (IVCRI)

0.877 0.270 0.946 5.630 × 105 0.733 76.221

0.856 0.383 0.983 2.124 × 105 0.922 24.859

0.845 0.467 0.990 1.931 × 106 0.700 168.90

0.490 0.862 0.937 6.580 × 105 0.865 40.84

0.609 0.706 0.993 2.581 × 105 0.661 93.02

ANFIS

EFuNN

DENFIS

0.853 0.286 0.780 2.968 × 106

0.720 0.566 0.946 7.247 × 105 0.756 72.542

0.805 0.411 0.995 5.240 × 104

0.875 38.062

0.810 69.824

RSPOP

231

Table 10 shows a benchmark of the number of rules and learning performance of RSPOP-CRI against POP-CRI on the Nakanishi data sets performed on a Pentium 4 2.0 GHz Dell PC with 512 Mb RAM. The table shows that as the dimensionality of data sets A to C increases from 4 to 10, the number of fuzzy rules identified using RSPOP remains bounded from above by the number of tuples of the training data. In contrast, the number of fuzzy rules identified using POP increases exponentially from 192 to 3 million. From the results on example 1 in Table 10, the number of fuzzy rules is notably reduced from 192 for POP-CRI to 17 for RSPOP-CRI, but the accuracy in terms of MSE is slightly compromised from 0.270 to 0.383. A quality control check on the accuracy with other neuro-fuzzy systems in Table 9 showed an exceptional performance in terms of MSE for POP-CRI, whereas RSPOP-CRI yielded comparable performance with ANFIS, DENFIS, and EFuNN. This shows that the interpretability of the fuzzy rules in POPFNN-CRI(S) identified using RSPOP improved, but accuracy is not significantly compromised on example 1. From the results on example 2 in Table 10, the number of fuzzy rules identified is significantly reduced from 1920 for POP-CRI to 14 for RSPOP-CRI, and the accuracy in terms of MSE improved marginally from 5.630 × 105 to 2.124 × 105 . A quality control check on the accuracy with other neuro-fuzzy systems in Table 9 showed exceptional performance in terms of MSE for DENFIS, whereas RSPOP-CRI yielded comparable performance with EFuNN. This shows that the interpretability of the fuzzy rules in POPFNN-CRI(S) identified using RSPOP improved significantly with a marginal improvement in accuracy on example 2. From the results on example 3 in Table 10, the number of fuzzy rules identified is extensively reduced from 3 million for POP-CRI to 29 for RSPOPCRI, and the accuracy in terms of MSE significantly improved from 76.2 to 24.9. With the increase in the dimension of the input space from 5 in example 2 to 10 in example 3, the number of rules identified using POP increases exponentially from 1920 to 3 million. The POP algorithm, which uses an outer product form of Hebbian learning, has to compute all possible rules given in equation 2.2 during the learning process in equation 2.4. The exponential increase in the number of rules identified with the increase in the dimension of input space and/or the number of input labels has been reported in Quek and Zhou (2001). Although the LazyPOP algorithm was proposed in Quek and Zhou (2001) to reduce the number of rules identified, it relies on heuristic thresholds. Thus, the accuracy and interpretability of the rules identified are heavily dependent on the selection of these threshold values. In contrast, RSPOP enhances POP using the rough set reduction and identifies the reduced number of fuzzy rules without the use of heuristic thresholds. A quality control check on the accuracy with other neuro-fuzzy systems in Table 9 showed superior performance in terms of MSE for RSPOP-CRI. This shows that both the interpretability and the accuracy of POPFNN-CRI(S) using RSPOP to identify fuzzy rules improved significantly on example 3.

232

K. Ang and C. Quek

In addition, Table 10 shows that the time taken to train RSPOP-CRI compared to POP-CRI is reduced from 1,082,866 ms to 184 ms. This shows that the proposed RSPOP algorithm executes significantly faster than the POP algorithm on example 3. Therefore, these experimental results show that as the dimensionality of the data set increases, significant improvement in both the interpretability and accuracy, as well as computation complexity, is obtained for POPFNN-CRI(S) using the proposed RSPOP algorithm to identify fuzzy rules compared to the POP algorithm. 5.2 Traffic Flow Prediction. This section outlines the experiments performed to evaluate the effectiveness of the fuzzy rules identified using the proposed RSPOP algorithm in the POPFNN-CRI(S) architecture on the modeling of traffic flow data where a precise mathematical model cannot be obtained. The raw traffic flow data for the experiments is obtained from Tan (1997). The raw data were collected at site 29 located at exit 15 along the eastbound Pan Island Expressway (PIE) in Singapore using loop detectors embedded beneath the road surface. The inductive loop detectors were preinstalled by the Land Transport Authority of Singapore (LTA) in 1996 along major roads to facilitate traffic flow data collection. Figure 7 shows a picture of the location where the traffic flow data were collected. There are five lanes at the site: two exit lanes (lanes 4–5) and three straight lanes (lanes 1–3) for the main traffic. In the experiment, only the traffic flow data for the three straight lanes (lanes 1–3) are used. The traffic flow data set consists of four attributes: time and traffic density of lanes 1–3. The POPFNN-CRI(S) architecture is used to model the traffic flow trend, and the trained neuro-fuzzy system is used to predict traffic density for time t + τ where τ = 5, 15, 30, 45, 60 minutes. Figure 8 shows the traffic flow density data for the three straight lanes spanning a period of six days: September 5–10, 1996. The data set is divided into three cross-validation groups of training and test sets, exactly as in Tung and Quek (2004), on the traffic flow prediction case study using generic self-organizing fuzzy neural network (GenSoFNN) (Tung & Quek, 2002b). These three data set groups are named CV1, CV2, and CV3 and are extracted from the traffic flow data set as shown in Figure 8. The recall of POPFNN-CRI(S) using MLVQ to generate gaussian membership functions and RSPOP to identify fuzzy rules (abbreviated as RSPOPCRI) with CV1, CV2, CV3 and the prediction results of CV2 and CV3, CV1 and CV3, CV1 and CV2 for τ = 5 minutes are presented in Figure 9. The square of the Pearson product-moment correlation value (R2 ) and the MSE are then computed and averaged to form the prediction results of lane 1 traffic flow density for τ = 5 minutes. A total of 15 prediction results from RSPOP-CRI are subsequently consolidated in Figure 10 in prediction of lanes 1 to 3 for τ = 5, 15, 30, 45, 60 minutes. Figure 10 also presents the consolidated traffic flow prediction results of GenSoFNN (from Tung & Quek, 2004), POPFNN-CRI(S) with the same gaussian membership functions gen-

4 5 10

25 35 50

192 1,920 3,000,000

46 485 1,082,866

0.270 5.630 × 105 76.221

MSE of Test Data

RSPOP-CRI

22 24 50

188 202 184

20 14 43

17 14 29

Number of Rules Train Time Number of Rules Number of Rules Identified (ms) After Attributes After Rule Reduction Reduction

0.383 2.124 × 105 24.859

MSE of Test Data

RSPOP-CRI POP-CRI GenSOFNN-CRI(S) EFuNN DENFIS

14.4 40.0 50.0 234.5 9.7

0.146 0.173 0.164 0.186 0.153

Neuro-fuzzy Systems Number of Rules MSE

Table 11: Benchmark on the Average Number of Rules and Average MSE of Using RSPOP-CRI Against POP-CRI and Other Neuro-Fuzzy Systems on Traffic Flow Prediction.

1 2 3

Number of Number of Number of Rules Train Time Example Attributes Training Tuples Identified (ms)

POP-CRI

Table 10: Benchmark of POPFNN-CRI(S) Using POP Against RSPOP on Nakanishi Data Sets.

RSPOP 233

234

K. Ang and C. Quek

Figure 7: Location of site 29 along the Pan Island Expressway in Singapore, where traffic flow data are collected.

erated but using POP to identify the fuzzy rules (abbreviated as POP-CRI), evolving fuzzy neural networks (EFuNN; Kasabov, 2001), and the dynamic evolving neural-fuzzy inference system (DENFIS; Kasabov & Song, 2002). Table 11 shows a benchmark on the average number of rules and the average MSE of RSPOP-CRI compared to POP-CRI and other neuro-fuzzy systems. Figure 10 clearly shows a superior prediction accuracy of RSPOP-CRI in terms of both R2 and MSE compared against other neuro-fuzzy systems. Table 11 shows that on average, RSPOP identifies significantly fewer rules with increased accuracy compared to POP on the same POPFNN-CRI(S) architecture. Table 11 also shows that RSPOP-CRI uses an average of 14.4 fuzzy rules, fewer than other neuro-fuzzy systems except DENFIS, which uses a lower average of 9.7 fuzzy rules. However, DENFIS is based on the TSK model (Sugeno & Kang, 1988; Takagi & Sugeno, 1985) used for precise fuzzy modeling, which generally yields decreased interpretability but increased accuracy (Casillas et al., 2003). In contrast, POPFNN-CRI(S) is based on the Mamdani model (Mamdani & Assilian, 1975) used for linguistic fuzzy modeling, which is focused on interpretability. The objective of comparing RSPOP-CRI with DENFIS is to obtain a quality check on the prediction accuracy of RSPOP-CRI since interpretability is not directly comparable between these two models. The experimental result shows that the proposed RSPOP

RSPOP

235

Lane 1

Lane 2

Lane 3

6

Normalized Density

5

CV2

CV1 4

5/9 Thur

6/9 Fri

7/9 Sat

CV3

8/9 Sun

9/9 Mon

10/9 Tue

3 2 1 0

0.68

0.18

0.70

0.20

0.70

0.18

0.68

0.18

0.68

0.18

Normalized Time

Figure 8: Traffic flow density of three straight lanes along PIE at site 29.

algorithm is able to identify effective fuzzy rules, which improves the accuracy of POPFNN-CRI(S) used for linguistic fuzzy modeling to the extent that it is comparable to DENFIS used for precise fuzzy modeling. Therefore, the experimental results once again show that significant improvement in both interpretability and accuracy is obtained for POPFNN-CRI(S) using the proposed RSPOP algorithm to identify fuzzy rules. 6 Conclusion A novel rough set pseudo outer-product (RSPOP) algorithm is proposed in this article for the identification of fuzzy rules in pseudo outer-productbased fuzzy neural networks (POPFNN) (Ang et al., 2003; Zhou & Quek, 1996; Quek & Zhou, 1999). The proposed RSPOP algorithm addressed the problem of exponential increase in the number of fuzzy rules identified as well as increased computational complexity faced by the pseudo outerproduct (POP) fuzzy rule identification algorithm (Quek & Zhou, 2001) with high-dimensional data. Instead of pruning learned fuzzy rules through the use of certain heuristic thresholds in existing rule identification algorithms, RSPOP integrates the sound concept of knowledge reduction from rough set theory with the POP algorithm. RSPOP not only performed feature selection through the reduction of redundant attributes but also extends the

236

K. Ang and C. Quek

3

Lane 1 τ =5 Recall for CV1

2 1.5 1 0.5 0

3

100

200 300 400 Time Instance

Lane 1 τ =5 Recall for CV2

Normalized Density

Normalized Density

1

1

200

400 600 Time Instance

800

Lane 1 τ =5 Prediction of CV1 & CV3 Actual Predicted

2.5

0.5

2 1.5 1 0.5

100

200 300 400 Time Instance

0

500

Lane 1 τ =5 Recall for CV3

3 Actual Recall

2 1.5 1 0.5

200

400 600 Time Instance

800

Lane 1 τ =5 Prediction of CV1 & CV2 Actual Predicted

2.5 Normalized Density

2.5 Normalized Density

1.5

3 Actual Recall

1.5

0

2

0

500

2

3

Actual Predicted

0.5

2.5

0

Lane 1 τ =5 Prediction of CV2 & CV3

2.5 Normalized Density

2.5 Normalized Density

3 Actual Recall

2 1.5 1 0.5

100

200 300 400 Time Instance

500

0

200

400 600 Time Instance

800

Figure 9: Recall and prediction of lane 1 traffic density of RSPOP-CRI for τ = 5.

reduction to the fuzzy rules without redundant attributes. Because there are many possible reducts for a given rule set, an objective measure is developed based on the POPFNN architecture to identify the reducts that improve rather than deteriorate the inferred consequence after attribute and rule reduction. Using the rough set approach and the objective measure, the proposed RSPOP algorithm is able to identify a significantly reduced number of fuzzy rules compared to the POP algorithm, which improves the interpretability of POPFNN. However, as neuro-fuzzy systems generally

RSPOP

237 Lane 1 Prediction

0.9

Average R 2

0.86

0.2 Average MSE

0.88

Lane 1 Error

0.25

RSPOP-CRI POP-CRI GENSOFNN EFuNN DENFIS

0.84 0.82 0.8

0.15 RSPOP-CRI POP-CRI GenSOFNN EuFNN DENFIS

0.1

0.78 10

20 30 40 Time Interval τ

0.05 0

60

10

Lane 2 Prediction

0.86

0.82

20 30 40 Time Interval τ

0.78 0.76

0.15

RSPOP-CRI POP-CRI GenSOFNN EuFNN DENFIS

0.1

0.74 10

20 30 40 Time Interval τ

10

20 30 40 Time Interval τ

Average MSE

0.2 0.15 RSPOP-CRI POP-CRI GenSOFNN EuFNN DENFIS

0.1

20 30 40 Time Interval τ

50

60

0.25

0.8

10

50

Lane 3 Error

0.3

0.85

0.75 0

0.05 0

60

RSPOP-CRI POP-CRI GENSOFNN EFuNN DENFIS

0.9 Average R 2

50

Lane 3 Prediction

0.95

60

0.2

0.8

0.72 0

50

Lane 2 Error RSPOP-CRI POP-CRI GenSOFNN EFuNN DENFIS

0.84 Average R 2

50

Average MSE

0.76 0

60

0.05 0

10

20 30 40 Time Interval τ

50

60

Figure 10: Consolidated experimental results of RSPOP-CRI on traffic flow prediction compared to POP-CRI and other neuro-fuzzy systems.

suffer from an interpretability versus accuracy dilemma, experiments are performed to evaluate both the interpretability and accuracy of using the proposed RSPOP algorithm. Two sets of experiments were performed on the pseudo outer-productbased fuzzy neural networks using the compositional rule of inference and singleton fuzzifier (POPFNN-CRI(S)) architecture using a modified variant of learning vector quantization (MLVQ) algorithm to generate gaussian

238

K. Ang and C. Quek

membership functions and the proposed RSPOP algorithm to identify fuzzy rules. In the first set of experiments, the fuzzy rules identified using RSPOP are studied on the published data sets in the work of Nakanishi et al. (1993). Three experiments were performed using group A and B for training and group C for testing from each data set, similar to the experiments conducted in Nakanishi et al. (1993). The fuzzy rules identified after each step of using RSPOP are clearly tabulated for each experiment. A semantic interpretation of the gaussian membership functions and the fuzzy rules identified are provided using the first experimental results. Close examination of the reduced fuzzy rules obtained showed that some of the attributes selected after RSPOP attribute reduction are similar to the attributes selected in Nakanishi et al. (1993). The reduced fuzzy rules showed that the RSPOP rule reduction process is capable of partial attribute selection, whereas the variable selection process in Nakanishi et al. (1993) is capable only of crisp attribute selection. In addition, the RSPOP rule reduction process operates mainly on the identified fuzzy rules, not on the raw training data. This gives RSPOP a stronger advantage on ultra-large data sets since the identified fuzzy sets used by the rules present a horizontally reduced input (Lin & Cercone, 1997). RSPOP subsequently complements this horizontal reduction with vertical reduction using attribute reduction and rule reduction. Most significant, experimental results showed that the number of fuzzy rules identified using the proposed RSPOP algorithm are bounded from above by the number of tuples of the training data. Thus, the number of rules identified using RSPOP are significantly reduced compared to POP. Furthermore, computational complexity is extensively reduced and accuracy in terms of MSE improved as the dimensionality of the data set increases. The modeling accuracy of POPFNN-CRI(S) using the proposed RSPOP algorithm is also compared to three other reasoning methods: the Mamdani’s version of Zadeh’s CRI, Turksen’s version of interval-valued CRI, and Sugeno’s version of position and gradient type of reasoning in Nakanishi et al. (1993). In addition, it is compared against other neuro-fuzzy systems: adaptive-network-based fuzzy inference systems (ANFIS; Jang, 1993), evolving fuzzy neural networks (EFuNN; Kasabov, 2001), and dynamic evolving neural-fuzzy inference system (DENFIS; Kasabov & Song, 2002). ANFIS and DENFIS yielded significantly good accuracy on all three data sets. As these two neuro-fuzzy systems are based on the TSK model (Sugeno & Kang, 1988; Takagi & Sugeno, 1985) used for precise fuzzy modeling, these models generally yield decreased interpretability but increased accuracy (Casillas et al., 2003). In contrast, POPFNN-CRI(S) is based on the Mamdani model (Mamdani & Assilian, 1975) used for linguistic fuzzy modeling, which is focused on interpretability. The objective of the comparison with the recent TSK-based models is to obtain a quality check on the accuracy performance of POPFNN-CRI(S) using RSPOP since interpretability is not directly comparable between these two models. The consolidated experimental results showed that with an increase in the dimensionality of the data, the pro-

RSPOP

239

posed RSPOP algorithm identifies a significantly reduced number of fuzzy rules, which improves interpretability and yields superior accuracy in the POPFNN-CRI(S) architecture compared to other neuro-fuzzy systems. The second set of experiments were performed to evaluate the effectiveness of fuzzy rules identified in the POPFNN-CRI(S) architecture using RSPOP on a real-world application involving traffic flow prediction. Experimental results are compared to POPFNN-CRI(S) using the same gaussian membership functions generated but fuzzy rules identified using POP, generic self-organizing fuzzy neural network (GenSoFNN) using BackPOLE for supervised learning, evolving fuzzy neural networks (EFuNN) (Kasabov, 2001), and dynamic evolving neural-fuzzy inference system (DENFIS) (Kasabov & Song, 2002). Comparing the experiment results of POPFNNCRI(S) using RSPOP against POPFNN-CRI(S) using POP, the attribute and rule reduction of the former not only reduced the number of fuzzy rules identified but also improved prediction accuracy. Comparing the results of the POPFNN-CRI(S) using RSPOP against GenSoFNN, the former again yielded better prediction accuracy with fewer rules. This shows that the fuzzy rules identified by the proposed RSPOP algorithm are effective and require no additional supervised tuning. Experimental results also showed that the POPFNN-CRI(S) using RSPOP yielded better prediction accuracy compared to EFuNN and DENFIS. However, the former used an average of 14.4 fuzzy rules, whereas DENFIS used only an average of 9.7 fuzzy rules. Although interpretability in terms of number of fuzzy rules is not directly comparable between the neuro-fuzzy systems, experimental results showed that the proposed RSPOP algorithm is able to identify effective fuzzy rules, which improves the accuracy of POPFNN-CRI(S) used for linguistic fuzzy modeling to the extent that it is comparable to DENFIS used for precise fuzzy modeling. Therefore, the experimental results once again show that significant improvement in both interpretability and accuracy is obtained for POPFNN-CRI(S) using the proposed RSPOP algorithm to identify fuzzy rules. The potential of the proposed RSPOP algorithm discussed in this article is exciting because this algorithm extends the application of POPFNN for ultra-large and redundant data without an exponential increase in rules and computational complexity. The proposed RSPOP algorithm not only improves the interpretability of POPFNN by identifying significantly fewer rules, it also improves the accuracy of POPFNN. The strong attribute and rule-reduction properties of RSPOP also rid the user of POPFNN from the tedious process of identifying the influential input attributes. This research is in line with the research direction of Centre for Computational Intelligence (formerly the Intelligent Systems Laboratory in Nanyang Technological University Singapore). The Centre for Computational Intelligence undertakes active research in intelligent neuro-fuzzy systems for the modeling of complex, nonlinear, and dynamic problem domains. Examples of neural and neuro-fuzzy systems developed are modified cerebellar artic-

240

K. Ang and C. Quek

ulation controller (MCMAC; Ang & Quek, 2000), generic self-organizing fuzzy neural network (GenSoFNN; Tung & Quek, 2002b), and pseudo outerproduct-based fuzzy neural network (POPFNN; Zhou & Quek, 1996). These have been applied to novel and interesting applications such as automated driving (Pasquier, Quek, & Toh, 2001), signature forgery detection (Quek & Zhou, 2002), gear control for automotive continuous variable transmission (Ang, Quek, & Wahab, 2002), and fingerprint verification (Quek, Tan, & Sagar, 2001). References Ang, K. K., & Quek, C. (2000). Improved MCMAC with momentum, neighborhood, and averaged. IEEE Transactions on Systems, Man and Cybernetics, Part B, 30(3), 491–500. Ang, K. K., Quek, C., & Pasquier, M. (2003). POPFNN-CRI(S): Pseudo outer product–based fuzzy neural network using the compositional rule of inference and singleton fuzzifier. IEEE Transactions on Systems, Man and Cybernetics, Part B, 33(6), 838–849. Ang, K. K., Quek, C., & Wahab, A. (2002). MCMAC-CVT: A novel on-line associative memory based CVT transmission control system. Neural Networks, 15(2), 219–236. Buckley, J. J., & Hayashi, Y. (1994). Fuzzy neural networks: A survey. Fuzzy Sets and Systems, 66(1), 1–13. Buckley, J. J., & Hayashi, Y. (1995). Neural nets for fuzzy systems. Fuzzy Sets and Systems, 71(3), 265–276. Casillas, J., Cordon, ´ O., Herrera, F., & Magdalena, L. (2003). Interpretability issues in fuzzy modeling. Berlin: Springer-Verlag. Castro, J. L. (1995). Fuzzy logic controllers are universal approximators. IEEE Transactions on Systems, Man and Cybernetics, 25(4), 629–635. Chakraborty, D., & Pal, N. R. (2004). A neuro-fuzzy scheme for simultaneous feature selection and fuzzy rule-based classification. IEEE Transactions on Neural Networks, 15(1), 110–123. Chiu, S. L. (1994). Fuzzy model identification based on cluster estimation. Journal of Intelligent and Fuzzy Systems, 2(3), 267–278. Curry, B. (2003). Rough sets: Current and future developments. Expert Systems, 20(5), 247–250. Guillaume, S. (2001). Designing fuzzy inference systems from data: An interpretability–oriented review. IEEE Transactions on Fuzzy Systems, 9(3), 426–443. Han, J., & Kamber, M. (2001). Data mining: Concepts and techniques. San Francisco: Morgan Kaufmann. Hayashi, I., Nomura, H., Yamasaki, H., & Wakami, N. (1992). Construction of fuzzy inference rules by NDF and NDFL. International Journal of Approximate Reasoning, 6(2), 241–266. Hebb, D. O. (1949). The organization of behavior: A neuropsychological theory. New York: Wiley.

RSPOP

241

Ishibuchi, H., Tanaka, H., & Okada, H. (1994). Interpolation of fuzzy if-then rules by neural networks. International Journal of Approximate Reasoning, 10(1), 3– 27. Jang, J.-S. R. (1993). ANFIS: Adaptive-network-based fuzzy inference system. IEEE Transactions on Systems, Man and Cybernetics, 23(3), 665–685. Jin, Y. (2000). Fuzzy modeling of high-dimensional systems: Complexity reduction. IEEE Transactions on Fuzzy Systems, 8(2), 212–221. Johansen, T. A., & Babuˇska, R. (2003). Multiobjective identification of TakagiSugeno fuzzy models. IEEE Transactions on Fuzzy Systems, 11(6), 847–860. Kasabov, N. (2001). Evolving fuzzy neural networks for supervised/unsupervised online. IEEE Transactions on Systems, Man and Cybernetics, Part B, 31(6), 902–918. Kasabov, N. K., & Song, Q. (2002). DENFIS: Dynamic evolving neural-fuzzy inference system and its application for time-series prediction. IEEE Transactions on Fuzzy Systems, 10(2), 144–154. Kaynak, O., Jezernik, K., & Szeghegyi, A. (2002). Complexity reduction of rule based models: A survey. In Fuzzy Systems, 2002. FUZZ-IEEE’02. Proceedings of the 2002 IEEE International Conference (Vol. 2, pp. 1216–1221). Piscataway, NJ: IEEE. Kohonen, T. (1989). Self-organization and associative memory (3rd ed.). Berlin: Springer-Verlag. Lin, C.-T., & Lee, C. S. G. (1991). Neural-network-based fuzzy logic control and decision system. IEEE Transactions on Computers, 40(12), 1320–1336. Lin, C.-T., & Lee, C. S. G. (1996). Neural fuzzy systems: A neuro-fuzzy synergism to intelligent systems. Upper Saddle River, NJ: Prentice Hall. Lin, T. Y., & Cercone, N. (1997). Rough sets and data mining: Analysis of imprecise data. Boston: Kluwer. Lozowski, A., Cholewo, T. J., & Zurada, J. M. (1996). Crisp rule extraction from perceptron network classifiers. In Proceedings of International Conference on Neural Networks, vol. Plenary, Panel and Special Sessions (pp. 94–99). Washington, D.C. Mamdani, E. H., & Assilian, S. (1975). An experiment in linguistic synthesis with a fuzzy logic controller. International Journal of Man-Machine Studies, 7(1), 1– 13. Mitra, S., & Hayashi, Y. (2000). Neuro-fuzzy rule generation: Survey in soft computing framework. IEEE Transactions on Neural Networks, 11(3), 748–768. Moser, B. (1999). Sugeno controllers with a bounded number of rules are nowhere dense. Fuzzy Sets and Systems, 104(2), 269–277. Nakanishi, H., Turksen, I. B., & Sugeno, M. (1993). A review and comparison of six reasoning methods. Fuzzy Sets and Systems, 57(3), 257–294. Nauck, D. D. (2003). Measuring interpretability in rule-based classification systems. In Fuzzy Systems, 2003. FUZZ ’03. 12th IEEE International Conference (Vol. 1, pp. 196–201). Piscataway, NJ: IEEE. Nauck, D., Klawonn, F., & Kruse, R. (1997). Foundations of neuro-fuzzy systems. New York: Wiley. Pal, N. R., & Pal, T. (1999). On rule pruning using fuzzy neural networks. Fuzzy Sets and Systems, 106(3), 335–347.

242

K. Ang and C. Quek

Pasquier, M., Quek, C., & Toh, M. (2001). Fuzzylot: A novel self-organising fuzzy–neural rule–based pilot system for automated vehicles. Neural Networks, 14(8), 1099–1112. Pawlak, Z. (1991). Rough sets: Theoretical aspects of reasoning about data. Boston: Kluwer. Quek, C., Tan, K. B., & Sagar, V. K. (2001). Pseudo-outer product based fuzzy neural network fingerprint verification system. Neural Networks, 14(3), 305– 323. Quek, C., & Zhou, R. W. (1999). POPFNN-AAR(S): A pseudo outer-product based fuzzy neural network. IEEE Transactions on Systems, Man and Cybernetics, Part B, 29(6), 859–870. Quek, C., & Zhou, R. W. (2001). The POP learning algorithms: Reducing work in identifying fuzzy rules. Neural Networks, 14(10), 1431–1445. Quek, C., & Zhou, R. W. (2002). Antiforgery: A novel pseudo-outer product based fuzzy neural network driven signature verification system. Pattern Recognition Letters, 23(14), 1795–1816. Shann, J. J., & Fu, H. C. (1995). A fuzzy neural network for rule acquiring on fuzzy control systems. Fuzzy Sets and Systems, 71(3), 345–357. Shen, Q., & Chouchoulas, A. (1999). Combining rough sets and data-driven fuzzy learning for generation of classification rules. Pattern Recognition, 32(12), 2073–2076. Shen, Q., & Chouchoulas, A. (2002). A rough-fuzzy approach for generating classification rules. Pattern Recognition, 35(11), 2425–2438. Sugeno, M., & Kang, G. T. (1988). Structure identification of fuzzy model. Fuzzy Sets and Systems, 28(1), 15–33. Sugeno, M., & Yasukawa, T. (1993). A fuzzy-logic-based approach to qualitative modeling. IEEE Transactions on Fuzzy Systems, 1(1), 7. Swiniarski, R. W., & Skowron, A. (2003). Rough set methods in feature selection and recognition. Pattern Recognition Letters, 24(6), 833–849. Takagi, T., & Sugeno, M. (1985). Fuzzy identification of systems and its applications to modeling and control. IEEE Transactions on Systems, Man and Cybernetics, 15(1), 116–132. Tan, G. K. (1997). Feasibility of predicting congestion states with neural network models. Unpublished final year project report, Nanyang Technological University, Singapore. Tikk, D., & Baranyi, P. (2003). Exact trade-off between approximation accuracy and interpretability: Solving the saturation problem for certain FRBSs. In J. Casillas, O. Cordon, ´ F. Herrera, & L. Magdalena (Eds.), Interpretability issues in fuzzy modeling (pp. 587–604). Berlin: Springer-Verlag. Tikk, D., Koczy, ´ L. T., & Gedeon, T. D. (2003). A survey on universal approximation and its limits in soft computing techniques. International Journal of Approximate Reasoning, 33(2), 185–202. Tung, W. L., & Quek, C. (2002a). DIC: A novel discrete incremental clustering technique for the derivation of fuzzy membership functions. In Proceedings of the 7th Pacific Rim International Conference on Artificial Intelligence (pp. 178– 187). Berlin: Springer-Verlag.

RSPOP

243

Tung, W. L., & Quek, C. (2002b). GenSoFNN: A generic self-organizing fuzzy neural network. IEEE Transactions on Neural Networks, 13(5), 1075–1086. Tung, W. L., & Quek, C. (2004). Supervised learning in neural fuzzy systems (Part 2)—BackPole: A back propagation algorithm based on objective learning errors. Manuscript submitted for publication. Yager, R. R. (1994). Modeling and formulating fuzzy knowledge bases using neural networks. Neural Networks, 7(8), 1273–1283. Yen, J., Wang, L., & Gillespie, C. W. (1998). Improving the interpretability of TSK fuzzy models by combining global learning and local learning. IEEE Transactions on Fuzzy Systems, 6(4), 530–537. Ying, H., Ding, Y., Li, S., & Shao, S. (1999). Comparison of necessary conditions for typical Takagi-Sugeno and Mamdani fuzzy systems as universal approximators. IEEE Transactions on Systems, Man and Cybernetics, Part A, 29(5), 508–514. Zadeh, L. A. (1975a). The concept of a linguistic variable and its application to approximate reasoning—I. Information Sciences, 8(3), 199–249. Zadeh, L. A. (1975b). The concept of a linguistic variable and its application to approximate reasoning—II. Information Sciences, 8(4), 301–357. Zadeh, L. A. (1975c). The concept of a linguistic variable and its application to approximate reasoning—III. Information Sciences, 9(1), 43–80. Zadeh, L. A. (1994). Fuzzy logic, neural networks, and soft computing. Communications of the ACM, 37(3). Zhou, R. W., & Quek, C. (1996). POPFNN: A pseudo outer-product based fuzzy neural network. Neural Networks, 9(9), 1569–1581. Received September 4, 2003; accepted June 2, 2004.

REVIEW

Communicated by Satinder Singh

Temporal Sequence Learning, Prediction, and Control: A Review of Different Models and Their Relation to Biological Mechanisms Florentin Worg ¨ otter ¨ [email protected]

Bernd Porr [email protected] Department of Psychology, University of Stirling, Stirling FK9 4LA, Scotland

In this review, we compare methods for temporal sequence learning (TSL) across the disciplines machine-control, classical conditioning, neuronal models for TSL as well as spike-timing-dependent plasticity (STDP). This review introduces the most influential models and focuses on two questions: To what degree are reward-based (e.g., TD learning) and correlationbased (Hebbian) learning related? and How do the different models correspond to possibly underlying biological mechanisms of synaptic plasticity? We first compare the different models in an open-loop condition, where behavioral feedback does not alter the learning. Here we observe that reward-based and correlation-based learning are indeed very similar. Machine control is then used to introduce the problem of closed-loop control (e.g., actor-critic architectures). Here the problem of evaluative (rewards) versus nonevaluative (correlations) feedback from the environment will be discussed, showing that both learning approaches are fundamentally different in the closed-loop condition. In trying to answer the second question, we compare neuronal versions of the different learning architectures to the anatomy of the involved brain structures (basal-ganglia, thalamus, and cortex) and the molecular biophysics of glutamatergic and dopaminergic synapses. Finally, we discuss the different algorithms used to model STDP and compare them to reward-based learning rules. Certain similarities are found in spite of the strongly different timescales. Here we focus on the biophysics of the different calciumrelease mechanisms known to be involved in STDP. 1 Introduction Flexible reaction in response to events requires foresight. This holds for humans and animals but also for robots or other agents that interact with their environment. Thus, predicting the future has been a central objective not only for soothsayers in ancient times and their more modern equivalents— the stock market analysts—but for every agent that needs to survive in a Neural Computation 17, 245–319 (2005)

c 2005 Massachusetts Institute of Technology


246

F. Worg ¨ otter ¨ and B. Porr

sometimes hostile environment. Accurate, and thus useful, predictions can be made only if they are based on prior knowledge, which must be obtained by analyzing the relations between past events. Many events encountered by an agent are linked in a temporally causal way through the laws of physics and the causal structure of the world. Examples from the living and the technical world, some more sensor related and others more related to motor actions, make this clear: 1. Smell precedes taste when foraging, and the sound of moving prey may precede its smell. 2. The sequence “warm,” “hot,” “hotter” should be followed by “very hot” or by “pain.” 3. Leaning sideways will lead to a reflex-like antagonistic motor action in order to prevent falling. 4. The motor action of feeding will lead to a taste sensation. 5. The bell precedes the food in Pavlov’s classical conditioning experiments. 6. The position of a robot arm is preceded by the force patterns that converge onto it. In these examples we have deliberately mixed sensor-sensor events (e.g., items 1, 2) with motor-motor (item 3) and sensor-motor events (item 4). We will see that theoretical treatment needs to take care of these distinctions. At the same time, we have also introduced unimodal (temperature over time in item 2) and multimodal events (all others), which will also require different treatment. The list of such causally related events or actions, which will occur during the lifetime of an agent, is endless. All of these events have in common that they are temporally (auto- or cross-) correlated with each other such that they form a temporal sequence. Agents that are able to act in response to earlier events in such a sequence chain have, without doubt, an evolutionary advantage. As a consequence, learning to correctly interpret temporal sequences and deduce appropriate actions is a major incentive for any agent fostering its survival. Actually, the problem with which we are faced is twofold: learn to predict an event, and learn to perform an anticipatory action. This has been called the prediction- and the control-problem (Sutton, 1999). 1.1 Structure of This Review. The goal of this review is threefold. (1) We use machine control to introduce the basic reinforcement learning terminology. (2) We then compare the different neuronal models for reward-based TD learning, which are used in so-called actor-critic architectures of animal control, with each other and with the anatomy and physiology of the basal gan-

Temporal Sequence, Prediction, and Control

247

Figure 1: Cross links between the different fields (see the text). Small equals signs denote that these algorithms produce identical results after convergence.

glia. Here will strongly focus on the biophysics of dopaminergic synapses and try to provide a summary of the most essential synaptic mechanisms that have been identified so far. (3) Finally, we compare reward-based learning mechanisms (e.g., TD learning) with correlation-based Hebbian learning, trying to relate them to the basic mechanisms of long-term potentiation (LTP) and long-term depression (LTD) as found in spike-timing-dependent plasticity (STDP). Figure 1 shows the organization of this review. This figure depicts the most important links between the different subfields and algorithms; the most influential ones are shown in red. The article proceeds from left (machine learning) to right (synaptic plasticity). The flow in the sections (direction of the arrows) is always from top to bottom, except for all “green” components, which are concerned with biophysical aspects. Here we proceed upward. For specific reviews on the three different fields we refer readers to Reinforcement learning (Kaelbling, Littman, & Moore, 1996; Sutton & Barto, 1998), animal conditioning (Sutton & Barto, 1990; Balkenius & Moren, 1998), the dopamine reward system of the brain (Schultz, 1998, 2002; Schultz & Dickinson, 2000), and data and models of synaptic plasticity (Martinez & Derrick, 1996; Malenka & Nicoll, 1999; Bennett, 2000; Bi & Poo, 2001; van Hemmen, 2001; Bi, 2002).

248

F. Worg ¨ otter ¨ and B. Porr

2 The Basic Concepts of Reinforcement Learning It is fair to say that currently, almost all methods for predictive control in robots rely on reinforcement learning (RL), and strong indications exist in the literature that this type of learning plays a central role in animals too. A major thread that is followed throughout this article concerns the distinction between reward-based versus correlation-based learning methods, or between evaluative versus nonevaluative feedback (see Figures 7 and 9). RL, at least in its original formulation, is strictly reward based. Hence, an early conclusion would be that animals rely heavily on reward-based learning. Synaptic plasticity, on the other hand, is correlation based (Hebbian). How can this apparent conflict be resolved? How can reward-based mechanisms be reformulated such that they can be captured by correlation-based learning rules? And what are the problems when trying to do this? These are the questions we address in this review. The central message of this article is that reward-based and correlationbased methods are very similar, if not identical, in the open-loop condition, where the actions of the learner will not influence the learning, while they are clearly different in the closed-loop condition, hence during the normally existing behavioral feedback. The algorithms for reinforcement learning are treated to a great extent in the literature (Sutton & Barto, 1998) and shall not be described here, but we must describe the basic assumptions of RL to be able to compare it to animal learning. To this end we will only discuss systems with a finite number of discrete states, commonly known as finite Markov decision problems (MDP).1 We assume that an RL agent is able to visit these states and that these states will convey information about the decision problem. RL further assumes that in visiting a state, a numerical reward will be collected, where negative numbers may represent punishments. Each state has a changeable value attached to it. From every state, there are subsequent states that can be reached by means of actions. The value of a given state is basically defined by the averaged future reward, which can be accumulated by starting actions from this particular state. Here, we could look at all or just at some of the follow-up actions that are possible in starting from the given state. The different algorithms for RL take different approaches

1 Thus, RL also assumes that such systems “follow the Markov property.” Essentially this means that it is unimportant along which path a certain state has been reached. Once there, the state itself contains all relevant information for future calculations. Many times the Markow property cannot be guaranteed in real-world decision problems, which poses a practical problem when wanting to employ RL methods. Also we note that conventional RL needs to be augmented by additional mechanisms if one wants to employ it to more complex (e.g., time and space continuous) systems. These aspects shall not be discussed here, but see Reynolds (2002), and Santos and Touzet (1999a, 1999b).

Temporal Sequence, Prediction, and Control

249

toward reward averaging, which shall not be discussed here. Actions will follow a policy, which can also change. The goal of RL is to maximize the expected cumulative reward (the “return”) by subsequently visiting a subset of states in the MDP. This, for example, can be done by assuming a given (unchangable) policy. Often, however, a subgoal of RL is to try to achieve this in an optimal way by finding the best action policy to travel through state-space. In this context, RL methods are mainly employed to address two related problems: the prediction and the control problem. 1. Prediction only. RL is used to learn the value function for the policy followed. At the end of learning, this value function describes for every visited state how much future reward we can expect when performing actions starting at this state. 2. Control. By means of RL, we wish to find that particular set of policies that maximizes the reward when traveling through state-space. This way, we have at the end obtained an optimal policy that allows for action planning and optimal control. Several algorithms have been designed to calculate or approximate value functions or to find optimal action policies, most notably, dynamic programming (Bellman, 1957), Monte Carlo prediction and control, TD learning (Sutton, 1988), SARSA (Rummery, 1995; Sutton, 1996), and Q learning (Watkins, 1989; Watkins & Dayan, 1992). They are reviewed in Sutton and Barto (1998). Most of the above-named algorithms for RL rely on the method of temporal differences (TD methods). The formalism of this method is described in the appendix and its neuronal version later in the main text. Using the TD method, the rewards, which determine learning, enter the formalism in an additive way (see equation A.3). Thus, TD methods are in the first instance strictly noncorrelative as opposed to Hebb rules, where a multiplicative correlation of pre- and postsynaptic activity drives the learning. Thus, at first it seems hard to introduce a correlative relation in RL to make it compatible with synaptic plasticity, hence with Hebbian learning. Two observations can be made to mitigate this situation. First, we note that the backward TD(λ) method (see the appendix) contains so-called eligibility traces x, which enter the algorithm in a multiplicative way (see equation A.11) and can thus be used to define a correlation-based process. The concept of such traces shall be explained in a more neuronal context in greater detail in section 3. Here it suffices to say that this way, TD learning becomes formally related to Hebbian learning. There is, however, a more interesting, but also more problematic, way to introduce a correlation-based process that is relevant for biological or biomimentic agents: one could try to define the rewards (or punishments) by means of signals derived from sensor inputs. Naively, pain is a punishment, while pleasure is a reward. Thus, rewards can be correlated to sensor events.

250

F. Worg ¨ otter ¨ and B. Porr

This seems to be a simple and obvious way to introduce a correlation-based process mainly used in so-called actor-critic models of machine or animal control, which are introduced later (in section 4.1). Actor-critic architectures are closed-loop control structures, where the actions of the agent (the animal) will influence its own (sensor) inputs. Thus, before we can discuss those, we must first describe the different algorithms in their open-loop versions. 3 Neuronal Architectures for Prediction and Control—Open Loop Condition The goal of the next sections is to transfer these concepts of RL, for example, TD learning, to neural networks related to different brain structures. At the same time, we will discuss other algorithms that are less directly related to the RL formalism. 3.1 Relating RL to Classical Conditioning. Classical conditioning represents in its simplest form a learning paradigm where the pairing of two subsequent stimuli is learned such that the presentation of the first stimulus is taken as a predictor of the second one. In the descriptions above, we have discussed the prediction problem but have treated only the (action-driven) transition between states. Only a vague indication has been given so far about how to use temporally correlated signals for learning. Thus, the goal of this section is to show how to augment these aspects such that we can use correlations for learning. This way, we will see that reinforcement learning and (differential) Hebbian learning are related to each other. 3.1.1 Early Neural Differential Hebbian Architectures. In the traditional example of Pavlov’s dog (Pavlov, 1927), we have the unconditioned stimulus (US), food, which is preceded by the conditioned stimulus (CS), bell. The CS predicts the US, and after learning, the dog starts salivating in response to the CS. This represents an open-loop paradigm: the action of salivating will not influence the presentation of the stimuli. This is different from instrumental conditioning, which represents a closed-loop paradigm discussed later in the context of actor-critic architectures. A model of classical conditioning should produce an output that, before learning, responds to the US only; after learning, the output should occur earlier in response to the CS. Furthermore, we note that the US and CS are temporally correlated. To capture this, we need a correlative property in reinforcement learning that in its machine-learning formalism (see the appendix) is not immediately visible, because machine learning relies entirely on the transition between states by means of actions, and correlations do not play any role.

Temporal Sequence, Prediction, and Control

X1

251

CS

x

e-trace

X0

Dw1 + w1 w0 = 1

v(t)

US (reward)

Figure 2: Simple correlative temporal learning mechanism applying an eligibility trace (e-trace) at input x1 (CS) to ensure signal overlap at the moment when x0 (US) occurs; ⊗ denotes the correlation.

This, however, can be achieved by using eligibility traces.2 To this end, we need to restructure the formalism in a neuronal way, where stimuli converge via synapses at a neuron. These architectures are in the context of classical conditioning generally called stimulus substitution architectures, because the first, earlier stimulus after learning substitutes in effect the second, later stimulus. The goal is to generate an output signal at this neuron that will, at the end of learning, directly respond to the CS. This way, the output can be used as a predictor signal for other processes, such as for control. This should be achieved by strengthening the synapse that transmits the CS during learning. Figure 2 shows such a structure at the beginning of learning. Note that this model level is still quite far removed from the biophysical models of synaptic plasticity, which we discuss later. The US converges with a strong, driving synapse ω0 at the neuron. To keep things consistent with the descriptions above, one could call the US “reward.” The CS precedes the US and converges with an initially weak synapse ω1 . However, the CS elicits a decaying trace (e-trace) at its own synapse. This, biophysically unspecified, trace is meant to capture the idea that the CS synapse remains eligible for modification for some time after the CS has ended. We can now correlate the output V, which at the beginning of learning mirrors the US, with this eligibility trace and use the result of this correlation, depicted by ω1 , to change the weight of the CS-synapse. This idea was first formalized by Sutton and

2 Historically eligibility traces were first developed in the context of classical conditioning models (Hull, 1939, 1943; Klopf, 1972, 1982) and only later were introduced in the context of machine learning (Sutton, 1988; Singh & Sutton, 1996). Our review emphasizes the structural and functional similarities of the different algorithms; therefore, we do not follow the historical route.

252

F. Worg ¨ otter ¨ and B. Porr

Barto (1981) in their classical modeling study:3 ω1 (t + 1) = ω1 (t) + γ [v(t) − v(t)]x(t),

(3.1)

where they introduced two eligibility traces, x(t) at the input and v(t) at the output, given by x(t + 1) = αx(t) + x(t)

(3.2)

v(t + 1) = βv(t) + (1 − β) v(t),

(3.3)

with control parameters α and β. Mainly they discuss the case of β = 0, where v(t) = v(t − 1), which turns their rule into ω1 (t + 1) = ω1 (t) + γ [v(t) − v(t − 1)]x(t).

(3.4)

Before learning, this neuron will respond only to the US, while after learning, it will respond to the CS as well. In a later review, Sutton and Barto (1990) discuss that their older model is faced with several problems. For example, for short or negative interstimulus intervals (ISIs) between CS and US, the model predicts strong inhibitory conditioning (see Figure 8 in Sutton & Barto, 1990). Some reports exist that show weak inhibitory conditioning, but mostly weak excitatory conditioning seems to be found in these cases as well (Prokasy, Hall, & Fawcett, 1962; Mackintosh, 1974, 1983; Gormezano, Kehoe, & Marshall, 1983).4 3.1.2 Isotropic Sequence Order Learning. Some of the problems of the Sutton and Barto (1981) model come from the fact that input lines are not treated identically, which leads to an asymmetrical behavior of this model. In a more recent approach, we have achieved this by employing a specific differential Hebbian learning rule onto all synaptic weights (Porr & Worg ¨ otter, ¨ 2002, 2003a, 2003b). The main distinguishing features of isotropic sequence order learning (ISO learning) are that (1) all input lines are treated equal, so there is no a priori built-in distinction between CS and US and all lines learn; (2) eligibility traces are created by bandpass filtering the inputs; (3) learning is purely correlation based and synapses can grow or shrink depending on 3 Here we briefly mention that we deliberately have not discussed some older work, like the Rescorla-Wagner rule (Rescorla & Wagner, 1972, Fig. 1) or the δ rule of Widrow and Hoff (1960), which is at the root node of all these algorithms. Indeed, analytical proofs exist that the results obtained with the δ rule are identical to those obtained with the RescorlaWagner rule (Sutton & Barto, 1981) and the TD(1) algorithm (Sutton, 1988), denoted by the small equals signs in Figure 1. 4 More problems are discussed in Sutton and Barto (1990), for example, with respect to the so-called delay-conditioning paradigm, which can be solved by some specific modifications of the Sutton and Barto (1981) model. We refer readers to Sutton and Barto (1990) for these specific issues.

Temporal Sequence, Prediction, and Control

253

Figure 3: Isotropic sequence order learning. (A) Structure of the algorithm for N + 1 inputs. For notations, see the text. A central property of ISO learning is that all weights can change. (B) Weight change curve calculated analytically for two inputs with identical resonator characteristics (h). The optimal temporal difference for learning is denoted as Topt . (C) Linear development of ω1 for two active inputs (x0 , x1 ). At time-step 40,000, input x0 is switched off, and as a consequence of the orthogonality property of ISO learning, ω1 stops to grow. (D) Development of 10 weights ω1i , i = 0, . . . , 9 in a robot experiment (Porr & Worg ¨ otter, ¨ 2003a). All weights are driven by input x1 but are connected to different resonators hi1 , which create a serial compound representation of x1 (see Figure 10A). The robot’s task was obstacle avoidance. At around t = 150 s, it has successfully mastered it, and the input x0 , which corresponds to a touch sensor, is not triggered again. As a consequence, we observe that the weights ω1i stop to change (compare to C).

the temporal sequence of their inputs; and (4) inputs can take any form of being analog or pulse coded. As a consequence, this approach is linear. Figure 3 A shows the structure of the algorithm The system consists of ¯ N + 1 linear filters h receiving inputs x and producing outputs x: x¯ i = xi ∗ hi ,

(3.5)

254

F. Worg ¨ otter ¨ and B. Porr

where the asterisk denotes a convolution. The transfer functions h shall be those of bandpass filters. They are specified by h(t) =

1 at e sin(bt), b

(3.6)

with: a := Re(p) = −π f/Q, b := Im(p) =




(2π f )2 − a2 .

(3.7)

f is the frequency of the oscillation and Q the damping characteristic. To get an idea what these filters do, we consider pulse inputs. In this case, the filtered signals will consist of damped oscillations (Grossberg & Schmajuk, 1989; Grossberg, 1995; Grossberg & Merrill, 1996), which span across some temporal interval until they fade. Thus, bandpass filtering essentially amounts to applying an eligibility trace to all inputs. The filtered signals connect with corresponding weights ω to one output unit v. The output v(t) is given as v(t) =

N 

ωi x¯ i .

(3.8)

i=0

Learning takes place according to a differential Hebbian learning rule, d ωi = µx¯ i v dt

µ 1,

(3.9)

where v is the temporal derivative of v. Note that µ is very small. (See Porr & Worg ¨ otter, ¨ 2003a, for a complete description of the ISO learning algorithm and its properties.) First, we note that the system is linear, and weight changes can be calculated analytically (see Figure 3B), as shown in Porr and Worg ¨ otter ¨ (2003a). If we consider just one input, then we find that it is after filtering orthogonal to the derivative of its output. Intuitively this can be understood when looking at two pulse inputs x0 , x1 . In this case, both trace signals x¯ 0 , x¯ 1 are damped sine waves, and the derivative of the output is thus a sum of damped cosine waves. Thus, if one input becomes zero, the derivative of the output will be orthogonal to the remaining input. Mathematically, it can be shown that this holds for more than two inputs too (Porr & Worg ¨ otter, ¨ 2003a). Thus, inputs will not influence their own synapses, and learning is strictly hetero-synaptic. As a consequence, a very nice feature emerges for pairs of synapses: weight change will stop as soon as one input becomes silent (see Figure 3C). This leads to an automatic self-stabilizing property for the network in control applications (Figure 3D; see also section 4.3.1).

Temporal Sequence, Prediction, and Control

255

3.1.3 The Basic Neural TD Formalism and Its Implementation. Most influential in the field of neuronal temporal sequence learning algorithms currently is the TD formalism. To define it in a neuronal way, we replace the “states” from traditional RL with “time-steps” and assume that rewards can be retrieved at each such time-step. Furthermore, we assume that distant rewards will count less than immediate rewards, introducing a discounting factor γ . Then we define the total reward (called the “return”) as the discounted sum of all expected future rewards (similar to the formalism in the appendix): R(t) = r(t + 1) + γ r(t + 2) + γ 2 r(t + 3) + . . . ,

(3.10)

and this can be rewritten as R(t) = r(t + 1) + γ R(t + 1).

(3.11)

Now we define a neuron v such that it will function as a prediction neuron. To this end, we assume that the output v is able to predict the reward. As a consequence, we would hope that R(t + 1) ≈ v(t + 1) and that R(t) ≈ v(t), replacing (compare equation A.5), v(t) ≈ r(t + 1) + γ v(t + 1).

(3.12)

Since this is only approximately true (until convergence), we can define an error (compare equation A.9) with δ(t) = r(t + 1) + γ v(t + 1) − v(t),

(3.13)

realizing that this error function contains a derivative-like term, v(t+1)−v(t), which is shifted one step into the future. This results from the fact that the update of the value of a state requires visiting at least the next following state. In an artificial neural network such as that depicted in Figure 4, this does not produce problems, but in a rigorous neuronal implementation, one should be careful to avoid this acausality, for example, applying the appropriate delays or a more realistic eligibility trace (see below). Now we can update the synaptic weights with ωi ← ωi + α δ(t) xi (t),

(3.14)

where xi (t) is the eligibility trace associated with stimulus xi . In an older review, Sutton and Barto (1990) discuss to what degree this specific TD model can explain the experimental observations in classical conditioning, and they show that it surpasses their older approach in many aspects. This shall not be discussed here, though, because we wish to concentrate next on the neuronal aspects of TD learning—its network implementations, its possible neuronal counterparts in the brain, and its synaptic biophysics.

256

F. Worg ¨ otter ¨ and B. Porr

A #1

Start: w0 = 0 w1 = 0

reward, US

Predictive Signals

Start: w0 = 1 w1 = 0

#2

X1 x

X0 x

v

~ v’

End: w0 = 1 w1 = 0

~ d=v’+ r reward

x

Xn X1

End: w0 = 1 w1 = 1

d x

#3

v

~ v’ (n-i)t

~ v’ v(t)

X0

~ d=v’+ r

B

C r d

x

w0 = 1

d

x

t t

~ v’

X1 X0 reward

r

v(t-t)

v(t)

X1

v(t)

X0 reward

w0 = 1

Figure 4: Basic neural implementation of the TD rule. (A) The first three learning steps (#1,#2,#3) performed with the circuit drawn in the shaded inset, which represents what probably is the most basic version of a neuronal TD implementation using a serial compound stimulus representation. The symbol v˜  represents a forward-shifted difference operation: v˜  (t) = v(t + 1) − v(t). Other symbols are explained in the text. (B,C) Same circuit but using x0 as reward signal. (C) The acausal forward shift of the difference operation is avoided by introducing unit delays τ into the learning circuits. This allows calculating the derivative by means of an inhibitory interneuron (black).

Temporal Sequence, Prediction, and Control

257

The circuit diagram in Figure 4 shows a simple implementation of the TD rule in a neuronal model. It is constructed to most closely relate to the basic neuronal TD formalism introduced above, keeping the number of necessary parameters minimal. The first neuronal models with an architecture similar to Figure 4 were devised by Montague, Dayan, Person, and Sejnowski (1995). The goal of our implementation is to arrive at an output v, which reacts after learning to the onset of the CS denoted as xn and maintains its activity until the reward terminates. Such an ideally rectangular signal is reminiscent of the response of so-called reward-expectation neurons, which shall be shown in section 3.3. To achieve this goal, we represent the CS internally by a chain of n + 1 delayed pulses xi , with a unit delay τ , which is equal to the pulse width. We construct this chain such that the signal x0 coincides with the reward. In some sense, the pulse chain xi represents a special (distributed) kind of eligibility trace; it is often called a serial compound representation (Sutton & Barto, 1990) first used in a neural network architecture by Montague, Dayan, & Sejnowski (1996). The pulse protocols 1, 2, and 3 in Figure 4 show the first three trials for this algorithm starting with all weights at zero. Thus, in the first trial (#1), the output v is zero, and a positive prediction error δ occurs together with the reward. The weight change is calculated using equation 3.14 with ωi = δ xi . Thus, in the first trial, 1, only the multiplication of x0 with δ will yield a result different from zero, and only this weight changes. In the next trial, v reacts together with x0 ; the derivative is shown below already time-shifted by one step forward as demanded above. As a consequence, δ also moves forward (Montague et al., 1996), and now the correlation between x1 and δ yields 1. This process continues in trial 3, where we show only a few traces, and so on until the last weight ωn grows. As a result, v becomes a rectangle pulse starting at xn and ending at the end of the reward pulse. Essentially we have implemented a backward TD(1) algorithm (see the appendix) this way. It is obvious that the special treatment of the reward as an extra line leading into the learning subcircuitry is not really necessary. Figure 4B shows that the input signal x0 can replace the reward signal when setting ω0 = 1 from the beginning. This shows that reward-based architectures are in some cases equivalent to stimulus-substitution architectures, and we are again approaching the original architecture of the old Sutton and Barto (1981) model. Note that the circuit diagram shown in Figures 4A and 4B cannot be implemented with analog (neuronal) hardware because it contains the forward shift of the derivative term denoted by v˜  . Figure 4C shows how to modify the circuit in order to ensure a causal structure. This requires adding the unit delay τ in the learning subcircuits to all input lines (and the reward, if an extra reward line exists). As a result, the δ signal will appear with the same delay. This modification, however, allows us also to introduce a small circuit for calculating the difference term by means of an inhibitory interneuron. The architecture in Figure 4C is presumably the simplest way to implement

258

F. Worg ¨ otter ¨ and B. Porr

A Sutton & Barto, 1981

B Sutton and Barto,1988 TD-learning

r

X

E x1

w1

S

S

X

v’

E x1

v

w1 S

x0

v’ v

C ISO-Learning X

x1

h1

x0

h0

wi

S

v’ v

X Figure 5: (A,B) Comparison between the two basic models of Sutton and Barto and ISO learning (C). Inputs = x, eligibility trace = E, reward = r, resonators = h. The symbol v denotes the difference operation between subsequent output values in A, B and the differentiation in C. The small amplifier symbol represents a changeable synaptic weight ω.

a TD rule neuronally without violating causality. 3.2 Comparing Correlation Based- and TD-Methods. Let us compare the formalism of the old Sutton and Barto (1981) model, ω1 (t + 1) = ω1 (t) + γ [v(t) − v(t − 1)]x(t),

(3.15)

with the neuronal TD procedure given by ωi ← ωi + α [r(t + 1) + γ v(t + 1) − v(t)] xi (t).

(3.16)

Learning in Sutton and Barto (1981) is correlative, as in Hebbian learning, but it depends on the difference of the output v ≈ v(t) − v(t − 1) (see  Figure 5A, differential Hebbian learning).5 Furthermore, since v(t) = i ω i xi , i ≥ 0 5 Other early differential Hebbian models have been devised by Kosco (1986) and Klopf (1986), who employ a derivative at the CS input to account for its possible inner transient representation.

Temporal Sequence, Prediction, and Control

259

in equation 3.15, we notice that the “reward” x0 does indeed occur in the stimulus substitution model of Sutton and Barto, but not as an independent term as in the reward-based TD-procedure. Reward-based architectures, like TD learning, allow treating the reward as an independent entity, which arises in addition to the sensor signals x (see Figure 5B). Thus, a reward signal could also be an intrinsically generated signal in the brain, which is not necessarily, or only indirectly, related to sensor inputs, as discussed above. In view of neuronal response recorded in the basal ganglia, this concept has strong appeal. This was one of the reasons that the TD formalism has been adopted to classical conditioning. How does ISO learning (see Figure 5C) compare to these methods? ISO learning uses different eligibility traces than Sutton and Barto (1981) and Sutton (1988) and employs them at all input pathways. Hence, they will influence learning via the output v (more specifically via v ). In the models of Sutton and Barto (1981) and Sutton (1988), an eligibility trace is applied only at input x1 and influences only the learning circuit but not the  output v. Thus, for more than two inputs, we find for ISO learning that v = ωi x¯ i ,  while for TD learning, we have v = ωi xi . If we assume, as before, that x0 is identical to the reward signal and by handling the notation concerning the derivatives in a somewhat sloppy way, we can rewrite the weight change in TD learning (see equation 3.16) as    dωk   ωi xi x¯ k (3.17) = (x0 + v )x¯ k = x0 + dt i>0 and that of ISO learning as    dωk   ωi x¯ i x¯ k . = v x¯ k = dt i=0

(3.18)

As opposed to ISO learning, the derivative in TD learning is not applied to the reward, and the output is an unfiltered (no traces) sum of the weighted inputs. These differences prevent TD learning from being orthogonal between inputs and outputs. 3.2.1 Conditions of Convergence. Furthermore, we note that conditions of convergence are different in ISO as compared to TD learning. Trivially, weight growth of ω1 stops in both algorithms when x1 = 0. Otherwise, weight growth stops in TD learning when r(t + 1) + γ v(t + 1) − v(t) = 0, a condition that requires the output to take a certain value (output condition). The orthogonality between input and output in ISO learning, on the other hand, leads to the situation that dω1 /dt = x¯ 1 v = 0 if x0 = 0, which is an input condition. It is interesting to discuss how this would affect animal learning in closed-loop situations with behavioral feedback. Animals cannot measure their “output.” All they can do is sense the consequences of an

260

F. Worg ¨ otter ¨ and B. Porr

action, hence sense the consequences of some produced output (behavior) after it has been transmitted back to the animal via the environment (see Figure 7). Thus, immediate output control can be performed only by an external observer, which might, however, misjudge the validity of an action, leading to bad convergence properties of such an algorithm. Alternatively, augmented output control could be performed by the learner after environmental filtering via its input sensors. This, however, might also go wrong if the filtering is not benign. Thus, there is a clear difference between input and output control, as discussed below (see section 4). 3.3 Neuronal Reward Systems. Before we try to embed the different algorithms into “behaving systems,” hence into closed-loop architectures, we would like to open a bracket and show why the reward paradigm appears to be so strong in animal learning. So far, we are still at the level of artificial neural network implementations of the TD rule and we mentioned only in passing that a neurophysiological motivation for this exists: the dopaminergic reward system. The computational focus of this review does not allow for an in-depth discussion of this topic, but some of the most important facts should be described before discussing the corresponding modeling approaches in section 4.2. Different response types are found in the mammalian brain that seem to be related to reward processing. We associate a few of them rather directly with the nomenclature of TD learning, noting that this is an oversimplification which is necessary to fit them into the perspective of a theoretician. (For more detailed discussions, see Schultz, 1998, 2002, and Schultz & Dickenson, 2000.) Most important, one finds: • Prediction-error neurons. These are dopamine neurons (DA neurons) in the pars compacta of the substantia nigra and the medially adjoining ventral tegmental area. These neurons seem to essentially capture the properties of the δ signal in TD learning (Miller, Sanghera, & German, 1981; Schultz, 1986; Mirenowicz & Schultz, 1994; Hollerman & Schultz, 1998, for reviews, see Schultz, 1998, 2002; Schultz & Dickenson, 2000; but see Redgrave, Prescott, & Gurney, 1999). In addition, they respond to novel, salient stimuli that have attentional and possibly rewarding properties. Before learning, these neurons respond to an unpredicted reward (see Figure 6A). During learning, this response diminishes (see Figure 6D; Ljungberg, Apicella, & Schultz, 1992; Hollerman & Schultz, 1998) and the neurons will now increase their firing in response to the reward-predicting stimulus (see Figure 6B). The more predictable the reward becomes, the less strongly the neuron fires when it appears. The δ signal in TD learning essentially corresponds to δ = reward occurred − reward predicted. Thus, these neurons respond with an inhibitory transient as soon as a predicted reward fails to be delivered (“Omission,” Figure 6C). Furthermore, these neurons

Temporal Sequence, Prediction, and Control Novelty Response: no prediction, reward occurs

A

After learning: predicted reward occurs

r

no CS

E

Continuous decrease of novelty response during learning

B

After learning: predicted reward does not occur

r

CS

D

261

CS

1.0 s

Response Transfer (Population Response)

L

G

K

0.5 s

J In

Reward Expectation

Tr

r

Reward Expectation (Population Response)

Tr Tr

r

F

H

I r

C

L

r

1.0 s

1.5 s

Figure 6: Neuronal response in the basal ganglia concerned with reward processing. (A) Response of a dopamine neuron in monkey to the presentation of a reward (r, drop of liquid) without preceding CS. (B) Response of the same neuron after learning that the CS will predict the reward. The neuron now responds to the CS but not to the reward. (C) If the expected reward fails to be delivered, the neuron will be inhibited (A–C, recompiled from Schultz et al. (1997). (D) Reduction of novelty response during learning. From top to bottom, the number of trials increases in chunks of five equivalent to a growing learning experience (D recompiled from Hollerman & Schultz, 1998). (E–J) Response transfer to the earliest reward-predicting stimulus (recompiled from Schultz, 1998). (E) Control situation, presentation of a visual stimulus (light, L) will not lead to a response. (F) Novelty response, situation similar to A. (G) Response to a reward predicting Trigger (Tr) stimulus (similar to B, left part of diagram) and (H) failure to respond to a correctly predicted reward (similar to B, right part of diagram). (I) Response to a newly learned Instruction (In) stimulus, which precedes the trigger. (J) In the same way, no response to the correctly predicted reward. (K) Response of a putamen neuron, which gradually increases and maintains its firing after a trigger until the reward is delivered ((K) recompiled from Hollerman et al., 1998). (L) Population diagram of 68 striatal neurons that show an expectation of reward type response ((L) recompiled from Suri & Schultz, 2001).

262

F. Worg ¨ otter ¨ and B. Porr

show response transfer properties: when an earlier reward-predicting stimulus is introduced, the response of the neuron during learning will move forward in time and begin to coincide with the new, earlier stimulus (see Figures 6E–J). • Reward-expectation neurons. These are in the striatum, orbitofrontal cortex, and amygdala (Hollerman, Tremblay, & Schultz, 1998; Tremblay, Hollerman, & Schultz, 1998; Tremblay & Schultz, 1999).6 They respond with a prolonged maintained discharge following a trigger stimulus until the reward is delivered (see Figures 6K and 6L). Thereby, they are similar to the “neuron” in Figure 4. Delayed delivery prolongs the response, while earlier delivery shortens it. Early, during learning, those neurons will always start to fire following the trigger stimulus, apparently naively expecting a reward in every trial. Only with experience do responses become specific to the actual reward-predicting trials. In addition, it has been suggested that these neurons fire mainly as a consequence of the motivational value of the reward and less strongly following its actual physical properties. Hence, in trials where rewards are compared, they seem to estimate the relative value of the different rewards with respect to each other instead of reacting to all of them in an absolute manner. • Goal-directed neurons. These are in the striatum, the supplementary motor area, and the dorsolateral premotor cortex (Kurata & Wise, 1988; Schultz & Dickinson, 2000). These neurons show an enhanced response prior to an internally planned motor action toward external rewards but in the absence of a triggering stimulus. Some neurons continue to fire until the reward is retrieved; others stop as soon as or just before the motor action is initiated. Using the terminology from above, one can interpret the first two types of neurons as being concerned with the prediction problem, while the third type seems to be involved in addressing the control problem. These single cell data have more recently been augmented by a substantial number of fMRI studies of the human brain, that also support the idea that the ventral striatum and other parts of the brain (e.g., amygdala, nucleus accumbens, orbitofrontal cortex) could be involved in the processing of reward- or expectation-related activity (Schoenbaum, Chiba, & Gallagher, 1998; Nobre, Coull, Frith, & Mesulam, 1999; Delgado, Nystrom, Fissell, Noll, & Fiez, 2000; Elliott, Friston, & Dolan, 2000; Berns, McClure, Pagoni, & Montague, 2001; Breiter, Aharon, Kahneman, Dale, & Shizgal, 2001; Knutson, Fong, Adams, Varner, & Hommer, 2001; O’Doherty, Rolls, Francis, Bowtell, & McGlone, 2001; O’Doherty, Deichman, Critchley, & Dolan, 2002; Pagoni, Zink, Mon6 These neurons are named according to Schultz and Dickinson (2000). Sometimes they are also called “reward-prediction neurons” (Suri & Schultz, 2001).

Temporal Sequence, Prediction, and Control

263

tague, & Berns, 2002) possibly related to TD-models (O’Doherty, Dayan, Friston, Critchley, & Dolan, 2003). However, the functional connectivity and the action of dopamine are by far more complex than suggested by the above paragraphs. Dopamine neurons ramify very broadly essentially all over the striatum. As a consequence, every striatal neuron (Lynd-Balta & Haber, 1994; Groves et al., 1995) and large numbers of neurons in superficial and deep layers of the frontal cortex (Berger, Trottier, Verney, Gaspar, & Alvarez, 1988; Williams & Goldman-Rakic, 1993) are contacted by each DA neuron. In addition, the responses of dopaminergic neurons vary to a large degree, and only a small number resemble those discussed here. Thus, the above interpretations of how the different neuron types are concerned with prediction and control may be too coarse, and the DA system may also to a large degree carry gating, motivational, novelty, saliency (Zink, Pagnoni, Martin, Dhamala, & Berns, 2003) or other context-dependent signals too. On the other hand, it has been found that dopamine-deficient mice mutants can easily perform reward learning (Cannon & Palmiter, 2003). Evidence that N-methyl-D-aspartate (NMDA) receptors and not dopamine (D2) receptors seem to be involved in specific reward processing (Hauber, Bohn, & Giertler, 2000) points in the same direction. (See also the discussion about the synaptic biophysics of the dopaminergic system section 5.) The fMRI studies also cannot help to resolve these issues, because they have shown that a variety of areas can be involved in reward processing, and some of them may be strongly modulated by attentional effects as well influencing learning (Dayan, Kakade, & Montague, 2000). In addition, the restricted spatiotemporal resolution of fMRI makes it of little help in actually designing neuronal models. This complexity cannot be exhaustively discussed in this article, and we refer readers to the literature (Schultz, 1998; Berridge & Robinson, 1998; Schultz & Dickinson, 2000; Dayan & Balleine, 2002). 4 Closed-Loop Architectures 4.1 Evaluative Feedback by Means of Actor-Critic Architectures. The action of animals or robots will normally always influence their sensor inputs. Thus, to be able to address the control problem, the algorithms discussed above will have to be embedded in closed-loop structures. To this end, so-called actor-critic models, which are strongly related to control theory, have been widely employed (Witten, 1977; Barto, Sutton, & Anderson, 1983; Sutton, 1984; Barto, 1995). Figure 7A shows a conventional feedback control system. A controller provides control signals to a controlled system, which is influenced by disturbances. Feedback allows the controller to adjust its signals. In addition, a set point is defined. In the equilibrium (without disturbance), the feedback signal X0 will take the negative value of the set point, which represents the “desired state” of the complete system. In the simplest case (set point = 0), this is zero too. The set point can

264

F. Worg ¨ otter ¨ and B. Porr

A

Disturbances

Set-Point Controller

X0

Control Signals

Controlled System

Feedback

B Context Critic Reinforcement Signal

Disturbances

Actor

Actions

Environment

(Controller)

(Control Signals)

(Controlled System)

X0 Feedback

Figure 7: Actor-critic architecture (modified from Barto, 1995). (A) Conventional feedback loop controller. (B) Actor-critic control system, where a critic influences action selection by means of a reinforcement signal.

be associated with the control goal of the system; reaching it by means of the feedback could be interpreted in the way that the system has attained homeostasis. Figure 7B shows how to extend this system into an actor-critic architecture. The critic produces evaluative, reinforcement feedback for the actor by observing the consequences of its actions. The critic takes the form of a TD error, which gives an indication if things have gone better or worse than expected with the preceding action. Thus, this TD error can be used to evaluate the preceding action. If the error is positive, the tendency to select this action should be strengthened or else lessened. Thus, actor and critic are adaptive through reinforcement learning. This relates these techniques to advanced feedforward control and feedforward compensation techniques. However, the set point is in this more general context replaced by context information. This indicates that that control goal can now be somewhat softened, which makes these architectures go beyond conventional, advanced model-free feedforward controllers. Many different ways exist to implement actor-critic architectures (see Sutton & Barto, 1998, for an example). They have become especially influential when discussing animal control, and we note that these specific architectures (see Figure 7) represent regulation problems to which animal control also belongs (Porr & Worg ¨ otter, ¨ 2003c). Other actor-critic architectures can also be designed but shall not be discussed here.

Temporal Sequence, Prediction, and Control

265

Two aspects are especially important for the discussion later: actor-critic architectures rely on the return maximization principle, which is common to all reinforcement learning paradigms, trying to maximize the expected return by choosing the best actions. Furthermore, they use evaluative feedback from the environment. Hence, feedback signals that come from the environment are not value free; instead, they are labeled reward (positive) or punishment (negative). In sections 7.1 and 7.4, we discuss how these two assumptions may pose problems when considering autonomous creatures. 4.2 Neuronal Actor-Critic Architectures. In general, all neuronal models for prediction and control that have been described in the literature so far follow an actor-critic architecture (Barto, 1995) and are focused on the interactions between the basal ganglia and the cortex (Houk, Adams, & Barto, 1995), sometimes including other brain structures as well. Most of the time the critic (predictor) is implemented in these models with great detail, while the actor (controller) in the earlier studies is only rather generally described and detail is added only in recent models. We first compare the different models of the critic and then the actors. 4.2.1 The Critic. The implementation of a TD critic shown in Figure 4C is called a reciprocal architecture because it assumes a reciprocal connection from the central summation neuron via the neuron, which calculates δ back to the synaptic modification circuit of the central summation neuron (see also Figures 8B and 8C). These types of models capture some of the basic properties of the observed neuronal responses. For example, the δ signal resembles the response of prediction-error neurons, showing the properties of response transfer and omission, while the v signal is to some degree similar to the reward-expectation neurons. Figure 8 shows a simplified circuit diagram of the basal ganglia, together with its most important inputs and outputs. We will now compare the existing models to this diagram. (For an in-depth treatment of this topic, see Daw, 2003.) • Parallel reciprocal architectures; Houk’s model. The first attempt to match the abstract architecture of a critic to the structure of cortex and basal ganglia was made by Houk et al. (1995) (see Figure 8B). So-called striosomal modules fulfill the functions of the adaptive critic. (Striosomal modules consist of striatal striosomes, subthalamic nucleus, and the DA neurons of the substantia nigra pars compacta. They are called the limbic striatum.) The prediction error (δ-) characteristics (see equation 3.13) of the DA neurons of the critic are generated by (1) equating the reward r with excitatory input from the lateral hypothalamus, (2) equating the term v(t) with indirect excitation at the DA neurons, which is initiated from striatal striosomes and channeled through the subthalamic nucleus onto the DA neurons, and (3) equating the term

266

F. Worg ¨ otter ¨ and B. Porr Frontal Cortex

Cortex (C)

A

Thalamus

VP SNr GPi

STN GPe

Striatum (S)

DA-System (SNc,VTA,RRA)

Houk et al.

C S

idealized reciprocal architecture (Montague, Suri)

B

C

C

STN +

C

S

E

Berns & Sejnowski

- DA

r

+

- DA

S

Contreras-Vidal & Schultz

D

S(VS)

+

r

- DA

Brown et al.

r

F

C (PFC)

(C) STN

S excitation

GP +

- DA inhibition

S unspecified (net excitation)

-

+DA

r

dopaminergic projection that modifies target weight

Figure 8: Matching TD learning to the basal ganglia. (A) Schematic wiring diagram of the basal ganglia showing its main inputs and outputs. VP = ventral pallidum, SNr = substantia nigra pars reticulata, SNc = substantia nigra pars compacta, GPi = globus pallidus pars interna, GPe = globus pallidus pars externa, VTA = ventral tegmental area, RRA = retrorubral area, STN = subthalamic nucleus. (B–F) Simplified circuit diagrams redrawn from the approaches of different groups adopting the same structure to make them comparable. Cortex = C, striatum = S, DA = dopamine system, PFC = prefrontal cortex, VS = ventral striatum, GP = globus pallidus, r = reward. B and C represent parallel-reciprocal architectures where both input streams to the DA system arise in parallel from the striatum, which in turn receives DA signals in a reciprocal way. Accordingly, D is a divergent (nonparallel) reciprocal architecture where the input to the DA system originates in the cortex via two separate striato-nigral pathways. The architecture in E is parallel, nonreciprocal, where the cortex is not explicitly mentioned, and that in F is divergent, nonreciprocal. Dashed lines with bullets denote a dopamine synapse. They end close to another (glutamatergic) synapse, which is the one that is modified.

Temporal Sequence, Prediction, and Control

267

v(t−1) with direct, long-lasting inhibition from striatal striosomes onto the DA neurons. This principle, depicted in Figure 8B, is a variation of Figure 4C, where we have used an interneuron to create the subtractive term. The long-lasting inhibitory component used to calculate the subtractive v(t − 1) term by Houk et al. (1995) subserves the same purpose. Is this model supported by anatomy, and will it produce the correct neuronal responses? At first, this comes down to the central question of whether a reciprocal architecture is supported by the connectivity between striatum and the DA system. Many models, like Houk’s, make the even more specific assumption that striosomes contain the main part of the reward-predicting neurons in the striatum with reciprocal connections (Houk et al., 1995; Brown, Bullock, & Grossberg, 1999; Contreras-Vidal & Schultz, 1999). This assumption is supported by the work of Gerfen (Gerfen, 1984, 1985, 1992; Gerfen, Herkenham, & Thibault, 1987). These studies in rats have shown that there is a reciprocal connection between the striosomes of the dorsal striatum and a small group of DA neurons of the SNc and SNr. In other species, anatomical evidence also supports the notion that at least weak reciprocal connections between both structures exist (Haber, Fudge, & McFarland, 2000; Joel & Weiner, 2000). There is, however, little support for the notion that this reciprocity is subcompartmentalized, because it does not seem to be the case that mostly the striosomal neurons would take part in a reciprocal connection pattern (Joel & Weiner, 2000). Rather, it seems that such connections can exist for striosomal and matrisomal neurons of the striatum in a similar way. • Comparing Houk’s model to idealized parallel reciprocal architectures. Figure 8C shows the idealized reciprocal architecture required to implement a TD rule by means of interactions between cortex (C) and striatum (S). This architecture assumes a direct excitatory pathway and an indirect inhibitory pathway. In reality the situation is reversed (Bunney, Chiodo, & Grace, 1991; Pucak & Grace, 1994; Haber et al., 2000; Joel & Weiner, 2000), and we observe as a second problem that the direct action of striatal activity onto DA neurons is inhibitory: excitation arises only indirectly as the consequence of disynaptic disinhibition (ventral striatal inhibition of GABAergic neurons in the ventral pallidum, which projects to most of the DA system; see Figure 8A). Thus, we would obtain a sign inverted situation with −v(t) and +v(t − 1), which is not in accordance with the TD rule. In conclusion, there is no support for implementing a reciprocal critic architecture relying on striosomal neurons only. However, even when relying on the entire population of striatal neurons (including the matrisomal neurons), we are still faced with the sign-inversion problem (Joel, Niv, & Ruppin, 2002). To avoid this, models must make specific

268

F. Worg ¨ otter ¨ and B. Porr

assumptions about the time courses of the involved signals. One possible solution to this problem might lie in the different, sometimes very slow, time courses of the different chemical compounds involved in the second messenger chains leading to dopamine-related synaptic modification (Houk et al., 1995). We discuss this in section 5. From a functional point of view, we find that the long-lasting inhibitory component used to calculate the subtractive v(t) term in Houk’s approach leads to the situation that this model cannot account for the precise timing of the depression during omission of a predicted reward (see Figure 6). Other timing problems arise as the consequence of the fact that this model has not implemented any kind of serial compound stimulus representation. This was first done by Montague et al., 1995, 1996) in the context of a reciprocal architecture and in the following in several models by Suri and coworkers (Suri & Schultz, 1998, 1999, 2001; Suri, Bargas, & Arbib, 2001). Suri et al. do not make an explicit link to anatomy but seem to imply that they are essentially following the approach of Houk. On closer look, however, their models are more strongly related to the ideal reciprocal architecture (see Figure 8C) than to Houk’s model (see Figure 8B). • Divergent reciprocal architectures. So far we have discussed what we would call parallel-reciprocal architectures (see the legend of Figures 8B and 8C). These models assume that two parallel pathways exist from the striatum to the DA system. An alternative source for these two streams is the limbic prefrontal cortex (PFC). Indeed, evidence exists that the PFC projects directly to the DA system (reviewed in Overton & Clark, 1997) and that an additional projection exists to the ventral striatum (Groenewegen, Berendse, Wolters, & Lohman, 1990; Parent, 1990). Via this pathway, delayed inhibition could be provided to the DA system. This is supported by findings that responses in the ventral striatum show reward expectation activity (Schultz, Apiccela, Scarnati, & Ljungberg, 1992). A model that uses such a divergent-reciprocal architecture was devised by Brown, Bullock, & Grossberg (1999; see Figure 8D). This model uses a special kind of serial compound stimulus representation by assuming a spectral timing approach (Grossberg & Schmajuk, 1989; Grossberg, 1995; Grossberg & Merrill, 1996) of a set of bandpass filtered pulses (resembling in shape that of an EPSP) with different onset times that cover the whole interstimulus interval between CS and US. In their model, these pulses represent the suprathreshold internal Ca2+ concentration. The cortex provides excitatory input to the striosomes, which project inhibitorily to the DA system (see Figure 8D). At the same time, the cortex also projects to the ventral striatum. This signal gets transferred via several stages to the DA system, where it finally exerts an excitatory action. Cortico-striatal synapses are modified in

Temporal Sequence, Prediction, and Control

269

the striosomes as well as in the ventral striatum. This architecture still fulfills the basic properties of a TD architecture; the nested, multistage computation and the synaptic modification rules used, however, no longer provide a direct link to the TD rule. The model reproduces most of the known experimental data. This is mainly a virtue of the different timing properties along both legs of the divergent input pathways to the DA system. This additional degree of freedom avoids several of the timing problems that have been found in the older models (see the discussion in Brown et al., 1999). • Parallel nonreciprocal architectures. Berns and Sejnowski (1998) have devised a parallel nonreciprocal architecture (see Figure 8E) that in essence resembles the model of Houk et al. (D), but implements several pathways, more accurately following the known anatomical structures than any of the other models. This architecture is called nonreciprocal because the weight modification does not take place in the striatum. In the model of Berns and Sejnowski (1998), weights of the STN→GP connection as well as those of the striato-nigral connection (S→DA) are modified, for which there is currently no direct evidence. The learning rule used is a three-factor Hebbian learning rule. Two factors come from the pre- and postsynaptic activity of the concerned connection; the third is a so-called error term e, calculated from the difference between the activity of the GP→DA and the S→DA connection. This error term is used as the reinforcement signal. Thus, this model does not use any primary reinforcement (reward r), and cortical input is also not explicitly mentioned. • Divergent nonreciprocal architectures. The model of Contreras-Vidal and Schultz (1999; see Figure 8F) most strongly deviates from the traditional TD rule. As in Berns and Sejnowski (1998), it assumes that the striato-nigral and not the cortico-striatal synapses are modified by means of the DA activity. In general, their model focuses on the action of the prefrontal cortex, which provides indirect input to the striosomes and from there on to the DA system. Thus, this model represents a divergent nonreciprocal architecture. The model is fairly opaque, and it seems as if this input line will lead to net excitation at the DA system, which is opposite to the assumption of the other models. Accordingly, the second pathway from the prefrontal cortex to the DA system acts inhibitorily in their scheme (divergent architecture). In addition, they have implemented an adaptive resonance network (ART-2; Carpenter & Grossberg, 1987) and use this to model an attentional and orienting subsystem. • Problems with serial compound representations. All of the more recent models use some kind of serial compound stimulus representation to cover the long temporal intervals between the stimuli. The problem of

270

F. Worg ¨ otter ¨ and B. Porr

a straightforward serial compound representation, like the one used in Figure 4, is that it predicts a gradual shift of the δ signal forward in time during learning (see steps 1–3 in Figure 4; Schultz, Dayan, & Montague, 1997). This, however, is in general not observed in real recordings where the response of the DA neurons remains in place and gradually diminishes during learning (see Figure 6D), while the response to the predictive stimulus increases. Furthermore, the model in Figure 4 also will not produce novelty responses to new and salient stimuli, which are very often observed in DA neurons. Accordingly, more recent models were designed, solving these problems by means of a more elaborate serial compound stimulus representation. For example, Suri and Schultz (1998, 1999) use a representation where only one delay of 100 ms is used between the first and all other xi , but where all xi , i > 1 are stretched in time with increasing duration and decreasing amplitude. With this architecture, they were able to more accurately reproduce the properties of real DA neurons (prediction-error neurons), including novelty responses and avoiding the forward shift of the δ signal. In this study, they found that only the one specific weight grows, which represents the temporal interval between prediction and reward. As a consequence, these models can learn only a single interstimulus interval correctly. When using other ISIs, the model will produce incorrect responses. In their later models, Suri and Schultz (2001) and Suri et al. (2001) return more closely to the architecture in Figure 4C. In addition to the serial compound representation of the stimulus, they also employ an analog eligibility trace at every xi , which they say speeds up learning and also leads to the specific shapes of the output responses. They do not show responses of prediction-error DA neurons (δ signals) during learning, and the complexity of this model is such that it is not immediately evident if it will again produce some kind of forward shift of the δ-signal. One study of Suri and Schultz (2001) focuses on the responses of reward-expectation neurons in the putamen and the orbitofrontal cortex. The other study implements a so-called extended TD model, which not only gets external stimulus input but is also internally driven by thalamic activity and by action-related signals. Thus, Suri et al. (2001) make serious attempts to implement an actor. This aspect is discussed in the next section. There is evidence that neurons in the striatum produce varying response latencies and durations (Schultz, 1998, 2002; Schultz & Dickinson, 2000) that could support the idea of a serial compound representation. Furthermore, the different models discussed so far can reproduce a wealth of physiological findings. Still, the valid parameter ranges of the existing models appear rather narrow, and the models have to be specifically tuned to reproduce the different response properties of the neurons. In addition, the wiring pattern for the serial compound rep-

Temporal Sequence, Prediction, and Control

271

resentation has to be rather specifically set up, and the unknown delay between CS and US requires a large number of neurons in the serial compound representation of xi . In general, one finds only a few shared principles in the different models. This concerns structure, where sometimes strongly differing anatomical assumptions are found between the models, as well as function, where different learning rules are used. Parallel reciprocal architectures (see Figures 8B, 8C, and 8D), which can be used to implement a neuronal version of the TD rule, are possibly supported by the connectivity of the basal ganglia. The assumption that such reciprocal connections are restricted to striosomes (which are used for the critic) does not seem to be valid, though. Idealized reciprocal architectures (see Figure 8C), which are essentially used in the models of Montague and Suri, are not directly supported by anatomy because of the sign inversion problem discussed above. From a functional point of view, accurate reproduction of the experimental data requires a fairly complex serial compound representation of the stimulus. This assumption may prove to be too inflexible, too. Divergent architectures (see Figures 8D and 8F) and nonreciprocal architectures (see Figures 8E and 8F) are not necessarily related to the TD rule anymore. Especially the divergent architectures offer additional degrees of freedom in adjusting the timing along the different input lines to the DA system. A serial compound representation is also needed in these models, and the question arises if the spectral timing approach of Brown et al. (1999) would be more flexible than the more conventional setup used in the other models. 4.2.2 The Actor. In general, less attention has been paid so far to the implementation of the actor. This may have to do with the fact that there are only few single-cell recordings available that could be matched to the activity of actor cells (goal direction neurons; see section 3.3), as opposed to the critic, where a wealth of data exist. The performance of the actor is thus judged by observing the obtained actions (“behavior”). The study of Houk et al. (1995) was also the first to suggest how to design an actor. The matrix modules (consisting of the striatal matrix, subthalamic nucleus, globus pallidus, thalamus, and frontal cortex and are called sensorimotor striatum) integrate the information from the striosomal modules (see above) and from cortical inputs in order to create output to the frontal cortex relayed through the thalamus. In contrast to the detailed discussion of the critic, Houk et al. (1995) provide only a rather general scheme of the implementation of the actor. According to their model, matrix modules generate signals that command actions or represent plans that organize other systems to generate the actual commands. Montague and colleagues (Montague et al., 1995, 1996) implement decision units as the actor in their models. This is not meant to be related to basal ganglia anatomy; rather, it represents an abstract binary decision net-

272

F. Worg ¨ otter ¨ and B. Porr

work. The TD error δ is used to influence the decision probability and to adjust the weights at the decision units appropriately during learning. This architecture allows only for binary decisions. In their 1999 model, Suri and Schultz also model a control task with an actor-critic architecture. In this model, the actor consists of one layer of neurons, each of which represents a certain action. It learned stimulus-action pairs based on the prediction error signal provided by the critic. A winnertake-all rule, implemented by means of lateral inhibition between the units, ensured that only one action was selected at a given time. In combination with the more sophisticated critic (discussed above), this still rather simple actor model was able to solve several relatively complex behavioral tasks. All of these actor models were still rather simplistic. In a recent model, Suri et al. (2001) made serious attempts to implement a more detailed actor, trying to match it to anatomy as well. As in Houk et al. (1995), the actor uses striatal matrisomes. These provide direct inhibition to the GPi/SNr complex (compare Figure 8A) and indirect excitation via the GPe and the STN. The GPi/SNr complex projects inhibitorily to the thalamus and from there on excitatorily to the cortex, which elicits the actions. The DA system provides input to the matrisomal neurons of the striatum. Its exerts an effect on the membrane potential of these cells but also on the weight of the corticostriatal synapses ω essentially with ω = DA × Pre × Post, comprising a three-factor learning rule, which relies on the activity of the DA system (DA) as well as the pre- and postsynaptic activity at the striatum cell (Miller et al., 1981; Schultz, 1998). The membrane activity of these cells follows the physiological observation that they can be in a depolarized “up” and a hyperpolarized “down” state. (We refer readers to section 5 for a more detailed description of these states.) Here, it seems that this property can improve reaction times as well as learning properties as compared to a model version without such membrane states. As in their older model, actor neurons represent single actions each, and actions are selected again by a winner-takes-all mechanism such that the winning inhibition will disinhibit the thalamus. As a consequence, the number of possible actions is limited to the number of actor neurons. The use of internal signals in the extended TD model of the critic in the model of Suri et al. (2001) leads to the situation that this model can now also perform some kind of planning. Planning refers to the behavioral strategy that an action can be selected without primary reinforcement, just as the consequence of having memorized previously experienced situation-action pairs. For example, if a reward has always occurred at the right side in a T-maze, the animal after learning will turn right without any external stimulation. Suri et al. (2001) achieve this by means of internal signals that enter the extended TD model and from there on the actor neurons. One could say that these internal signals form an internal, explicit representation of a chain of events that can be used for learning and acting.

Temporal Sequence, Prediction, and Control

273

Figure 9: Correlation-based control architecture, where the control signal is derived from the correlations between two temporally related input signals.

The performance of this model is rich and reproduces many existing data sets. However, its architecture is fairly advanced, and as a consequence it contains many free parameters, few of which have direct physiological support. The assumption that striosomes form part of the critic while matrisomes belong to the actor is not supported by the anatomy of the basal ganglia, as discussed above. Therefore, it is questionable to employ two different learning rules (TD rule versus three-factor rule) at these two sets of neurons. From a functional point of view, up- and down-membrane states are also found across all medium spiny striatal neurons (for a review, see Nicola, Surmeier, & Malenka, 2000) and not only in the matrix neurons. From a more conceptional point of view, two aspects may be problematic in the currently existing actor models. First, the number of possible actions is matched to the number of actor neurons. Thus, these neurons represent some kind of “grandmother cell” concept, where only a rather limited number of discrete decisions is possible. This may work in restricted-choice lab situations. The complexity of real-world situations, however, requires a different type of action network. Second, in general, the basal ganglia include many indirect pathways, which often amount to disinhibition. Disinhibition, however, normally acts permissive or facilatory and cannot directly be equated with excitation. Specific motor commands, on the other hand, require the concerted action of many specific and sometimes temporally rather finely tuned excitations. We believe that the release of inhibition at the thalamus performed by the actor pathway in, for example, the model of Suri et al. (2001) can at best act gating on such motor actions. Thus, action selection by means of such a gating process seems to be too crude a model for animal (motor) control. 4.3 Nonevaluative Correlation-Based Control. We have stated that actor-critic architectures follow the return maximization principle by means of evaluative feedback from the environment. Figure 9 suggests a schematic architecture that accommodates a different approach: nonevaluative feedback and disturbance minimization. This architecture utilizes the basic feed-

274

F. Worg ¨ otter ¨ and B. Porr

back loop controller from Figure 7A, but it assumes that the environment will, in a temporal sequence learning situation, provide temporally correlated signals about upcoming events like those mentioned in the introduction (e.g., smell predicts taste). This architecture follows the learning goal: learn to keep the later signal (x0 ), whatever it is, minimal by employing the earlier signal (x1 ) to elicit an appropriate action. In conventional actor-critic architectures, the critic provides an evaluation of the action (e.g., “good” or “bad”), and this evaluation influences future action selections. Such evaluations are, however, always subjective. In correlation-based control, the situation is fundamentally different. Here, the system relies on the objective difference between “early” and “late,” which arises from the structure of the input signals. Evaluations do not take place at this point. Instead the “re-”action of the feedback control loop in response to x0 , be it an attraction or a repulsion reaction, will be shifted forward in time to occur earlier now in response to x1 . Thus, in this system, evaluations do not take place during learning; instead, they are implicitly built into the (sign of the) reaction behavior of the inner x0 loop: repulsion or attraction. As a consequence, critic and actor are not necessarily separate entities anymore and can be merged into the same architectural building block. Furthermore, we note that this control strategy does not seek to maximize returns. On a complex decision topology, such maxima (or even near-maximal regions) may be hard to find by means of RL. Disturbance minimization, on the other hand, offers the advantage that the associated regions will almost always be much bigger, promising better convergence properties. 4.3.1 ISO Control: Merging Critic and Actor. In this section, we describe how ISO learning can be used to implement correlation-based control introduced in an abstract way in the previous section. The central assumption of ISO control is that any control system should start with a stable negative feedback loop (see Figure 9), for example, a reflex loop. Feedback controllers, however, suffer from a major disadvantage: they will always react only after a disturbance has taken place (see the inner loop in Figure 10A). Thus, the desired state (e.g., x0 = 0; see also Figure 9) cannot be maintained all the time. In other words, disturbances will not yet be minimal when employing feedback control. ISO control can improve on this if a temporal correlation exists between the primary disturbance and some other earlier-occurring signal (denoted by the delay τ between the inner and the outer loop in Figure 10). The ISO learning algorithm allows for learning this correlation, and as a result, the primary reflex reaction will be “shifted forward” in time, now occurring earlier, that is, before the primary reflex would have been triggered. Thus, if learning is successful, the primary reflex will be fully avoided, and disturbances are now minimal (ideally zero). Note that in the architecture shown in Figure 10A, critic and actor are no longer separate (compare Figure 7B with Figure 9). There is indeed recent experimental evidence that neurons exist where behavior and reward (actor and critic) are

Temporal Sequence, Prediction, and Control

275

Figure 10: Applying ISO learning in a control task. (A) This architecture is reminiscent of an actor-critic architecture (see Figures 7 and 9), but here the system does not use evaluative feedback (“rewards”) from the environment. Instead, it relies only on correlations between the inputs. Hence, it is assumed that the “organism” receives temporally correlated inputs, where x1 arrives earlier (e.g., a signal from a range finder reflected from an obstacle) and x0 arrives later (e.g., a signal from a touch sensor triggered at the moment of touching the obstacle). P0 , P1 denote environmental transfer functions, D a signal (“disturbance”), which arrives at the inputs: undelayed at x1 and with delay τ at x0 . Other symbols are as in Figure 3. Here we have also implemented a filter bank of 10 filters with different frequencies, all driven by the same input x1 , creating something like a serial compound representation. This was done purely to speed up learning and to create smoother output signals. Note that a filter bank approach does not destroy orthogonality and weights will still self-stabilize (see Figure 3D). After successful learning, the output V will fully compensate the disturbance D at the summation node of the inner loop leading to x0 = 0, which is equivalent to a functional elimination of the inner loop. The system has learned the inverse controller of the inner loop (Porr, von Ferber, & Worg ¨ otter, ¨ 2003). (B,C) Trajectory of a real robot early (B) and late (C) during learning in an arena with three obstacles (boxes). Collisions are denoted by the small circles (forward = solid, backward = dashed). Only forward collisions can be used for learning. In such an environment, the robot never needed more than 12 forward collisions to learn the task. This way, it is as fast as the best RL algorithms, which require sophisticated credit structuring and temporal assignment mechanisms. (Touzet, 1999; also Touzet, pers. communication, August 2003).

276

F. Worg ¨ otter ¨ and B. Porr

more closely linked (Hollerman & Schultz, 1998; Kawagoe, Takikawa, & Hikosaka, 1998; Hassani, Cromwell, & Schultz, 2001). This principle has been employed in several real-robot experiments (see Figures 10B and 10C), (http://www.cn.stir.ac.uk/predictor). We simulate touch and range-finder signals. Before learning, the simulated robot will perform a built-in retraction reaction when touching an obstacle (primary reflex reaction). All weights ωk are initially zero except the weights that belong to the touch sensor inputs, which we set to one. Thus, the output is at this stage just the signal v = x¯ 0 , where x¯ 0 is the bandpass-filtered touch senor input x¯ 0 = h0 ∗x0 . This signal is sent sign-inverted (negative feedback), but otherwise unaltered, to the motors,7 which leads to a retraction reaction. The range-finders provide the necessary earlier signal because they respond before the touch sensor is triggered.  ISO learning learns this correlation. After learning, the output is v = ωk x¯ k , where k ≥ 1, because the touch sensors (x0 ) are no longer triggered. This signal will now, in same way as before but earlier, lead to a retraction reaction, and the primary reflex will be avoided. Interestingly, the principle of disturbance minimization by reflex avoidance can be employed in the same way to learn a food-retrieval task (see Porr & Worg ¨ otter, ¨ 2003b). Here a behavior emerges that looks like reward retrieval (or return maximization) but actually follows the disturbanceminimization principle. 5 Biophysics of Synaptic Plasticity in the Striatum The more recent of the discussed models of the basal ganglia begin to make rather specific assumptions about cellular and subcellular mechanisms. The striatum is currently the best-understood substrate concerning the interactions between glutamatergic (Glu) and dopaminergic (DA) synapses. Such interactions also take place in many other brain structures, but shall not be discussed here. Furthermore, we will restrict the discussion to facts that are now fairly generally acknowledged. This is meant to facilitate the computational perspective and should allow us to design restricted but at least valid models without too many open degrees of freedom. 5.1 Basic Observations. Figure 11 gives a summary of the different types of synaptic modifications at corticostriatal glutamatergic synapses targeting a spiny projection neuron in the striatum. Three possible influences can in principle affect the synapse: (1) presynaptic stimulation of the corticostriatal pathway, (2) postsynaptic activation of the projection neuron, and (3) activa7 This description is slightly simplified, because we employ steering and accelerating control, and thus two sets of neurons. The correct cross-wiring is described in Porr and Worg ¨ otter ¨ (2003a). Of importance here is that ISO control essentially works without any signal postprocessing or conditioning.

Temporal Sequence, Prediction, and Control

Nigrostriatal (”DA”) DA

Corticostriatal (”pre”) Glu

Medium-sized Spiny Projection Neuron in the Striatum (”post”)

277 pre

post

DA

Result

1

X

0

0

0

2

0

X

0

0

3

0

0

X

0

4

X

X

0

LTD (dl)

5

X

0

X

0

6

0

X

X

0

7

X

X

X

LTD (DA tonic) LTP (DA phasic)

LTP (dm)

*

Figure 11: Effects of the different activation protocols on the plasticity of corticostriatal synapses. The small x in the table denotes an active influence. Only paired pre- and postsynaptic activation (row 4), with or without DA activation (row 7) will lead to synaptic plasticity. For further explanations, see the text. References for the table: row 1: Calabresi et al., 1992a; Choi & Lovinger, 1997; Calabresi, Centonze, Gubellini, Marfia, & Bernardi, 1999. Row 2: Calabresi et al., 1992b; Choi & Lovinger, 1997; Calabresi et al., 1999. Row 3: Calabresi et al., 1987, 1992b; Umemiya & Raymond, 1997. Row 4: (LTD) Calabresi et al., 1992b; Lovinger, Tyler, & Merritt, 1993; Walsh, 1993; Wickens et al., 1996. Row 4: (LTP and mixed effects): Charpier & Deniau, 1997; Charpier, Mahon & Deniau, 1999; Akopian, Musleh, Smith, & Walsh, 2000; Partridge et al., 2000; Spencer & Murphy, 2000. Row 5 (nucleus accumbens) Pennartz, Ameeron, Groenewegan, & Lopes da Silva, 1993. Row 5: (striatum): Calabresi et al., 1999. Row 6: Calabresi et al., 1999. Row 7 (LTP with Mg2+ removed): Calabresi et al., 1992b; Walsh & Dunia, 1993. Row 7 (LTD with normal conditions, i.e., with Mg2+ ): Calabresi et al., 1992a, 1992b; Tang et al., 2001. Row 7 (LTP with pulsed DA application): Wickens et al., 1996.

tion of the nigrostriatal, dopaminergic pathway. By itself, none of the three influences can affect the Glu synapse (top; for literature references, see the figure legend). If, however, presynaptic corticostriatal stimulation is paired with postsynaptic activity, early studies have unequivocally reported that LTD occurs at the Glu synapse. More recently, these observations have been augmented by findings showing that LTP occurs mainly in the dorsomedial (dm) part of the striatum, whereas LTD is found in the dorsolateral (dl) part following the gradient of D2-like receptor density (Joyce & Marshall, 1987; Russell, Allin, Lamm, & Taljaard, 1992). In addition, the expression of LTP or LTD will also depend on the applied stimulation protocol (for review, see Reynolds & Wickens, 2002). In general, the tendency for LTD seems to be stronger than that for LTP, indicated by the different font sizes in the figure. Failing to pair pre- and postsynaptic activation, on the other

278

F. Worg ¨ otter ¨ and B. Porr

hand, will induce neither LTP nor LTD, regardless of the activation state of the dopaminergic pathway. The most robust protocol to induce LTP or LTD consists of the stimulation of all three influences. It was found that tonic, long-lasting activation of the DA pathway will induce LTD (Calabresi, Maj, Mercuri, Bernardi, 1992a; Calabresi, Maj, Pisani, Mercuri, & Bernaldi, 1992b; Tang, Low, Grandy, & Lovinger, 2001), whereas phasic, pulse-like activation leads to a less strong D1-like receptor desensitization (Memo, Lovenberg, & Hanbauer, 1982) will lead to LTP (Wickens, Begg, & Arbuthnott, 1996). Thus, this has been called a “three-factor” learning rule (Miller et al., 1981; Schultz, 1998). This terminology, however, may be misleading. Dopamine is a potent modulator of synaptic plasticity, but should not enter the equation in a multiplicative way, because the conjoint action of pre- and postsynaptic influences (middle of table, row 4) seems to suggest that two factors already suffice for synaptic modification, and in this case we would ideally set dopamine = 0. It is, however, currently a matter of debate whether the dopamine concentration indeed approaches zero under these experimental conditions, because traces of DA can be detected by high-pressure liquid chromatography (HPLC) following high-frequency stimulation of the corticostriatal pathway (Calabresi et al., 1995). Thus, it is still conceivable that under physiological conditions, the lack of a DA signal would prevent LTP or LTD altogether, because chronic near total depletion of DA prevents LTD (Calabresi et al., 1992a) and LTP (Centonze et al., 1999) (chronic denervation protocols). As a consequence, the validity of a three-factor rule cannot be conclusively ruled out. 5.2 Intracellular Processes. Figure 12 shows a summary of the intracellular actions that take place at DA and Glu synapses as far as they are understood today. The diagram is to some degree simplified; and more detailed accounts can be found in Greengard, Allen, and Nairn (1999) and Centonze, Picconi, Gubellini, Bernardi, and Calabresi (2001). Both LTP and LTD are introduced in the striatum by repetitive stimulation of the corticostriatal fibers, and this event produces massive release of both glutamate and dopamine. In the striatum, dopamine acts mainly via two receptor subtypes (D1-like, and D2-like; Sibley & Monsma, 1992). We consider here the action of glutamate onto alpha-amino-3-hydroxy-5-methyl-4-propionate (AMPA) and N-methyl-D-aspartate (NMDA) receptors only. The more modulatory action of metabotropic glutamate receptors shall not be discussed. Synaptic strength, measured, for example, by the size and slope of EPSPs, can partly be associated to the number of phosphorilated AMPA and NMDA receptors (p-AMPA and p-NMDA) which are integrated into the membrane, while dephosphorilated receptors are not active. In general, one observes that LTP occurs only when NMDA channels are active (Calabresi, Pisani, Mercuri, & Bernardi, 1996; Yamamoto et al., 1999; Partridge, Tang, & Lovinger, 2000), which in experimental in vitro conditions can be achieved by removing Mg2+ from the medium (Calabresi et al., 1992c; Calabresi, Pisani, Mercuri,

Temporal Sequence, Prediction, and Control

A

279

DA

Receptor types Dopamine Glu

NO

DA DA

Ca

AMPA

P

cGMP

2+

Ca

PKG

PP-1

5)

p-DARPP32

(T7

AMPA L-Ca

CaMKII

T75

PKA

2+

NMDA

CaM

(T34)

cAMP 2+

Ca

Glutamate

NMDA

D1-like D2-like

PP-2B

T34

DARPP32

B

Glu

AMPA

P

DA

Ca

2+

NMDA

CaMKII

PP-1

2+

(T34)

cAMP

CaM

p-DARPP32

Ca

2+

Ca

PP-2B

PKA

T34

DARPP32

DA

C

Glu DA

NO

AMPA

NMDA

P

DA

Ca2+

2+

Ca

PP-1

cGMP

CaM CaMKII

cAMP

)

(T75 PKA

p-DARPP32

PKG

PP-2B

T75

DARPP32

Figure 12: Biophysics of striatal synapses (see the text).

& Bernardi, 1996; Centonze et al., 1999). In vivo, this would require a depolarized state of the neuron, where the Mg2+ block at the NMDA channels is lessened or removed. LTD, on the other hand, is independent of the NMDAchannels (Calabresi et al., 1996; Yamamoto et al., 1999; Partridge et al., 2000) and can thus also take place in vivo during less depolarized membrane states. Basically we distinguish three pathways that can lead to the modification of the synaptic strength of the Glu synapse (see Figure 12A).

280

F. Worg ¨ otter ¨ and B. Porr

1. Glu-NMDA-Ca2+ -CaMKII pathway (right side of the diagram). This is the traditional pathway involved in the generation of LTP. During elevated calcium levels, the increased activity of Calcium/Calmodulindependent Protein Kinase II (CaMKII) leads to an increase in the phosphorilated receptors at the membrane. A counteraction arises, however, from Protein Phosphatase-2B (Calcineurin, PP-2B), which gets stimulated by Ca2+ and leads to a dephosphorilation of DARPP32 (dopamine and cyclic adenosine 3’-5’-monophosphate-regulated phosphoprotein, 32kDa) (King et al., 1984; Nishi, Snyder, & Greengard, 1997). This acts by dephosphorilating the receptors along the second pathway, discussed next. The complete cascade involved in the first pathway is more complex than drawn here (see section 6.1 for a more detailed discussion of the literature). 2. DA-D1-PKA-pDARPP32 Pathway (outer left part of the diagram). The action of dopamine onto D1-like receptors leads to an elevated level of cyclic AMP (cAMP) and increased Protein Kinase A (PKA) stimulation (Stoof & Kebabian, 1981). PKA can directly act by phosphorilating AMPA channels (Roche, O’Brien, Mamnen, Bernhardt, & Huganir, 1996; Tingley et al., 1997). At the same time, PKA shifts the equilibrium of DARPP32 toward its active phosphorilated form (pDARPP32) by phosphorilating its threonine-34 residue (T34 in the diagram: DA induced, Nishi et al., 1997; adenosine induced, Svenningsson et al., 1998), which inhibits the action of PP-1 (protein phosphatase1, Hemmings, Greengard, Tung, & Cohen, 1984). The protein PP-1 normally acts by dephosphorilating Glu receptors. Thus, for this pathway, we find an inhibition (by p-DARPP32) of dephosphorilation (by PP-1). In a side path, PKA also helps phosphorilating L-type Ca2+ channels (Surmeier, Bargas, Hemmings, Nairn, & Greengard, 1995), which adds to the calcium pool. 3. DA/NO-D2-PKG-pDARPP32 Pathway (inner left part of the diagram). The action of dopamine onto D2-like receptors seems less well understood. Recent observations from Calabresi et al. (2000) suggest that there is at least a twofold action possible. It seems that dopamine can directly act on the D2-like receptors, which leads to an inhibition of PKA (Centonze et al., 2001). However, at the same time, there exists an indirect pathway via Nitrousoxide (NO)-syntase positive interneurons (Morris et al., 1997). At these neurons, dopamine acts on D1-like receptors, which leads to the release of NO at their terminals.8 NO enhances the level of cyclic Guanosine monophosphate (cGMP) (Al-

8 This explanation is currently used to explain why a conjoint stimulation of D1-like and D2-like receptors (however, at different neuron subtypes) leads to LTD, while the stimulation of D1-like receptors only will lead to LTP (Centonze et al., 2001).

Temporal Sequence, Prediction, and Control

281

tar, Boyar, & Kim, 1990) and stimulates Protein Kinase G (PKG). PKG increases the level of phosphorilated DARPP32, this time, however, phosphorilating Thr34 and Thr75 (T75 in the diagram; (Calabresi et al., 2000). This type of phosphorilated DARPP32 acts inhibiting onto PKA (Bibb et al., 1999) but cannot efficiently inhibit PP-1. As a consequence, dephosphorilation can take place more easily at the Glu receptors. In summary, pathway 1 acts actively phosphorilating, pathways 2 acts mainly “permissive,” preventing dephosphorilation of the Glu receptors, and pathway 3 facilitates dephosphorilation. The question arises under which membrane-potential conditions these pathways are likely to be triggered. It is known that in vivo striatal projection neurons fluctuate between a hyperpolarized “down-state” and a depolarized “up-state” (for a review, see Nicola et al., 2000). The downstate corresponds approximately to the resting membrane level in a slice. Dopamine exerts different effects in the down- and the up-state. In the down-state, dopamine increases via D1-like receptors the activity of rectifying potassium currents (Pacheco-Cano, Bargas, Hernandez-Lopez, Tapia, & Galarraga, 1996) and thereby counteracts possible depolarizing influences. Near the up-state, one still observes reduced excitability from the action of the D1 receptors which leads to a decrease of Na+ as well as N- and P-type Ca2+ currents (Calabresi, Mercuri, Stanzione, Stefani, & Bernardi, 1987; Surmeier & Kitai, 1993; Surmeier et al., 1995). If sustained depolarization (e.g., from the corticostriatal pathway) drives the neuron into the up-state, dopamine will act differently. In this case, it influences again via the D1-like receptors an L-type Ca2+ -current (Hernandez-Lopez, Bargas, Surmeier, Reyes, & Galarraga, 1997) and stabilizes the depolarization this way. Thus, one could say that dopamine will in both states introduce a hysteresis to the membrane potential behavior, trying to keep it at the currently existing level. This principle has recently been modeled to some detail by Gruber, Solla, Surmeier, and Houk (2003), who report such a hysteresis effect in a membrane model that includes some aspects of the D1 cascade and L-type Ca2+ -currents. Note that such a hysteresis will also act as a thresholding mechanism, filtering weak inputs and letting stronger ones through to the pallidum (Yim & Mogenson, 1982; Brown & Arbuthnott, 1983; Toan & Schultz, 1985). Above, we have stated that in vivo LTP requires an elevated membrane potential level to remove the Mg2+ block from the NMDA-channels. Figure 12B shows in a schematic way how this would affect the concentrations of the different compounds. Active NMDA channels will lead to a strongly elevated level of Ca2+ , and this will lead, via CaMKII, to an increased tendency to phosphorilate Glu-receptors (pathway 1). Thus, plain NMDA influence, without the action of dopamine, can already lead to LTP. This conforms with the observation that pre- and postsynaptic stimulation without DA can induce LTP. It seems, however, that the depolarization level reached by this

282

F. Worg ¨ otter ¨ and B. Porr

protocol is often too weak to allow for a large enough Ca2+ influx. Thus, LTD occurs many times, which is normally associated with lower levels of Ca2+ (Malenka & Nicoll, 1999). However, if at the same time D1-like receptors are activated, depolarization gets stabilized (up-state), and pathway 2 becomes active. The increased action of L-type Ca2+ channels by PKA will increase the calcium level, and the active form of p-DARPP32 is substantially enhanced. This removes the possible dephosphorilation via PP-1 and enhances the LTP effect (preventing LTD). A balancing effect arises, though, from the increased level of Ca2+ , which stimulates PP-2B, which in turn reduces p-DARPP32 (glutamate-mediated dephosphorilation of p-DARPP32; King et al., 1984; Nishi et al., 1997). During a less depolarized membrane state (see Figure 12C), pathway 1 is largely inactive, because less Ca2+ can enter via the NMDA channels. This leads to a strongly reduced tendency of Glu-receptor phosphorilation. On the other hand, we also find that lack of Ca2+ leads to a reduced stimulation of PP-2B. The final balance of these two opposing effects, however, shifts the DARPP32 equilibrium toward its dephosphorilated, inactive form. The action of dopamine in conjunction with LTD is less well understood. If D2-like receptors get stimulated via the indirect NO pathway, DARPP32 gets activated via PKG. In vitro studies suggest that PKA and PKG, in a similar way, phosphorilate threonine-34 at DARPP32 but PKG in addition phosphorilates Thr75 (Calabresi et al., 2000). This difference may explain why p-DARPP32 activated by PKG is less efficient in inhibiting PP-1, but final, conclusive experimental evidence for this is still missing. At the moment, however, it seems safe to say that, taken together, the expected level of active, p-DARPP32 (Thr34) should be substantially lower than in the depolarized state discussed above. As a consequence, PP-1 does not get inhibited much, and it can exert its dephosphorilizing action. In parallel, the PKA pathway is inhibited by the action of the D2-like receptor cascade. Thus, the direct phosphorilizing action of PKA does also not take place. As the final result, we expect a reduction of phosphorilated Glu receptors, which leads to LTD. 5.3 Reassessment of the Different Learning Rules in View of Synaptic Biophysics. The basic neuronal formalism for the TD-rule has been given in equation 3.14 as ωi ← ωi + α δ(t) xi (t),

(5.1)

which states that a multiplication of the prediction error signal with the predictive stimulus (-trace) should drive the synaptic weight change. This rule would in principle permit learning by the correlation of the nigrostriatal DA input with the corticostriatal Glu input only, without requiring postsynaptic activity. This, however, has not been found (see the case marked by an asterisk in Figure 11). The fact that the δ signal of the TD rule is in turn derived from the neuron’s output may, however, implicitly account for this, because

Temporal Sequence, Prediction, and Control

283

trace or other temporal representation

d X1

DA

w1 x BP v(t) X0

reward

w0 = 1

Figure 13: Schematic diagram of the modulatory action of dopamine inputs (DA), calculated by a reciprocal TD architecture, onto the synaptic weight modification of a cortico-striatal synapse ω1 (gray box). The main mechanisms for the weight change is assumed to be Hebbian by means of the correlation x between the trace of a presynaptic signal (x1 ) with a postsynaptic signal (BP) in the form of a backpropagating spike, which arises in response to the input x0 .

unavoidably there will be postsynaptic activity at the striatal neuron as soon as this circuit becomes activated. The discussion of the biophysical mechanisms above strongly supports the view that the “traditional” Hebbian correlation between input (presynaptic) signal and output (postsynaptic) signal should be fundamental to the weight change, while the DA signal performs a strong modulatory action. Currently it is widely believed that Hebbian correlation between input and output, which can lead to synaptic plasticity, is mediated by backpropagating action potentials (BP spikes) or dendritic spikes (DS-spikes), which are actively or passively transmitted to the regarded synapse (Stuart & Sakmann, 1994; Buzsaki, Penttonen, Nadasdy, & Bragin, 1996; Hoffman, Magee, Colbert, & Johnston, 1997; Migliore, Hoffmann, Magee, & Johnston, 1999; for a review, see Linden, 1999, but see Goldberg, Holthoff, & Yuste, 2002). The underlying biophysical mechanisms shall be discussed later (section 6.1). In this simplified connection scheme, it is, however, conceivable that the activation of the circuit that calculates the δ signal will be associated with a backpropagating spike that travels to the synapse and provides the necessary postsynaptic depolarization. This would lead to a three-factor rule combining input, output, and the DA signal, where the DA signal and the postsynaptic activation are causally coupled (Miller et al., 1981; Schultz, 1998; Suri et al., 2001). There are, however, still some problems remaining. For example, if the result holds that pure pre-post correlation without DA signal will drive LTP and LTD, then we should not use the DA input in a multiplicative way at all. In addition, the problem occurs that we have to deal with the effects of all causally connected

284

F. Worg ¨ otter ¨ and B. Porr

events whenever presynaptic activity will drive the cell into (postsynaptic) firing. This occurs in a rather complex way in all neuronal TD models, which use a serial compound stimulus representation. Thus, the question of how all these inputs will drive the cell before and during learning, generating BP spikes, which by themselves will influence learning (together with DA), needs to be carefully addressed when trying to model temporal sequence learning by a rule that combines inputs and outputs causally. While the answer to this is still unknown, it is nonetheless clear that a plain three-factor rule is too simple to account for this. Figure 13 tries to capture these aspects in a schematic way. The circuit on the right side would be suited to calculate the δ signal by computing v(t) − v(t − 1) using delayed inhibition via an interneuron. Synaptic modification of the x1 synapse requires the correlation between pre- and postsynaptic activity (see the gray box with x) and the postsynaptic activity is supposed to be transmitted to the synapse via a BP spike (dashed line). The DA signal acts modulatory onto this correlation, but we will not rule out the possibility that it could have a gating (multiplicitive) influence, preventing LTP when it is absent. In comparison to TD, ISO learning does not attempt to model DA responses. In section 6.2.3, we will show that its algorithmic structure is more strongly related to models of spike-timing-dependent plasticity correlating presynaptic signals with postsynaptic signals only (see below). 6 Fast Differential Hebb—Relations to Spike-Timing-Dependent Plasticity The discussion has shown that there are certain—at least formal—similarities between the TD rule, the three-factor rule, and conventional correlationbased learning. Specifically, differential Hebb rules in general lead to weight growth for causally related temporal sequences, while weight shrinkage occurs when the signals are inverse causally coupled. Such a bimodal characteristic was first observed in the classical model of Sutton and Barto (1981), where inhibitory conditioning was found for negative interstimulus intervals (ISIs) between CS and US (an unwanted effect in this context). On much shorter timescales, effects have been observed at real neurons, where the sequence of pre- and postsynaptic activation influences synaptic modification. A different set of differential Hebbian learning rules has been developed to explain these effects. We will discuss some of these algorithms in this section. We will also treat the biophysics of spike-timing-dependent plasticity in order to put the learning rules into their biophysical context. 6.1 Spike-Timing-Dependent Plasticity: A Short Account. 6.1.1 Basic Observations. Hebbian (correlation-based) learning requires that pre- and postsynaptic spikes arrive within a certain small time window, which leads to an increase of the synaptic weight (Hebb, 1949). Originally

Temporal Sequence, Prediction, and Control

285

Synaptic change [%]

100

0

-60

-80

0 T (ms) 80

Figure 14: Experimental data and exponential curve fits for spike-timingdependent plasticity in a hippocampal glutaminergic neuron after repetitive correlated firing (pulses at 1Hz). The right part of the diagram shows LTP, the left part LTD. LTP occurs when the presynaptic activation precedes postsynaptic firing (defined as T > 0); for LTD the order of pre- and post- is reversed (defined as T < 0). Recompiled from Bi and Poo (2001).

it had been supposed that the temporal order of both signals is irrelevant (Bliss & Lomo, 1970, 1973; Bliss & Gardner-Edwin, 1973). However, rather early first indications arose that temporal order is indeed important (Levy & Steward, 1983; Gustafsson, Wigstrom, Abraham, & Huang, 1987; Debanne, Gahwiler, & Thompson, 1994). This notion has been extended into a clear picture of spike-timing-dependent plasticity (STDP) by the experiments of Markram, Lubke, ¨ Frotscher, and Sakmann, (1997) as well as Magee and Johnston (1997). Markram et al. (1997) used whole-cell recordings from two interconnected layer V cortical pyramidal neurons, and they found that repetitive costimulation of the two cells, with postsynaptic spikes following presynaptic spike by 10 ms, induced LTP, while reversing the pulse-protocol would induce LTD. Magee and Johnston (1997) found in the hippocampus that subthreshold synaptic inputs into a CA1 pyramidal cell could amplify backpropagating spikes when paired with postsynaptic stimulation. This results in Ca2+ influx and LTP. In the following, a large number of studies have confirmed that the expression of LTP or LTD in many cells depends on the order of pre- and postsynaptic stimulation (Bell, Han, Sugawara, & Grant, 1997; Bi & Poo, 1998; Debanne, Gahwiler, & Thompson, 1998; Zhang, Tao, Holt, Harris, & Poo, 1998; Egger, Feldmeyer, & Sakmann, 1999; Feldman, 2000; Nishiyama, Hong, Mikoshiba, Poo, & Kato, 2000). Figure 14 shows a typical example of how a synapse in the hippocampus changes when ap-

286

F. Worg ¨ otter ¨ and B. Porr

plying a protocol of pairing single pre- and postsynaptic depolarizations (Bi & Poo, 2001). The curve shows that this synapse decreases in strength when the postsynaptic signal precedes the presynaptic signal (defined here as T < 0), while it grows if the temporal order is reversed (thus, T > 0) (Markram et al., 1997; Magee & Johnston, 1997; Bi & Poo, 2001). T denotes the temporal interval between post- and presynaptic signals (T := tpost −tpre ). This fairly symmetrical diagram (see Figure 14) represents in some sense the most generic type of STDP. However, several other timing-dependent synaptic modification curves have also been found, which shall only be briefly mentioned here (for a review, see Bi, 2002; Roberts & Bell, 2002). For example, Feldman (2000) found that in layer II/III pyramidal cells, the LTD part of the curve is strongly extended. In the cerebellum (electric fish), inverse STDP curves have been observed, where T > 0 induces LTD and T < 0 LTP (Bell et al., 1997; Han, Grant, & Bell, 2000). Other cell types (e.g., spiny stellate cells in layer IV of the cortex; Markram et al., 1997) seem to follow a more traditional Hebbian learning characteristic. In addition, one observes that individual STDP curves can show a high degree of variability even within the same cell class and that the zero crossing between the LTP and LTD part of the curve does not necessarily occur at T = 0. 6.1.2 Synaptic Biophysics of STDP. Currently it is widely believed that Hebbian correlation between input and output, which can lead to synaptic plasticity, is mediated by backpropagating action potentials (BP spike) which are actively or passively transmitted to the regarded synapse (Stuart & Sakmann, 1994; Buzsaki et al., 1996; Hoffman et al., 1997; Migliore et al., 1999) by means of passive and active properties of dendrites (Lasser-Ross & Ross, 1992; Regehr, Konnerth, & Armstrong, 1992; Johnston, Magee, Colbert, & Cristie, 1996). At distal dendrites, where a BP spike might already have faded, dendritic spikes can serve the same purpose (Stuart, Spruston, Sakmann, & H¨ausser, 1997; Golding, Kath, & Spruston, 2001; Saudargiene, Porr, & Worg ¨ otter, ¨ 2004; Mehta, 2004). It is generally believed that a transient increase in intracellular Ca2+ is of crucial importance for STDP as in most other forms of synaptic plasticity (Artola & Singer, 1993; Zucker, 1999). In hippocampal slices, increased Ca2+ influx is correlated with the induction of LTP (Magee & Johnston, 1997). In addition, STDP is found to depend critically on NMDA receptors that are highly permeable to Ca2+ (Magee & Johnston, 1997; Markram et al., 1997; Bi & Poo, 1998; Debanne et al., 1998; Zhang et al., 1998; Feldman, 2000; Nishiyama et al., 2000). A simple model of the mechanisms that underlie LTP and LTD can be summarized as follows: high-level intracellular Ca2+ elevation activates the calcium/calmodulin-dependent kinase II (CaMKII, see Figure 12) as well as other protein kinases and leads to subsequent LTP, whereas moderate-level Ca2+ elevation activates phosphatases (e.g., calcineurin) and results in LTD (Lisman, 1989; Malenka et al., 1989; Malinow, Schulman, & Tsien, 1989; Artola & Singer, 1993; Mulkey, Endo, Shenolikar,

Temporal Sequence, Prediction, and Control

287

& Malenka, 1994). In a first approximation, this model can also be applied to STDP, because when presynaptic stimulation immediately precedes a postsynaptic spike, the backpropagating postsynaptic spike can remove the Mg2+ block at the NMDA receptors (Mayer, Westbrook, & Guthrie, 1984; Nowak, Bregestovski, Ascher, Hebert, & Prochiantz, 1984), thereby causing high-level Ca2+ influx into the synapse (Magee & Johnston, 1997; Koester & Sakmann, 1998), whereas when presynaptic stimulation follows the postsynaptic spike, only low-level Ca2+ influx may occur through voltage-gated calcium channels and the partially blocked NMDA receptors. This model, however, poses a problem at large values of T. Experimentally, one still observes weak LTP, but the model would suggest that Ca2+ influx should be low for large T, rather leading to LTD. Thus, one would expect to find a second LTD window at larger T. However, such an additional LTD window was not observed in most studies of STDP (Bi & Poo, 1998; Debanne et al., 1998; Zhang et al., 1998; Feldman, 2000; Sjostr ¨ om, ¨ Turrigiano, & Nelson, 2001; Yao & Dan, 2001; Froemke & Dan, 2002) except for hippocampal slices, where spike timing of 20 ms indeed resulted in LTD (Nishiyama et al., 2000). A possible explanation for the missing secondary LTD window would be that not only the level but also the transient of Ca2+ determines if LTP or LTD will be induced. Indeed, LTP or LTD induced by postsynaptic photolysis of caged Ca2+ depends critically on not only the level but also the time course of the light-induced Ca2+ transient (Yang, Tang, & Zucker, 1999; Zucker, 1999). In contrast, when the process of intracellular Ca2+ elevation is slow, the condition for subsequent enzymatic reaction may be closer to a steady-state situation; therefore, the overall level of Ca2+ could become the only crucial parameter in determining the resultant synaptic modification. Currently, there is no direct evidence for this hypothesis, but the complexity of the postsynaptic density argues for a differential action when comparing a steady-state situation with a situation of a steep Ca2+ gradient. For example, Ca2+ influxes from different channels appear to activate preferentially different kinase signaling pathways (Deisseroth, Heist, & Tsien, 1998; Graef et al., 1999; Dolmetsch, Pajvani, Fife, Spotts, & Greenberg, 2001; West et al., 2001). Furthermore it is known that CaMKII can respond differently to Ca2+ oscillation at different frequencies (DeKoninck & Schulmann, 1998). Additional complexity comes from the dynamics of intracellular Ca2+ stores (Berridge, 1998; Svoboda & Mainen, 1999; Rose & Konnerth, 2001) mainly consisting of the lumen of the endoplasmatic reticulum, which stores Ca2+ at a high concentration. These stores can be accessed through ryanodine receptors as well as inositol 1,4,5-trisphosphate (IP3) receptors (Berridge, 1998), which are both Ca2+ sensitive. In addition, IP3 receptors are also activated by metabotropic glutamate receptors. Note, however, that Ca2+ release from intracellular stores is generally slow but long lasting and more global as compared to Ca2+ influx through synaptic channels. The role of Ca2+ stores in STDP is unclear, but it has been shown that in CA1 of the hippocampus, inhibiting store-release blocks the induction of LTD and some-

288

F. Worg ¨ otter ¨ and B. Porr

Figure 15: Theoretical results for STDP. (A) Analytically calculated final weight distributions after reaching equilibrium from the model of van Rossum et al. (2000). The unimodal curve is obtained from their original model assuming a multiplicative weight normalization mechanism. The bimodal curve is obtained when potentiation and depression do not depend on the weight (additive model with hard boundaries). (B–D) Ca2+ concentration (B,C) and weight change curve (D) obtained by Shouval et al. (2002). The insets in B and C show the timing of the backpropagating spike (“needle”) relative to the NMDA potential (elongated curve). Only for positive T (T = 10 ms, (C)) a strong rise in Ca2+ is observed, while it remains moderate for T < 0 (T = −10 ms, (B)). (D) The inset shows the shape of Shouval’s  function. They assume that LTP occurs if the Ca2+ concentration passes threshold “P”, while LTD occurs if it stays below threshold “D.” The weight change curve (D) has the characteristic shape, but an additional negative peak is found for large T, which may not correspond to experimental findings as discussed in section 6.1.

times reverses it to LTP (Futatsugi et al., 1999; Nishiyama et al., 2000). Such effects, however, may be highly sensitive to the experimental protocol and cell-type. In summary, it seems that a rapid change to a high-level Ca2+ concentration at the postsynaptic density, probably through activated NMDA receptors, is most likely responsible for inducing LTP. Slow and prolonged Ca2+ increase, on the other hand, possibly from different sources, will lead to LTD. 6.2 Models of Fast Differential Hebbian Learning. During the past few years a wide variety of different models for STDP has been designed,

Temporal Sequence, Prediction, and Control

289

which can roughly be subdivided into two groups with different biophysical complexity; spike based and rate based. 6.2.1 Network Models of STDP. The first group of models, which began to emerge around 1996 (Gerstner, Kempter, van Hemmen, & Wagner, 1996) and, thus, anticipating the experimental findings published in 1997 (Markram et al., 1997; Magee and Johnston, 1997), are relatively abstract and do not implement any biophysical mechanism in order to generate the weight change curve. Instead these models assume a certain shape of the weight change curve as the learning rule (Gerstner et al., 1996; Song, Miller, & Abbott, 2000; Rubin, Lee, & Sompolinsky, 2001; Kempter, Leibold, Wagner, & van Hemmen, 2001b; Gerstner & Kistler, 2002), which is most often an exponential fit to both parts of the data sets in Figure 14. These models look at general network properties that arise from learning. It has long been known that plain Hebbian learning without additional mechanisms will lead to the divergence of all synaptic weights in a network, because even random correlations will lead to weight growth. Thus, network stability has been one of the central issues in theories of correlationbased learning, and several add-on mechanisms have been discussed in the literature to solve this problem. In view of this problem, the property that LTP and LTD can occur at the same synapse depending on the pre-post correlations led to the attention of theoreticians. Indeed, it was found that STDP leads to self-stabilization of weights and rates in the network as soon as the LTD-side dominates over the LTP side in the weight change curve (Song et al., 2000; Kempter, Gerstner, & van Hemmen, 2001a). As a second problem of high theoretical interest, networks need to operate competitively. Thus, normally it does not make sense to drive all synapses into similar states by means of learning, because the network will then no longer contain inner structure. Instead, most real networks in the brain are highly structured, forming maps or other subdivisions. Thus, the problem of synaptic competition also needs to be solved by the applied learning mechanisms. Accordingly, more recently network STDP models have also been successfully applied to generate (i.e., to develop) some physiological properties such as map structures (Song & Abbott, 2001; Leibold, Kempter, & van Hemmen, 2001; Kempter et al., 2001b; Leibold & van Hemmen, 2002), direction selectivity (Buchs & Senn, 2002; Senn & Buchs, 2003), or temporal receptive fields (Leibold & van Hemmen, 2001). In addition, it was found that such networks can store patterns (Abbott & Blum, 1996; Seung, 1998; Abbott & Song, 1999; Fusi, 2002). In general one can subdivide the different network models of STDP into those that use an additive versus a multiplicative learning rule. Additive models assume that changes in synaptic weights do not scale with synaptic strength, and hard boundaries are imposed to prevent divergence (Abbott & Blum, 1996; Gerstner et al., 1996; Eurich, Pawelzik, Ernst, Cowan, & Milton, 1999; Kempter, Gerstner, & van Hemmen, 1999; Roberts, 1999; Song et al.,

290

F. Worg ¨ otter ¨ and B. Porr

2000; Levy, Horn, Meilijson, & Ruppin, 2001; Cateau, Kitano, & Fukai, 2002). These models exhibit strong competition but produce unstable dynamics, most often resulting in binary synaptic distributions (see Figure 15A). As an additional problem, one finds that these models often subdivide the neuronal population not according to the actual input correlations but rather as a consequence of the self-amplification of small, initially existing deviations from symmetry. This effect also results from the strong competition taking place in these networks. Multiplicative models, on the other hand, assume a linear attenuation of the weight change as soon as the upper or lower boundaries are approached (Kistler & van Hemmen, 2000; van Rossum, Bi, & Turrigiano, 2000; Rubin, Lee, & Sompolinsky, 2001). Such a mechanism seems to correspond more closely to the experimental data, because Bi and Poo (1998) have reported that the growth of synaptic weights becomes smaller if the weight is already strong. The smoothing effects of such an attenuation lead to stable network dynamics, but because of the reduced competition, all synapses are driven into a similar equilibrium value (see Figure 15A). In a recent model devised by Gutig, ¨ Aharonov, Rotter, and Sompolinsky (2003), it has been suggested to introduce a nonlinearity in the STDP rule to control the attenuation of the weight change. As a consequence, this model sits in between the hard-threshold additive approaches and the linear-attenuation multiplicative models, and the authors can show that competition and stability are obtained across a much larger parameter space. Rate-based models rely on the average firing rate of a neuron, not on its precise firing time (Sutton & Barto, 1981; Kosco, 1986; Klopf, 1986, 1988; Kempter et al., 2001b; Kistler, 2002; Porr & Worg ¨ otter, ¨ 2002, 2003a). As such they are at first sight unrelated to the spike-based models. The earlier models, which appeared before 1990, used an implicit functional description (an equation), which relied on the derivative of the signal(s) to define their learning rule. The same “differential Hebbian” relation was reintroduced by Gerstner et al. in 1996 in the context of a spike-based model and by means of a parametric description (a bimodal curve) of the learning rule. More recently, physiological results became available that pointed out that spike timing as well as the rate can influence synaptic plasticity (Sjostr ¨ om ¨ et al., 2001; Froemke & Dan, 2002). Thus, the question arises how to relate both model classes. Roberts (1999) as well as Xie and Seung (2000) have observed that it is possible to derive a rate-based model from a spiking model but the opposite cannot be achieved without additional assumptions. Gerstner and Kistler (Gerstner & Kistler, 2002; Kistler, 2002) provide a relatively complex unifying approach to these ends, using their spike-response model of firing, which, however, makes several oversimplifying assumptions with respect to the biophysics of neurons. 6.2.2 Critique of the Network STDP Models. The central advantage of network STDP models lies in the fact that they can be treated mathematically

Temporal Sequence, Prediction, and Control

291

to a large extent. Thus, theoretical statements, for example, about network stability and competition, could be obtained, albeit at the cost of biophysical realism. In network models of STDP, the learning rule (the assumed weight change curve) remains unchanged across the local properties of the cell. The complexity of the biophysical mechanisms that control the Ca2+ influence at any given synapses does not support this assumption. It seems much more likely that different shapes of weight change curves can exist, for example, along a dendrite or at spines. As a consequence, all network models of STDP are rather homogeneous in their learning behavior. Furthermore, these models do not attempt to implement different cell types, for example, inhibitory cells. This, however, may be problematic in view of the problems of stability and competition, and the question arises to what degree these problems would still surface in more realistic networks. In addition, normally these models assume that only pairwise interactions exist between pre- and postsynaptic signals (adiabatic condition). Thus, bursts of input spikes, common to most brain areas, are not considered, which could substantially influence local Ca2+ gradients. In a similar way, triplets (or multiplets) of spikes would normally create complex sequences of pre- and postsynaptic events (Froemke & Dan, 2002) are also not considered in these models. 6.2.3 Biophysical Models of STDP and Their Relation to Differential Hebb. These models are no longer concerned with the effect of STDP on network behavior; instead, they want to explain how the characteristic shape of the STDP curves is generated by means of intra-cellular mechanisms. Some use kinetic models (Senn, Markram, & Tsodyks, 2000; Castellani, Quinlan, Cooper, & Shouval, 2001). Others follow a state-variable approach trying to find a set of appropriate differential equations to describe STDP (Rao & Sejnowski, 2001; Abarbanel, Huerta, & Rabinovich, 2002; Karmarkar & Buonomano, 2002; Karmarkar, Najarian, & Buonomano, 2002; Shouval, Bear, & Cooper, 2002; Abarbanel, Gibb, Huerta, & Rabinovich, 2003). The kinetic models implement a rather high degree of biophysical detail, sometimes including calcium, transmitter, and enzyme kinetics. The power of such models lies in the chance to understand and predict intra- or subcellular mechanism, for example, the aspect of AMPA receptor phosphorilation (Castellani et al., 2001), which is known to centrally influence the synaptic strength (Malenka & Nicoll, 1999; Luscher ¨ & Frerking, 2001; Song & Huganir, 2002). Furthermore, both approaches (Senn et al., 2000; Castellani et al., 2001) can show a relation between the designed model and Hebbian as well as BCM rules (Bienenstock, Cooper, & Munro, 1982), which proposes a sliding synaptic modification threshold. (For an elaboration on the mathematical similarity between STDP and BCM see Izhikevich & Desai, 2003.)

292

F. Worg ¨ otter ¨ and B. Porr

The approaches of Rao and Sejnowski (2001), Shouval et al. (2002), and Karmarkar and coworkers (Karmarkar & Buonomano, 2002; Karmarkar, Najarian, & Buonomano, 2002) follow a state-variable approach. Rao and Sejnowski (2001) implemented a rule based on activity differences, which makes their approach related to differential Hebbian learning and less to TD learning (Dayan, 2002). The other two models investigate the effects of different calcium concentration levels by assuming certain (e.g., exponential) functional characteristics to govern its changes. This allows them to address the question of how different calcium levels will lead to LTD or LTP (Nishiyama et al., 2000) and one of the models (Karmarkar & Buonomano, 2002) proposes to employ two different coincidence detector mechanisms to this end. The model of Shouval et al. (2002) (see Figures 15B–15D) implicitly assumes a differential Hebbian characteristic by the bimodal shape of their  function (see Figure 15D, inset), which they used to capture the calcium influence. This group discussed, among other aspects, the role of the shape of the BP spike, and they concluded that a slow after-depolarization potential (more commonly known as repolarization) must exist in order to generate STDP. Thus, in their study, the shape of the BP spike will influence the shape of the weight change curve. In general, they find that the LTP part of the curve is stronger than the LTD part. This observation would prevent selfstabilization of the activity in network models (Song et al., 2000; Kempter et al., 2001a), which require a larger LTD part for achieving this effect. Interestingly, however, Shouval et al. find a second LTD part for larger positive values of T, which could perhaps be used to counteract such an activity amplification. In the hippocampus, there is currently conflicting evidence if such a second LTD part exists for large T (Pike, Meredith, Olding, & Paulsen, 1999; Nishiyama et al., 2000). ISO learning generically produces a bimodal weight change curve (see Figure 3B). Thus, at the right timescale, it should be possible to use this algorithm also in the context of STDP, provided there is a justifiable match between the components of ISO learning and the biophysical synaptic mechanisms. Figure 16 shows how to associate the components of a membrane model to those of ISO learning. We assume a plastic NMDA synapse ω1 , which receives the presynaptic activation.9 An NMDA potential gˆ N from this synapse is shown in the inset. This signal is matched to x¯ 1 = x1 ∗ h1 from ISO learning, where the asterisk denotes a convolution. The postsynaptic signal x0 arises from a BP spike. For simplicity, the BP spike is also modeled as a conductance change, integrated into the membrane potential V. Thus, the

9 The more realistic case of a mixed AMPA/NMDA synapse is discussed in the original article (Saudargiene et al., 2004).

Temporal Sequence, Prediction, and Control

A

V (mV) -30

postsyn. event x0

t

BP-spikes of different shapes

B

presyn. event x1 at plastic synapse

T

293

-40 -50 -60

Plastic Synapse

x1

g h1

w1 NMDA

gN

S

dV = g (E -V) i i dt

-70

N

0

C 0

ms

100

(gBP ) BP-Spike

x0

Dw1

50

100

150 ms

Weight-change-curves

0.06 0.04 0.02 0 -0.02 -0.04 -0.06 -0.08 -80

-40

0

40 T (ms)

Figure 16: Applying the ISO learning algorithm to STDP. (A) Schematic diagram of the model. The inset shows a comparison between an NMDA potential gˆ N and a resonator response h1 from the original ISO learning algorithm. (B) BP spikes of different shapes. (C) Weight change curves obtained when using the BP spikes from B as the postsynaptic signals.

ISO learning rule (see equation 3.9) turns into dω1 = µx¯ 1 v = µ gˆ N V  , dt

(6.1)

where V  is the derivative of the membrane potential. It is known that the shape of a BP spike changes with the parameters of the dendrite, getting flatter and more spread out at distal dendrites. In particular, at distal dendrites, BP spike may not play a large role anymore (Stuart et al., 1997; Golding et al., 2001) and local signals, such as dendritic spikes will provide the depolarizing signal (Schiller, Schiller, Stuart, & Sakmann, 1997; H¨ausser, Spruston, & Stuart, 2000; Golding, Staff, & Spruston, 2002; Saudargiene et al., 2004; Mehta, 2004). The ISO model captures these effects, because it can model arbitrary depolarizing signals. Figure 16B demonstrates this aspect, discussing the special case of an attenuated BP spike. Figure 16C shows that the weight change curves obtained with these different BP spikes are significantly different in shape. This emphasizes that fact that different STDP characteristics are to be expected at different locations on the dendrite or at spines and this would lead to different computational properties. The differential Hebbian rule we employed leads to the observed results as the consequence of the fact that the derivative of any generic (unimodal) postsynaptic membrane signal (like a BP-spike) will lead to a bimodal curve.

294

F. Worg ¨ otter ¨ and B. Porr

The relative temporal location of the presynaptic depolarization signal with respect to the positive (or negative) hump of this bimodal curve will then determine if the product in the learning rule is positive (weight growth) or negative (weight shrinkage). This can also lead to the fact that differential Hebbian learning turns into plain Hebbian learning without having to change the learning rule if the rise time of the BP spike is shallow (see Figure 5 in Saudargiene et al., 2004). The ISO model of STDP is a spike-based model derived from a rate-based model (ISO learning). Central to the transfer between both domains was the realization that at any synapse, pulse coding turns into analog (rate) coding as the consequence of the filtering properties of the membrane, especially the transmitter-receptor interactions. This shows that at the level of a synapse, the distinction between spike-based STDP models and rate-based STDP models may not be of great importance anymore. 6.2.4 Critique of the Biophysical STDP Models. As opposed to network models, the biophysical ones in principle have the potential to explain the subcellular mechanisms that underlie STDP in more detail. However, at the moment, the level of biophysical realism is still not very high in these models. The model of Senn et al. (2000) is restricted to the kinetics of transmitter release without implementing an explicit Ca2+ mechanism. The others implicitly or explicitly assume that low levels of Ca2+ will lead to LTD, while high levels lead to LTP. The aspect that the Ca2+ gradient also seems to be important is not addressed in most models, and more complex reaction chains are also not yet implemented in the context of STDP.10 The state-variable models (Shouval et al., 2002; Karmarkar & Buonomano, 2002; Karmarkar et al., 2002) were designed to produce a zero crossing (transition between LTD and LTP) at T = 0, which is not always the case in the measured weight change curves, which show transitions between more LTP- and more LTD-dominated shapes depending on the cell type and the stimulation protocol (Roberts & Bell, 2002). An important advantage over network models, however, is that biophysical models permit different STDP characteristics at different synaptic sites at the same cell. The weight change curve depends in all of these models on subcellular parameters and on the shape of the potential changes at the synapses (Rao & Sejnowski, 2001; Saudargiene et al., 2004). The question of differently shaped weight change curves is of relevance for aspects of dendritic synaptic development and, consequentially, also for dendritic computations. Therefore, some groups have started to address this problem by explicitly parameterizing weight change curves along dendritic structures (Panchev, Wermter, & Chen, 2002; Sterratt & van Ooyen, 2002).

10 The model of Castellani et al. (2001) focuses on bidirectional synaptic plasticity but does not directly relate this to STDP.

Temporal Sequence, Prediction, and Control

295

6.3 The Time-Gap Problem. In the preceding sections, we have mainly discussed how spike-timing-dependent plasticity can be modeled with correlation-based differential Hebbian learning rules. We started this discussion with a more general account on differential Hebbian learning, first developed by Sutton and Barto (1981) as well as Klopf (1986, 1988) and Kosco (1986). These rules, recently augmented by ISO learning (Porr & Worg ¨ otter, ¨ 2002, 2003a), have been used with rather long temporal intervals between CS and US (seconds). The same rules, however, can almost without modification also be used to account for STDP, albeit on a much faster timescale (milliseconds). This is currently probably one of the biggest computational problems of this whole field: How would it be possible to bridge the gap between the timescales of correlation-based synaptic plasticity and those of behavioral learning? It is by now widely accepted that LTP and LTD are strongly involved in processes of learning and memory (Martin, Grimwood, & Morris, 2000). At the moment, there is no straightforward way that would lead from an STDP model to, for example, a model of classical conditioning, and only a few attempts exist in this direction (Mehta, Lee, & Wilson, 2002; Sato & Yamaguchi, 2003; Melamed, Gerstner, Maass, Tsodyks, & Markram, 2004). Maybe the serial compound stimulus representations, introduced in section 3.1.3, could be “recycled” in this context, but such a representation seems to be a rather crude approximation of the neuronal interactions in the areas involved. Reverberating synfire chains (Abeles, 1991) would be another possible mechanisms to bridge the time gap (Kitano, Okamato, & Fukai, 2003), but this concept is also quite heavily debated. Thus, the relationship between fast and slow differential Hebb rules may turn out to be a mere structural one. 7 Discussion: Are Reward-Based and Correlation-Based Learning Indeed Similar? Part of the description above has shown that neuronal TD models are related to correlation-based approaches, when treating the reward input not differently from any of the other inputs (see Figure 4C). Thus, at least in their open-loop condition, reward-based learning can be equated to correlationbased rules, and they appear similar. The difference between those two approaches, however, is more subtle and surfaces only when considering closed-loop situations. Reward-based learning requires evaluative feedback from the environment, where someone (an external observer) or some process (the “critic”) defines what is rewarding and what is punishing for the learner. Correlation-based learning does not need evaluative feedback. All feedback signals from the environment that reach the learner are totally value free, and only the temporal correlations between the different input signals drive the learning. This difference has been known since the early 1980s (Klopf, 1982, 1988), and reinforcement learning had been introduced in order to address some of

296

F. Worg ¨ otter ¨ and B. Porr

the problems that exist with correlation-based methods. First and foremost, it had been observed that it is hard to define the boundary conditions for correlation-based methods. Wrong boundary conditions will lead to wrong convergence of learning. In addition, it was deemed impossible to learn conflict solving by means of correlation-based methods (McFarland, 1989). Reinforcement learning seems to be better suited to address these problems, and hopes were high that RL techniques could be used to address complex learning tasks. Alas, many robotics researchers can now confirm that RL will not solve their learning problems, and the question arises: Why is this the case? 7.1 Evaluative Versus Nonevaluative Feedback: The Credit Structuring Problem. Evaluative feedback, however, is not unproblematic. Since RL methods rely on rewards, we must find a way to define them. So far we have somewhat naively assumed that we know how much reward can be associated with a given state or situation. This, however, requires a process that places the rewards appropriately (credit structuring problem). 7.1.1 External Evaluative Feedback. Rewards would have to be placed such that learning is efficient; it is not good enough to just reward some final learning outcome (which is normally much easier). For example, the goal “learn to win in chess,” hence placing a reward at the end of a successful game, will lead to unacceptably long times to convergence (if at all) of learning.11 Faster convergence can be achieved by associating appropriate rewards with ideally many intermediate configurations of the board.12 This, however requires an in-depth analysis of the structure of the problem before credit structuring can take place. This analysis may at the end be almost as complex as the learning process itself. Thus, credit structuring is a major challenge, especially in time- and space-continuous control tasks, to which all animal behavioral tasks belong. In these cases, there are an infinite number of state-action pairs, to which rewards or punishments could be associated beforehand by defining a mapping function from the state-action space to the reward-punishment space with a so-called reinforcement function (Santos & Touzet, 1999a, 1999b). Such procedures for credit structuring can be called external evaluative feedback: an external structure, a “teacher,” explicitly provides the rewards. Obviously this situation does not normally apply to animals. 11 TD-Gammon (Tesauro, 1992) is commonly cited as one the major successes of TD learning. One should, however, keep in mind that it took millions of iterations of self-play to converge. Furthermore, it has been discussed that the structure of backgammon may be especially well suited for TD learning (Pollack & Blair, 1997). 12 In general it is observed that RL methods often converge very slowly on fine-grain or time-space continuous problems. Therefore, efforts have been made up to accelerate convergence (e.g., Wiering & Schmidhuber, 1998; see Reynolds, 2002 for a discussion).

Temporal Sequence, Prediction, and Control

297

7.1.2 Internal Evaluative Feedback. A better strategy may be to assume that the agent/algorithm performs credit structuring on its own. This can be achieved, for example, by associating only very general aversive or attractive properties (like good or bad taste of food, pain or pleasure) to states and letting the agent “experience” them via some sensor inputs. We would call such a procedure internal evaluative feedback: The agent itself provides the value of the rewards. The agent consists of a critic who criticizes the actions of the actor, such as discussed in section 4.1. There is, however, a hitch. How does the critic know how to criticize? At least for a constructed artificial agent, someone (the designer) must have provided a frame to allow the critic to do its job. Thus, we still need an external structure (the designer) that explicitly defines the reference frame, the reinforcement function, for what should be rewarding (or punishing). In simple situations, this seems easy—for example, food is rewarding, pain not. However, as soon as one would like the agent to learn a complex control task with multiple and often conflicting demands (think of robot soccer), setting a “good” frame set to drive the learning becomes nontrivial. We believe this problem is related to the famous frame problem of Dennett (1984), initially pointed out by Klopf (1988) in the context of classical conditioning. In brief, Dennett claims that is is impossible to define a complete frame set (a world model) to guide the behavior of an autonomous agent or an AI system, because the world model of the designer will never match the world as experienced (as “sensed”) by the agent. Dennett’s arguments concerned behavioral control. However, in trying to control and guide learning (by means of a critic), it may be equally impossible to define such a frame set for the reward structure as soon as the task of the critic is complex enough. We would like to call this the secondorder frame problem and would argue that the strange lack of success of RL methods in complex state action spaces may be partially due to this problem. 7.1.3 Nonevaluative Feedback. One possible solution out of this secondorder frame problem is to try to design a system that operates strictly with nonevaluative feedback where any external structure (here, the environment) will provide only value-free signals. A possible starting point for such a system would be some kind of very generally prespecified homeostasis, and the only goal of the agent would be not to maximize rewards but to minimize disturbances of this homeostasis. This way, external interference by the designer of the agent would be absolutely minimal, because even the initial state could be generated by evolutionary methods and the agent will in all instances “decide by itself” (i.e., by internal evaluation within its own reference frame) if something is rewarding or punishing. In principle, it would in this case not even be necessary to make this evaluation explicit in terms of a dedicated reward or punishment signal. Without such signals, learning would have to rely exclusively on correlative input properties, and traditional TD architectures cannot be used. Only with secondarily derived

298

F. Worg ¨ otter ¨ and B. Porr

inner reward-related signals, which seem to be represented by the dopamine responses of neurons in the basal ganglia, does TD learning become again feasible. 7.2 Bootstrapping Reward Systems? Compelling evidence exists that the dopaminergic system in the brain is related to reward processing. Thus, the above arguments by which we criticized essentially all evaluative feedback mechanisms must be mitigated to some degree. Animals seem indeed to use evaluative feedback mechanisms successfully for learning. Here, however, the situation is to some degree different. Such reward systems have presumably been bootstrapped by evolution, which has built at least the most basic reward and punishment structures into the “critic,” such that even newborn animals can rely on them. Evolutionary mechanisms of variation and selection are indeed well suited to develop value systems. Futhermore, it is conceivable that more complex internal evaluative structures can be developed on top of these early reward mechanisms. This could be done by either correlation-based learning or by learning secondary (derived) value systems from primary ones. Several interesting questions arise here. For example, how finely can such reward systems be structured in animals? Is it conceivable that several “critics” operate in parallel in an animal in order to disentangle conflicting situations? Is is possible to bootstrap reward-based systems by means of value-free correlation-based learning? Thus, in general, from this discussion, the question arises: How does the learner (the critic) learn to criticize? 7.3 The Credit Assignment Problem. In a simple summary, one could say that the above discussion has dealt with the “spatial” structure of the world: How and where to place rewards before learning will start? The temporal credit assignment problem, on the other hand, refers to the fact that rewards, especially in fine grained state-action spaces, can be terribly temporally delayed. For example, a robot will normally perform many moves through its state-action space where immediate punishments or rewards are (almost) zero and where more relevant events are rather distant in the future. As a consequence, such reward signals will only very weakly affect all temporally distant states that have preceded it. It is almost as if the influence of a reward gets more and more diluted over time, and this can lead to bad convergence properties of the RL mechanism. Many steps must be performed by any iterative RL algorithm to propagate the influence of delayed reinforcement to all states and actions that have an effect on that reinforcement (Anderson & Crawford-Hines, 1994). In addition, we observe a somewhat paradoxical situation for credit assignment that arises during learning: If an agent successfully avoids a punishment, then it will receive no punishment signal. A “no-signal” situation, however, normally occurs also for every “boring” state-action pair where the machine is far away from a source of reward or punishment. Thus, “no-signal” (or to be more accurate

Temporal Sequence, Prediction, and Control

299

“faint-signal”) states prevail throughout most of an agent’s lifetime and it is almost impossible to tell if such a state has come about through successful punishment avoidance or, more likely, just by chance. 7.4 Maximizing Rewards or Minimizing Disturbances? Speculations About Learning. Return maximization is the central paradigm of reinforcement learning, and this gives rise to the problems of credit structuring and credit assignment. Most actor-critic architectures have also been implemented adhering to this paradigm. The architecture in Figure 9 suggests an alternative where actor and critic are merged. It starts with a negative feedback loop (see Figure 7A), which creates a stable starting condition: whenever a deviation from the desired state (set point) occurs, the feedback loop will try to correct it. This property is then also used to guide the learning by means of a minimization instead of a maximization principle: the learning goal is to try to minimize deviations from the desired state, that is, to minimize disturbances of the homeostasis of the feedback loop. These two paradigms are in most cases not equivalent. Most often, one finds that maximal return is associated with a single (or a few) point(s) on the decision surface. Minimal disturbance, however, will cover a dense manifold of points, all of which represent solutions of the learning problem. Thus, correlation-based temporal sequence learning (see Figure 9) offers the advantage that credit structuring takes place without effort: every signal that enters the reflex loop will drive the learning, and if there is no signal, the situation is stable (desirable) for at least the moment. Rewards do not exist in this scheme. Credit assignment during learning also ceases to be a problem. Since we do not attempt to seek maximal return and are instead satisfied with any disturbance-free solution, we can live without credits almost everywhere on the decision surface and let the agent choose its own path until a (rare) disturbance happens. Early observations have pointed to the fact that pure homeostatic mechanisms will not develop “interesting” or even intelligent behavior (Ashby, 1952). This certainly holds for pure feedback loop control as in Figure 7A. However, as shown above, the stereotyped reflex behavior of such a homeostat can be augmented by anticipatory correlation-based learning adding a second loop (see Figure 10). As long as there are anticipatory signals, we can continue to pile loops on top of loops, creating a complex subsumption architecture (Porr & Worg ¨ otter, ¨ 2003c). However, there exist only a rather limited number of causally related anticipatory sensor events that come from the environment. A chain of three such signals could be given: sound may precede smell of the prey, which in turn precedes taste. As a consequence, the depth of a subsumption architecture, which relies exclusively on sensor events, will remain rather limited. However, in principle, nothing prevents us from using intrinsic signals (“thoughts”) to the same purpose. It does not matter to the learner where an anticipatory signal came from, such that this type of correlation-based learning should also work with internal

300

F. Worg ¨ otter ¨ and B. Porr

signals. Naively, if I think this and a certain event almost always follows, then this “thought should be strengthened.” This is certainly highly speculative, but we would like to add that through such a process of anticipatory correlation-based learning, “values” can actually be developed: anticipatory learning carries the implicit semantics that “earlier is better.” This could be a natural starting point to develop more complex value (hence “reward”) structures without having to assume external interference. 8 Conclusion This article provides an overview across the field of temporal sequence learning. Two problems were mainly discussed: how to match neuronal models to physiology and how to design biologically plausible control methods, possibly using such algorithms. We believe that there are three main questions that need to be addressed in the future: 1. How can we bridge the temporal gap between STDP (or LTP, LTD) and behavioral learning? 2. When do animals (humans) follow the return-maximization principle? Do they sometimes (often?) adopt different strategies like disturbance minimization? And how do they decide when to use the different strategies? 3. How can we implement (evolve, develop) nonevaluative feedback or strictly internal evaluative feedback, without adopting the perspective of an external observer, to arrive at strictly agent-driven prediction and control? Without answering the first question, it will be impossible to explain animal learning by means of biophysical network models of brain function. Without addressing the second set of questions, control agents may be too limited and inflexible in their behavioral and decision-making potential. Without addressing the last question, we will always design only control agents that follow our own intentions (or at least our own frame of reference). Our reference frame depends on our own internal states; thus, it can never accurately match that of any other agent which receives different inputs and performs different internal processing. As a consequence, such an agent is unavoidably doomed to fail when left to its own (really our!) devices in its own (really its!) world. It will fail to be autonomous. Appendix This appendix provides the background on the use of eligibility traces in the TD formalism of reinforcement learning. This way it becomes clear that the concept of eligibility traces has penetrated different algorithms and that the backward TD(λ) algorithms contain a term that can be used to define a

Temporal Sequence, Prediction, and Control

301

correlative process. This way, backward TD(λ) can, if desired, be related to Hebbian learning. A.1 The TD Formalism for the Prediction Problem in Reinforcement Learning. We consider a sequence st , rt+1 , st+1 , rt+2 , . . . , rT , sT . Note that rewards occur downstream (in the future) from a visited state.13 Thus, rt+1 is the next future reward that can be reached starting from state st . For simplicity, actions are not denoted here. The complete return Rt , to be expected in the future from state st , is thus given by Rt = rt+1 + γ rt+2 + γ 2 rt+3 + . . . + γ T−t−1 rT ,

(A.1)

where γ ≤ 1 is the discount factor. Reinforcement learning assumes that the value of a state V(s) is directly equivalent to the expected return E at this state, where π denotes the (here unspecified) action policy to be followed: V(s) = Eπ {Rt |st = s}.

(A.2)

Thus, the value of state st can be iteratively updated with V(st ) ← V(st ) + α[Rt − V(st )].

(A.3)

The left side of this equation represents the new value of st calculated by using the complete return Rt and holding it against the old value of st (right side). We use α as a step-size parameter, which is not of great importance here, though, and can be held constant. Note that if V(st ) correctly predicts the expected complete return Rt , the update will be zero, and we have found the final value. This method is called constant-α Monte Carlo update. It requires waiting until a sequence has reached its terminal state before the update can commence. For long sequences, this may be problematic. Thus, one should try to use an incremental procedure instead. We define a different update rule with V(st ) ← V(st ) + α[rt+1 + γ V(st+1 ) − V(st )].

(A.4)

The elegant trick is to assume that if the process converges, the value of the next state V(st+1 ) should be an accurate estimate of the expected return downstream to this state (i.e., downstream to st+1 ). Thus, we would hope that Rt = rt+1 + γ V(st+1 )

(A.5)

13 We are using a slightly simplified version of the notation conventions from Sutton and Barto (1998). This book should be consulted in case of detailed questions.

302

F. Worg ¨ otter ¨ and B. Porr

holds. Indeed, proofs exist (Sutton, 1988) that under certain boundary conditions, this procedure, known as TD(0), converges to the optimal value function for all states. In principle, the same procedure can be applied all the way downstream, writing, 2 n−1 R(n) rt+n + γ n V(st+n ). t = rt+1 + γ rt+2 + γ rt+3 + · · · + γ

(A.6)

Thus, we could update the value of state st by moving downstream to some future state st+n−1 , accumulating all rewards along the way, including the last future reward rt+n , and then approximating the missing bit until the terminal state by the estimated value of state st+n given as V(st+n ). Furthermore, we can even take different such update rules and average their results in the following way: Rλt = (1 − λ)

∞ 

λn−1 R(n) t

(A.7)

n=1

V(st ) ← V(st ) + α[Rλt − V(st )],

(A.8)

where 0 ≤ λ ≤ 1. This is the most general formalism for a TD rule known as forward TD(λ)-algorithm (Sutton, 1988), where we assume an infinitely long sequence. Convergence proofs are given by Dayan (1992), Peng (1993), and Dayan and Sejnowski (1994). The disadvantage of this formalism is still that for all λ > 0, we have to wait until we have reached the terminal state until update of the value of state st can commence. There is a way to overcome this problem by introducing so-called eligibility traces re-introduced in the context of RL by Watkins (1989) and Jaakkola, Jordan, and Singh (1995). Let us assume that we came from state A, and now we are currently visiting state B of an MDP. B’s value can be updated by the TD(0) rule after we have moved on by only a single step to, say, state C. We define the incremental update as before as δt = rt+1 + γ V(st+1 ) − V(st ).

(A.9)

Normally we would assign a new value to state B by performing V(sB ) ← V(sB ) + αδB , not considering any other previously visited states. In using eligibility traces, we do something different and assign new values to all previously visited states, making sure that changes at states long in the past are much smaller than those at states visited just recently. To this end, we define the eligibility trace of a state as  xt (s) =

γ λxt−1 (s) γ λxt−1 (s) + 1

if if

s = st . s = st

(A.10)

Temporal Sequence, Prediction, and Control

303

Thus, the eligibility trace of the currently visited state is incremented by one, while the eligibility traces of all states decay with a factor of γ λ. Instead of updating just the most recently left state st we will now loop through all states visited in the past of this trial, which still have an eligibility trace larger than zero and update them according to V(st ) ← V(st ) + αδt xt (s).

(A.11)

In our example, we will thus also update the value of state A by V(sA ) ← V(sA ) + α δB xB (A). This means we are using the TD error δB from the state transition B → C (see equation A.9), weight it with the currently existing numerical value of the eligibility trace of state A given by xB (A), and use this to correct the value of state A “a little bit.” This procedure always requires only a single newly computed TD error using the computationally very cheap TD(0)-rule, and all updates can be performed online when moving through the state-space without having to wait for the terminal state. The whole procedure is known as backward TD(λ)-algorithm, and it can be shown that it is mathematically equivalent to forward TD(λ) described above (Sutton & Barto, 1998). All these algorithms have been designed for time- and spacediscrete problems (MDPs). A continuous approach has been described by Doya (2000). Acknowledgments This work was supported by the SHEFC INCITE and the EU ECOVISON grants. We thank P. Dayan, A. Saudargiene, and L. Smith for helpful discussions. References Abarbanel, H. D. I., Gibb, L., Huerta, R., & Rabinovich, M. I. (2003). Biophysical model of synaptic plasticity dynamics. Biol. Cybern., 89(3), 214–226. Abarbanel, H. D. I., Huerta, R., & Rabinovich, M. I. (2002). Dynamical model of long-term synaptic plasticity. Proc. Natl. Acad. Sci. (USA), 99(15), 10132–10137. Abbott, L. F., & Blum, K. I. (1996). Functional significance of long-term potentiation for sequence learning and prediction. Cereb. Cortex, 6, 406–416. Abbott, L. S., & Song, S. (1999). Temporal asymmetric Hebbian learning, spike timing and neuronal response variability. In M. S. Kearns, S. Solla, & D. A. Cohn, (Eds.), Advances in neural information processing systems, (pp. 69–75). Cambridge, MA: MIT Press. Abeles, M. (1991). Corticotronics: Neural circuits of the cerebral cortex. Cambridge. Cambridge University Press. Akopian, G., Musleh, W., Smith, R., & Walsh, J. P. (2000). Functional state of corticostriatal synapses determines their expression of short- and long-term plasticity. Synapse, 38, 271–280.

304

F. Worg ¨ otter ¨ and B. Porr

Altar, C. A., Boyar, W. C., & Kim, H. S. (1990). Discriminatory roles for D1 and D2 dopamine receptor subtypes in the in vivo control of neostriatal cyclic GMP. Eur. J. Pharmacol., 181, 17–21. Anderson, C., & Crawford-Hines, S. (1994). Multigrid q-learning (Tech. Rep. No. CS-94-121), Fort Collins: Colorado State University. Artola, A., & Singer, W. (1993). Long-term depression of excitatory synaptic transmission and its relationship to long-term potentation. Trends Neurosci., 16, 480–487. Ashby, W. R. (1952). Design for a brain. London: Chapman & Hall. Balkenius, C., & Moren, J. (1998). Computational models of classical conditioning: A comparative study. (LUCS 62 ISSN 1101-8453). Lund, Sweden Lund University Cognitive Studies. Barto, A. (1995). Adaptive critics and the basal ganglia. In J. C. Houk, J. L. Davis, & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 215–232). Cambridge, MA: MIT Press. Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13, 835–846. Bell, C. C., Han, V. Z., Sugawara, Y., & Grankt, K. (1997). Synaptic plasticity in a cerebellum-like structure depends on temporal order. Nature, 387, 278–281. Bellman, R. E. (1957). Dynamic programming. Princeton, NJ: Princeton University Press. Benett, M. R. (2000). The concept of long term potentiation of transmission at synapses. Prog. Neurobiol., 60, 109–137. Berger, B., Trottier, S., Verney, C., Gaspar, P., & Alvarez, C. (1988). Regional and laminar distribution of the dopamine and serotonin innervation in the macaque cerebral cortex: A radioautographic study. J. Comput. Neurol., 273, 99–119. Berns, G. S., McClure, S. M., Pagoni, G., & Montague, P. R. (2001). Predictability modulates human brain responses to reward. J. Neurosci., 21, 2793–2798. Berns, G. S., & Sejnowski, T. J. (1998). A computational model of how the basal ganglia produce sequences. J. Cogn. Neurosci., 10 (1), 108–121. Berridge, K. C., & Robinson, T. E. (1998). What is the role of dopamine in reward: Hedonic impact, reward learning or incentive salience? Brain Res. Rev., 28, 309–369. Berridge, M. J. (1998). Neuronal calcium signaling. Neuron, 21, 13–26. Bi, G. Q. (2002). Spatiotemporal specificity of synaptic plasticity: Cellular rules and mechanisms. Biol. Cybern., 87, 319–332. Bi, G. Q., & Poo, M. M. (1998). Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci., 18, 10464–10472. Bi, G.-Q., & Poo, M. (2001). Synaptic modification by correlated activity: Hebb’s postulate revisited. Annu. Rev. Neurosci., 24, 139–166. Bibb, J. A., Snyder, G. L., Nishi, A., Yan, Z., Meijer, L., Fienberg, A. A., Tsai, L. H., Kwon, Y. T., Girault, J. A., Czernik, A. J., Huganir, R. L., Hemmings, H. C., Nairn, A. C., and Greengard, P. (1999). Phosphorylation of

Temporal Sequence, Prediction, and Control

305

DARPP-32 by Cdk5 modulates dopamine signalling in neurons. Nature, 402, 669–671. Bienenstock, E., Cooper, L., & Munro, P. (1982). Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. J. Neurosci., 2(1), 32–48. Bliss, T. V., & Gardner-Edwin, A. R. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the unanaesthetized rabbit following stimulation of the perforant path. J. Physiol. (Lond.), 232, 357–374. Bliss, T. V., & Lomo, T. (1970). Plasticity in a monosynaptic cortical pathway. J. Physiol. (Lond.), 207, 61P. Bliss, T. V., & Lomo, T. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. J. Physiol. (Lond.), 232, 331–356. Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., & Shizgal, P. (2001). Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron, 30, 619–639. Brown, J., Bullock, D., & Grossberg, S. (1999). How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J. Neurosci., 19(23), 10502–10511. Brown, J. R., & Arbuthnott, G. W. (1983). The electrophysiology of dopamine D2 receptors: A study of the actions of dopamine on corticostriatal transmission. Neurosci., 10, 349–355. Buchs, N. J., & Senn, W. (2002). Spike-based synaptic plasticity and the emergence of direction selective simple cells: Simultation results. J. Comput. Neurosci., 13, 167–186. Bunney, B. S., Chiodo, L. A., & Grace, A. A. (1991). Midbrain dopamine system electrophysiological functioning: A review and new hypothesis. Synapse, 9, 79–94. Buzsaki, G., Penttonen, M., Nadasdy, Z., & Bragin, A. (1996). Pattern and inhibition-dependent invasion of pyramidal cell dendrites by fast spikes in the hippocampus in vivo. Proc. Natl. Acad. Sci. USA, 93, 9921–9925. Calabresi, P., Centonze, D., Gubellini, P., Marfia, G. A., & Bernardi, G. (1999). Glutamate-triggered events inducing corticostriatal long-term depression. J. Neurosci., 19, 6102–6110. Calabresi, P., Fedele, E., Pisani, A., Fontana, G., Mercuri, N. B., Bernardi, G., & Raiteri, M. (1995). Transmitter release associated with long-term synaptic depression in rat corticostriatal slices. European J. Neurosci., 7, 1889–1894. Calabresi, P., Gubellini, P., Centonze, D., Picconi, B., Bernardi, G., Chergui, K., Svenningsson, P., Fienberg, A. A., & Greengard, P. (2000). Dopamine and cAMP-regulated phosphoprotein 32 kDa controls both striatal long-term depression and long-term potentiation opposing forms of synaptic plasticity. J. Neurosci., 20(22), 8443–8451. Calabresi, P., Maj, R., Mercuri, N. B., & Bernardi, G. (1992a). Coactivation of D1 and D2 dopamine receptors is required for long-term synaptic depression in the striatum. Neurosci. Letters, 142, 95–99.

306

F. Worg ¨ otter ¨ and B. Porr

Calabresi, P., Maj, R., Pisani, A., Mercuri, N. B., & Bernardi, G. (1992b). Longterm synaptic depression in the striatum: Physiological and pharmacological characterization. J. Neurosci., 12, 4224–4233. Calabresi, P., Mercuri, N., Stanzione, P., Stefani, A., & Bernardi, G. (1987). Intracellular studies on the dopamine-induced firing inhibition of neostriatal neurons in vitro: Evidence for D1 receptor involvement. Neurosci., 20, 757– 771. Calabresi, P., Pisani, A., Mercuri, N. B., & Bernardi, G. (1992c). Long-term potentiation in striatum is unmasked by removing the voltage-dependent magnesium block of NMDA receptor channels. European J. Neurosci., 4, 929–935. Calabresi, P., Pisani, A., Mercuri, N. B., & Bernardi, G. (1996). The corticostriatal projection: From synaptic plasticity to dysfunctions of the basal ganglia. Trends Neurosci., 19, 19–24. Cannon, C. M., & Palmiter, R. D. (2003). Reward without dopamine. J. Neurosci, 23(34), 10827–10831. Carpenter, G. A., & Grossberg, S. (1987). ART-2: Self-organization of stable category recognition codes for analog input pattern. Applied Optics, 26, 4919– 4930. Castellani, G. C., Quinlan, E. M., Cooper, L. N., & Shouval, H. Z. (2001). A biophysical model of bidirectional synaptic plasticity: Dependence on AMPA and NMDA receptors. Proc. Natl. Acad. Sci. (USA), 98(22), 12772–12777. Cateau, H., Kitano, K., & Fukai, T. (2002). An accurate and widely applicable method to determine the distribution of synaptic strengths formed by the spike-timing-dependent learning. Neurocomputing, 44(46), 343–351. Centonze, D., Gubellini, P., Picconi, B., Calabresi, P., Giacomini, P., & Bernardi, G. (1999). Unilateral dopamine denervation blocks corticostriatal LTP. J. Neurophysiol., 82, 3575–3579. Centonze, D., Picconi, B., Gubellini, P., Bernardi, G., & Calabresi, P. (2001). Dopaminergic control of synaptic plasticity in the dorsal striatum. European J. Neurosci., 13, 1071–1077. Charpier, S., & Deniau, J. M. (1997). In vivo activity-dependent plasticity at cortico-striatal connections: Evidence for physiological long-term potentiation. Proc. Natl. Acad. Sci. USA, 94, 7036–7040. Charpier, S., Mahon, S., & Deniau, J. M. (1999). In vivo induction of striatal long-term potentiation by low-frequency stimulation of the cerebral cortex. Neurosci., 91, 1209–1222. Choi, S., & Lovinger, D. M. (1997). Decreased probability of neuro-transmitter release underlies long-term depression and postnatal development of corticostriatal synapses. Proc. Natl. Acad. Sci. USA, 94, 2665–2670. Contreras-Vidal, J. L., & Schultz, W. (1999). A predictive reinforcement model of dopamine neurons for learning approach behavior. J. Comput. Neurosci., 6, 191–214. Daw, N. D. (2003). Reinforcement learning models of the dopamine system and their behavioral implications. Unpublished doctoral dissertation, Carnegie Mellon University, Pittsburgh, PA. Dayan, P. (1992). The convergence of TD(λ). Mach. Learn., 8(3/4), 341–362. Dayan, P. (2002). Matters temporal. Trends Cogn. Sci., 6(3), 105–106.

Temporal Sequence, Prediction, and Control

307

Dayan, P., & Balleine, B. W. (2002). Reward, motivation, and reinforcement learning. Neuron, 36(2), 285–298. Dayan, P., Kakade, S., & Montague, P. R. (2000). Learning and selective attention. Nature Neurosci., 3, 1218–1223. Dayan, P., & Seynowski, T. (1994). TD(λ) converges with probability 1. Mach. Learn., 14, 295–301. Debanne, D., Gahwiler, B. T., & Thompson, S. H. (1994). Asynchronous pre- and postsynaptic activity induces associative long-term depression in area CAI of the rat hippocampus in vitro. Proc. Natl. Acad. Sci. (USA), 91, 1148–1152. Debanne, D., Gahwiler, B., & Thompson, S. (1998). Long-term synaptic plasticity between pairs of individual CA3 pyramidal cells in rat hippocampal slice cultures. J. Physiol. (Lond.), 507, 237–247. Deisseroth, K., Heist, E. K., & Tsien, R. W. (1998). Translocation of calmodulin to the nucleus supports CREB phosphorylation in hippocampal neurons. Nature, 392, 198–202. DeKoninck, P., & Schulmann, H. (1998). Sensitivity of CaM kinase II to the frequency of Ca2+ oscillations. Science, 279, 227–230. Delgado, M. R., Nystrom, L. E., Fissell, C., Noll, D. C., & Fiez, J. A. (2000). Tracking the hemodynamic responses to reward and punishment in the striatum. J. Neurophysiol., 84, 3072–3077. Dennett, D. C. (1984). Cognitive wheels: The frame problem of AI. In C. Hookway (Eds.), Minds, machines and evolution (pp. 129–151). Cambridge: Cambridge University Press. Dolmetsch, R. E., Pajvani, U., Fife, K., Spotts, J. M., & Greenberg, M. E. (2001). Signaling to the nucleus by an L-type calcium channel-calmodulin complex through the MAP kinase pathway. Science, 294, 333–339. Doya, K. (2000). Reinforcement learning in continuous time and space. Neural Comp., 12(1), 219–245. Egger, V., Feldmeyer, D., & Sakmann, B. (1999). Coincidence detection and changes of synaptic efficacy in spiny stellate neurons in rat barrel cortex. Nature Neurosci., 2, 1098–1105. Elliott, R., Friston, K. J., & Dolan, R. J. (2000). Dissociable neural responses in human reward systems. J. Neurosci., 20, 6159–6165. Eurich, C. W., Pawelzik, K., Ernst, U., Cowan, J. D., & Milton, J. G. (1999). Dynamics of self-organized delay adaptation. Phys. Rev. Lett., 82, 1594–1597. Feldman, D. (2000). Timing-based LTP and LTD at vertical inputs to layer II/III pyramidal cells in rat barrel cortex. Neuron, 27, 45–56. Froemke, R. C., & Dan, Y. (2002). Spike-timing-dependent synaptic modification induced by natural spike trains. Nature, 416, 433–438. Fusi, S. (2002). Hebbian spike-driven synaptic plasticity for learning patterns of mean firing rates. Biol. Cybern., 87, 459–470. Futatsugi, A., Kato, K., Ogura, H., Li, S. T., Nagata, E., Kuwajima, G., Tanaka, K., Itohara, S., & Mikoshiba, K. (1999). Facilitation of NMDAR-independent LTP and spatial learning in mutant mice lacking ryanodine receptor type 3. Neuron, 24, 701–713. Gerfen, C. R. (1984). The neostriatal mosaic: Compartmentalization of corticostriatal input and striatonigral output systems. Nature, 311, 461–464.

308

F. Worg ¨ otter ¨ and B. Porr

Gerfen, C. R. (1985). The neostriatal mosaic. I. Compartmental organization of projections from the striatum to the substantia nigra in the rat. J. Comp. Neurol., 236, 454–476. Gerfen, C. R. (1992). The neostriatal mosaic: Multiple levels of compartmental organization in the basal ganglia. Annu. Rev. Neurosci., 15, 285–320. Gerfen, C. R., Herkenham, M., & Thibault, J. (1987). The neostriatal mosaic: II. Patch- and matrix- directed mesostriatal dopaminergic and nondopaminergic systems. J. Neurosci., 7, 3915–3934. Gerstner, W., Kempter, R., van Hemmen, J. L., & Wagner, H. (1996). A neuronal learning rule for sub-millisecond temporal coding. Nature, 383, 76–78. Gerstner, W., & Kistler, W. M. (2002). Mathematical formulations of Hebbian learning. Biol. Cybern., 87, 404–415. Goldberg, J., Holthoff, K., & Yuste, R. (2002). A problem with Hebb and local spikes. Trends Neurosci., 25(9), 433–435. Golding, N., Kath, W. L., & Spruston, N. (2001). Dichotomy of action-potential backpropagation in CA1 pyramidal neuron dendrites. J. Neurophysiol., 86, 2998–3010. Golding, N. L., Staff, P. N., & Spruston, N. (2002). Dendritic spikes as a mechanism for cooperative long-term potentiation. Nature, 418, 326–331. Gormezano, I., Kehoe, E. J., & Marshall, B. S. (1983). Twenty years of classical conditioning research with the rabbit. In J. M. Sprague, & A. N. Epstein, (Eds.), Progress of psychobiology and physiological psychology (pp. 198–274). Orlando, FL: Academic Press. Graef, I. A., Mermelstein, P. G., Stankunas, K., Neilson, J. R., Deisseroth, K., Tsien, R. W., & Crabtree, G. R. (1999). L-type calcium channels and GSK-3 regulate the activity of NF-ATc4 in hippocampal neurons. Nature, 401, 703– 708. Greengard, P., Allen, P. B., & Nairn, A. C. (1999). Beyond the dopamine receptor: The DARPP-32/protein phosphatase-1 cascade. Neuron, 23, 435–447. Groenewegen, H. J., Berendse, H. W., Wolters, J. G., & Lohman, A. H. M. (1990). The anatomical relationship of the prefrontal cortex with the striatopallidal system, the thalamus and the amygdala: Evidence for a parallel organization. Prog. Brain Res., 85, 95–118. Grossberg, S. (1995). A spectral network model of pitch perception. J. Acoust. Soc. Am., 98(2), 862–879. Grossberg, S., & Merrill, J. (1996). The hippocampus and cerebellum in adaptively timed learning, recognition and movement. J. Cogn. Neurosci., 8, 257– 277. Grossberg, S., & Schmajuk, N. (1989). Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Networks, 2, 79– 102. Groves, P. M., Garcia-Munoz, M., Linder, J. C., Manley, M. S., Martone, M. E., & Young, S. J. (1995). Elements of the intrinsic organization and information processing in the neostriatum. In J. C. Houk, J. L. Davis, & D. G., Beiser, (Eds.), Models of information processing in the basal ganglia (pp. 51–96). Cambridge, MA: MIT Press.

Temporal Sequence, Prediction, and Control

309

Gruber, A. J., Solla, S. A., Surmeier, D. J., & Houk, J. C. (2003). Modulation of striatal single units by expected reward: A spiny neuron model displaying dopamine-induced bistability. J. Neurophysiol., 90, 1095–1114. Gustafsson, B., Wigstrom, H., Abraham, W. C., & Huang, Y.-Y. (1987). Longterm potentiation in the hippocampus using depolarizing current pulses as the conditioning stimulus to single volley synaptic potentials. J. Neurosci., 7, 774–780. Gutig, ¨ R., Aharonov, R., Rotter, S., & Sompolinsky, H. (2003). Learning input correlations through nonlinear temporally asymmetric Hebbian plasticity. J. Neurosci., 23(9), 3697–3714. Haber, S. N., Fudge, J. L., & McFarland, N. R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J. Neurosci., 20, 2369–2382. Han, V. Z., Grant, K., & Bell, C. C. (2000). Reversible associative depression and nonassociative potentiation at a parallel fiber synapse. Neuron, 27, 611–622. Hassani, O. K., Cromwell, H. C., & Schultz, W. (2001). Influence of expectation of different rewards on behavior-related neuronal activity in the striatum. J. Neurophysiol., 85, 2477–2489. Hauber, W., Bohn, I., & Giertler, C. (2000). NMDA, but not dopamine D2, receptors in the rat nucleus accumbens are involved in guidance of instrumental behavior by stimuli predicting reward magnitude. J. Neurosci., 20(16), 6282– 6288. H¨ausser, M., Spruston, N., & Stuart, G. J. (2000). Diversity and dynamics of dendritic signaling. Science, 11, 739–744. Hebb, D. O. (1949). The organization of behavior: A neuropsychological study. New York: Wiley-Interscience. Hemmings, H. C., J., Greengard, P., Tung, H. Y., & Cohen, P. (1984). DARPP-32, a dopamine-regulated neuronal phosphoprotein, is a potent inhibitor of protein phosphatase-1. Nature, 310, 503–505. Hernandez-Lopez, S., Bargas, J., Surmeier, D. J., Reyes, A., & Galarraga, E. (1997). D1 receptor activation enhances evoked discharge in neostriatal medium spiny neurons by modulating an L-type Ca2 + conductance. J. Neurosci., 17, 3334–3342. Hoffman, D. A., Magee, J. C., Colbert, C. M., & Johnston, D. (1997). K+ channel regulation of signal propagation in dendrites of hippocampal pyramidal neurons. Nature, 387, 869–875. Hollerman, J. R., & Schultz, W. (1998). Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci., 1(4), 304–309. Hollerman, J. R., Tremblay, L., & Schultz, W. (1998). Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol., 80, 947–963. Houk, J. C., Adams, J. L., & Barto, A. G. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In J. C. Houk, J. L. Davis, & D. G., Beiser (Eds.), Models of information processing in the basal ganglia (pp. 249–270). Cambridge, MA: MIT Press.

310

F. Worg ¨ otter ¨ and B. Porr

Hull, C. L. (1939). The problem of stimulus equivalence in behavior theory. Psychological Review, 46, 9–30. Hull, C. L. (1943). Principles of behavior. New York: Appleton-Century-Crofts. Izhikevich, E. M., & Desai, N. S. (2003). Relating STDP to BCM. Neural Comp., 15, 1511–1523. Jaakkola, T., Jordan, M. I., & Singh, S. P. (1995). On the convergence of stochastic iterative dynamic programming algorithms. Neural Comp., 6(6), 1185–1201. Joel, D., Niv, Y., & Ruppin, E. (2002). Actor-critic models of the basal ganglia: New anatomical and computational perspectives. Neural Networks, 15, 535– 547. Joel, D., & Weiner, I. (2000). The connections of the dopaminergic system with the striatum in rats and primates: An analysis with respect to the functional and compartmental organization of the striatum. Neurosci., 96, 451–474. Johnston, D., Magee, J. C., Colbert, C. M., & Cristie, B. R. (1996). Active properties of neuronal dendrites. Annu. Rev. Neurosci., 19, 165–186. Joyce, J. N., & Marshall, J. F. (1987). Quantitative autoradiography of dopamine D2 sites in rat caudate-putamen: Localization to intrinsic neurons and not to neocortical afferents. Neurosci., 20, 773–795. Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237–285. Karmarkar, U. R., & Buonomano, D. V. (2002). A model of spike-timing dependent plasticity: One or two coincidence detectors? J. Neurophysiol., 88, 507–513. Karmarkar, U. R., Najarian, M. T., & Buonomano, D. V. (2002). Mechanisms and significance of spike-timing dependent plasticity. Biol. Cybern., 87, 373–382. Kawagoe, R., Takikawa, Y., & Hikosaka, O. (1998). Expectation of reward modulates cognitive signals in the basal ganglia. Nat. Neurosci., 1, 411– 416. Kempter, R., Gerstner, W., & van Hemmen, J. L. (1999). Hebbian learning and spiking neurons. Phys. Rev. E., 59, 4498–4515. Kempter, R., Gerstner, W., & van Hemmen J. L. (2001a). Intrinsic stabilization of output rates by spike-based Hebbian learning. Neural Comp., 13, 2709–2741. Kempter, R., Leibold, C., Wagner, H., & van Hemmen, J. L. (2001b). Formation of temporal-feature maps by axonal propagation of synaptic learning. Proc. Natl. Acad. Sci. (USA), 98(7), 4166–4171. King, M. M., Huang, C. Y., Chock, P. B., Nairn, A. C., Hemmings, H. C., J., Chan, K. F., & Greengard, P. (1984). Mammalian brain phosphoproteins as substrates for calcineurin. J. Biol. Chem., 259, 8080–8083. Kistler, W. M. (2002). Spike-timing dependent synaptic plasticity: A phenomenological framework. Biol. Cybern., 87, 416–427. Kistler, W. M., & van Hemmen, J. L. (2000). Modeling synaptic plasticity in conjunction with the timing of pre- and post-synaptic action potentials. Neural Comput., 12, 385–405. Kitano, K., Okamoto, H., & Fukai, T. (2003). Time representing cortical activities: Two models inspired by prefrontal persistent activity. Biol. Cybern, 88(5), 387– 394.

Temporal Sequence, Prediction, and Control

311

Klopf, A. H. (1972). Brain function and adaptive systems—a heterostatic theory. (Tech. Rep. Air Force Cambridge Research Laboratories Special Report No. 133), Alexandria, VA: Defense Technical Information Center, Cameron Station. Klopf, A. H. (1982). The hedonistic neuron: A theory of memory, learning, and intelligence. Washington, DC: Hemisphere. Klopf, A. H. (1986). A drive-reinforcement model of single neuron function. In J. S. Denker (Eds.), Neural networks for computing: AIP Conf. Proc., New York: American Institute of Physics. Klopf, A. H. (1988). A neuronal model of classical conditioning. Psychobiol., 16(2), 85–123. Knutson, B., Fong, G. W., Adams, C. M., Varner, J. L., & Hommer, D. (2001). Dissociation of reward anticipation and outcome with event-related fMRI. Neuroreport, 12, 3683–3687. Koester, H. J., & Sakmann, B. (1998). Calcium dynamics in single spines during coincident pre- and postsynaptic activity depend on relative timing of back-propagating action potentials and subthreshold excitatory postsynaptic potentials. Proc. Natl. Acad. Sci. USA, 95, 9596–9601. Kosco, B. (1986). Differential Hebbian learning. In J. S. Denker, (Ed.), Neural networks for computing: AIP Conference Proc. New York: American Institute of Physics. Kurata, K., & Wise, S. P. (1988). Premotor and supplementary motor cortex in rhesus monkeys: Neuronal activity during externally- and internally-instructed motor tasks. Exp. Brain Res., 72, 237–248. Lasser-Ross, N., & Ross, W. N. (1992). Imaging voltage and synaptically activated sodium transients in cerebellar Purkinje cells. Proc. R. Soc. B. Biol. Sci., 247, 35–39. Leibold, C., Kempter, R., & van Hemmen, J. L. (2001). Temporal map formation in the barn owl’s brain. Phys. Rev. Lett., 87(24), 248101–1–248101–4. Leibold, C., & van Hemmen, J. L. (2001). Temporal receptive fields, spikes, and Hebbian delay selection. Neural Networks, 14(6-7), 805–813. Leibold, C., & van Hemmen, J. L. (2002). Mapping time. Biol. Cybern., 87, 428–439. Levy, N., Horn, D., Meilijson, I., & Ruppin, E. (2001). Distributed synchrony in a cell assembly of spiking neurons. Neural Networks, 14, 815–824. Levy, W. B., & Steward, O. (1983). Temporal contiguity requirements for longterm associative potentiation/depression in the hippocampus. Neurosci., 8, 791–797. Linden, D. J. (1999). The return of the spike: Postsynaptic action potentials and the induction of LTP and LTD. Neuron, 22, 661–666. Lisman, J. (1989). A mechanism for the Hebb and the anti-Hebb processes underlying learning and memory. Proc. Natl. Acad. Sci. USA, 86, 9574–9578. Ljungberg, T., Apicella, P., & Schultz, W. (1992). Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol., 67, 145–163. Lovinger, D. M., Tyler, E. C., & Merritt, A. (1993). Short- and long-term synaptic depression in rat neostratium. J. Neurophysiol., 70, 1937–1949. Luscher, ¨ C., & Frerking, M. (2001). Restless AMPA receptors: Implications for synaptic transmission and plasticity. Trends Neurosci., 24(11), 665–670.

312

F. Worg ¨ otter ¨ and B. Porr

Lynd-Balta, E., & Haber, S. N. (1994). Primate striatonigral projections: A comparison of the sensorimotor-related striatum and the ventral striatum. J. Comp. Neurol., 345, 562–578. Mackintosh, N. J. (1974). The psychology of animal learning. Orlando, FL: Academic Press. Mackintosh, N. J. (1983). Conditioning and associative learning. New York: Oxford University Press. Magee, J. C., & Johnston, D. (1997). A synaptically controlled, associative signal for Hebbian plasticity in hippocampal neurons. Science, 275, 209–213. Malenka, R. C., Kauer, J. A., Perkel, D. J., Mauk, M. D., Kelly, P. T., Nicoll, R. A., & Waxham, M. N. (1989). An essential role for postsynaptic calmodulin and protein kinase activity in long-term potentiation. Nature, 340, 554–557. Malenka, R. C., & Nicoll, R. A. (1999). Long-term potentiation—a decade of progress? Science, 285, 1870–1874. Malinow, R., Schulman, H., & Tsien, R. W. (1989). Inhibition of postsynaptic PKC or CaMKII blocks induction but not expression of LTP. Science, 245, 862–866. Markram, H., Lubke, ¨ J., Frotscher, M., & Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science, 275, 213–215. Martin, S. J., Grimwood, P. D., & Morris, R. G. M. (2000). Synaptic plasticity and memory: An evaluation of the hypothesis. Annu. Rev. Neurosci., 23, 649–711. Martinez, J. L., & Derrick, B. E. (1996). Long-term potentiation and learning. Annu. Rev. Psychol., 47, 173–203. Mayer, M. L., Westbrook, G. L., & Guthrie, P. B. (1984). Voltage-dependent block by Mg2 + of NMDA responses in spinal cord neurones. Nature, 309, 261–263. McFarland, D. J. (1989). Problems of animal behavior. Harlow: Longman Scientific & Technical. Mehta, M. R. (2004). Cooperative LTP can map memory sequences on dendritic branches. TINS, 27, 69–72. Mehta, M. R., Lee, A. K., & Wilson, M. A. (2002). Role of experience and oscillations in transforming a rate code into a temporal code. Nature, 417(6890), 741–746. Melamed, O., Gerstner, W., Maass, W., Tsodyks, M., & Markram, H. (2004). Coding and learning of behavioral sequences. Trends Neurosci, 27(1), 11–14. Memo, M., Lovenberg, W., & Hanbauer, I. (1982). Agonist-induced subsensitivity of adenylate cyclase coupled with a dopamine receptor in slices from rat corpus striatum. Proc. Natl. Acad. Sci. USA, 79, 4456–4460. Migliore, M., Hoffmann, D. A., Magee, J. C., & Johnston, D. (1999). Role of an A-type K+ conductance in the back-propagation of action potentials in the dendrites of hippocampal pyramidal neurons. J. Comput. Neurosci., 7, 5–15. Miller, J. D., Sanghera, M. K., & German, D. C. (1981). Mesencephalic dopaminergic unit activity in the behaviorally conditioned rat. Life Sci., 29, 1255–1263. Mirenowicz, J., & Schultz, W. (1994). Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol., 72(2), 1024–1027. Montague, P. R., Dayan, P., Person, C., & Sejnowski, T. J. (1995). Bee foraging in uncertain environments using predictive Hebbian learning. Nature, 377, 725–728.

Temporal Sequence, Prediction, and Control

313

Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci., 16(5), 1936–1947. Morris, B. J., Simpson, C. S., Mundell, S., Maceachern, K., Johnston, H. M., & Nolan, A. M. (1997). Dynamic changes in NADPH-diaphorase staining reflect activity of nitric oxide synthase: Evidence for a dopaminergic regulation of striatal nitric oxide release. Neuropharmacology, 36, 1589–1599. Mulkey, R. M., Endo, S., Shenolikar, S., & Malenka, R. C. (1994). Involvement of a calcineurin/inhibitor-1 phosphatase cascade in hippocampal long-term depression. Nature, 369, 486–488. Nicola, S. M., Surmeier, J., & Malenka, R. C. (2000). Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. Ann. Rev. Neurosci., 23, 185–215. Nishi, A., Snyder, G. L., & Greengard, P. (1997). Bidirectional regulation of DARPP-32 phosphorylation by dopamine. J. Neurosci., 17, 8147–8155. Nishiyama, M., Hong, K., Mikoshiba, K., Poo, M., & Kato, K. (2000). Calcium release from internal stores regulates polarity and input specificity of synaptic modification. Nature, 408, 584–588. Nobre, A. C., Coull, J. T., Frith, C. D., & Mesulam, M. M. (1999). Orbitofrontal cortex is activated during breaches of expectation in tasks of visual attention. Nature Neurosci., 2, 11–12. Nowak, L., Bregestovski, P., Ascher, P., Herbet, A., & Prochiantz, A. (1984). Magnesium gates glutamate-activated channels in mouse central neurones. Nature, 307, 462–465. O’Doherty, J., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron, 28, 329–337. O’Doherty, J., Deichman, R., Critchley, H. D., & Dolan, R. J. (2002). Neural responses during anticipation of primary taste reward. Neuron, 33, 815–826. O’Doherty, J., Rolls, E. T., Francis, S., Bowtell, R., & McGlone, F. (2001). Representation of pleasant and aversive taste in the human brain. J. Neurophysiol., 85, 1315–1321. Overton, P. G., & Clark, D. (1997). Burst firing in midbrain dopaminergic neurons. Brain Res. Rev., 25, 312–334. Pacheco-Cano, M. T., Bargas, J., Hernandez-Lopez, S., Tapia, D., & Galarraga, E. (1996). Inhibitory action of dopamine involves a subthreshold Cs+ -sensitive conductance in neostriatal neurons. Exp. Brain Res., 110, 205–211. Pagoni, G., Zink, C. F., Montague, P. R., & Berns, G. S. (2002). Activity in human ventral striatum locked to errors of reward prediction. Nature Neurosci., 5, 97–98. Panchev, C., Wermter, S., & Chen, H. (2002). Spike-timing dependent competitive learning of integrate-and-fire neurons with active dendrites. In Lecture Notes in Computer Science. Proc. Int. Conf. Artificial Neural Networks (pp. 896– 901). Berlin: Springer. Parent, A. (1990). Extrinsic connections of the basal ganglia. Trends Neurosci., 13, 254–258.

314

F. Worg ¨ otter ¨ and B. Porr

Partridge, J. G., Tang, K. C., & Lovinger, D. M. (2000). Regional and postnatal heterogeneity of activity-dependent long-term changes in synaptic efficacy in the dorsal striatum. J. Neurophysiol., 84, 1422–1429. Pavlov, P. I. (1927). Conditioned reflexes. New York: Oxford University Press. Peng, J. (1993). Efficient dynamic programming-based learning for control. Unpublished doctoral dissertation, Northeastern University. Pennartz, C. M., Ameerun, R. F., Groenewegen, H. J., & Lopes da Silva, F. H. (1993). Synaptic plasticity in an in vitro slice preparation of the rat nucleus accumbens. European J. Neurosci., 5, 107–117. Pike, F. G., Meredith, R. M., Olding, A. A., & Paulsen, O. (1999). Postsynaptic bursting is essential for ”Hebbian” induction of associative long-term potentiation at excitatory synapses in rat hippocampus. J. Physiol. (Lond.), 518, 571–576. Pollack, J. B., & Blair, A. D. (1997). Why did TD-Gammon work? In M. Mozer, M. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems, (pp. 10–16). Cambrdige, MA: MIT Press. Porr, B., von Ferber, C., & Worg ¨ otter, ¨ F. (2003). ISO-learning approximates a solution to the inverse-controller problem in an unsupervised behavioral paradigm. Neural Comp., 15, 865–884. Porr, B., & Worg ¨ otter, ¨ F. (2002). Isotropic sequence order learning using a novel linear algorithm in a closed loop behavioural system. Biosystems, 67(1–3), 195–202. Porr, B., & Worg ¨ otter, ¨ F. (2003a). Isotropic sequence order learning. Neural Comp., 15, 831–864. Porr, B., & Worg ¨ otter, ¨ F. (2003b). Isotropic sequence order learning in a closed loop behavioural system. Proc. Roy. Soc. B. Porr, B., & Worg ¨ otter, ¨ F. (2003c). Isotrpic sequence order learning in a closed loop behavioural system. Roy. Soc. Phil. Trans. Mathematical, Physical, & Engineering Sciences, 361, 2225–2244. Prokasy, W. F., Hall, J. F., & Fawcett, J. T. (1962). Adaptation, sensitization, forward and backward conditioning, and pseudo-conditioning of the GSR. Psychol. Rep., 10, 103–106. Pucak, M. L., & Grace, A. A. (1994). Regulation of substantia nigra dopamine neurons. Crit. Rev. Neurobiol., 9, 67–89. Rao, R. P. N., & Sejnowski, T. J. (2001). Spike-timing-dependent Hebbian plasticity as temporal difference learning. Neural Comp., 13, 2221–2237. Redgrave, P., Prescott, T. J., & Gurney, K. (1999). Is the short-latency dopamine response too short to signal reward? Trends Neurosci., 22, 146–151. Regehr, W. G., Konnerth, A., & Armstrong, C. M. (1992). Sodium action potentials in the dendrites of cerebellar Purkinje cells. Proc. Natl. Acad. Sci. USA, 89, 5492–5496. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory. New York: Appleton Century Crofts. Reynolds, J. N. J., & Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks, 15, 507–521.

Temporal Sequence, Prediction, and Control

315

Reynolds, S. I. (2002). Reinforcement learning with exploration. Unpublished doctoral dissertation, University of Birmingham. Roberts, P. D. (1999). Computational consequences of temporally asymmetric learning rules: I. Differential Hebbian learning. J. Comput. Neurosci., 7(3), 235–246. Roberts, P. D., & Bell, C. C. (2002). Spike timing dependent synaptic plasticity in biological systems. Biol. Cybern., 87, 392–403. Roche, K. W., O’Brian, R. J., Mammen, A. L., Bernhardt, J., & Huganir, R. L. (1996). Characterization of multiple phosphorylation sites on the AMPA receptor GluR1 subunit. Neuron, 16, 1179–1188. Rose, C. R., & Konnerth, A. (2001). Stores not just for storage, intracellular calcium release and synaptic plasticity. Neuron, 31, 519–522. Rubin, J., Lee, D. D., & Sompolinsky, H. (2001). Equilibrium properties of temporally asymmetric Hebbian plasticity. Phys. Rev. Lett., 86(2), 364–367. Rummery, G. A. (1995). Problem solving with reinforcement learning. Unpublished doctoral dissertation, Cambridge University. Russell, V. A., Allin, R., Lamm, M. C., & Taljaard, J. J. (1992). Regional distribution of monoamines and dopamine D1- and D2-receptors in the striatum of the rat. Neurochemical Res., 17, 387–395. Santos, J. M., & Touzet, C. (1999a). Dynamic update of the reinforcement function during learning. Connection Science [Special issue], 11 (3–4). Santos, J. M. & Touzet, C. (1999b). Exploration tuned reinforcement function. Neurocomputing [Special issue], 28(1–3), 93–105. Sato, N., & Yamaguchi, Y. (2003). Memory encoding by theta phase precession in the hippocampal network. Neural Computation, 15, 2379–2397. Saudargiene, A., Porr, B., & Worg ¨ otter, ¨ F. (2004). How the shape of pre- and postsynaptic signals can influence STDP: A biophysical model. Neural Comp., 16, 595–626. Schiller, J., Schiller, Y., Stuart, G., & Sakmann, B. (1997). Calcium action potentials restricted to distal apical dendrites of rat neocortical pyramidal neurons. J. Physiol., 505, 605–616. Schoenbaum, G., Chiba, A. A., & Gallagher, M. (1998). Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nature Neurosci., 1, 155–159. Schultz, W. (1986). Responses of midbrain dopamine neurons to behavioral trigger stimuli in the monkey. J. Neurophysiol., 56, 1439–1462. Schultz, W. (1998). Predictive reward signal of dopamine neurons. J. Neurophysiol., 80, 1–27. Schultz, W. (2002). Getting formal with dopamine and reward. Neuron, 36, 241–263. Schultz, W., Apiccela, P., Scarnati, E., & Ljungberg, T. (1992). Neuronal activity in monkey ventral striatum related to the expectation of reward. J. Neurosci., 12, 4595–4610. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593–1599. Schultz, W., & Dickinson, A. (2000). Neuronal coding of prediction errors. Annu. Rev. Neurosci., 23, 473–500.

316

F. Worg ¨ otter ¨ and B. Porr

Senn, W. & Buchs, N. J. (2003). Spike-based synaptic plasticity and the emergence of direction selective simple cells: Mathematical analysis. J. Comput. Neurosci., 14, 119–138. Senn, W., Markram, H., & Tsodyks, M. (2000). An algorithm for modifying neurotransmitter release probability based on pre-and postsynaptic spike timing. Neural Comp., 13, 35–67. Seung, H. S. (1998). Learning continous attractors in recurrent networks. In M. Kearns, M. Jordan, & S. Solla, (Eds.), Advances in neural information processing systems, 10 (pp. 654–660). Cambridge, MA: MIT Press. Shouval, H. Z., Bear, M. F., & Cooper, L. N. (2002). A unified model of NMDA receptor-dependent bidirectional synaptic plasticity. Proc. Natl. Acad. Sci. (USA), 99(16), 10831–10836. Sibley, D. R., & Monsma, F. J. J. (1992). Molecular biology of dopamine receptors. Trends Pharmakol. Sci., 13, 61–69. Singh, S. P., & Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces. Mach. Learn., 22, 123–158. Sjostr ¨ om, ¨ P. J., Turrigiano, G. G., & Nelson, S. B. (2001). Rate, timing, and cooperativity jointly determine cortical synaptic plasticity. Neuron, 32, 1149–1164. Song, I., & Huganir, R. L. (2002). Regulation of AMPA receptors during synaptic plasticity. Trends Neurosci., 25(11), 578–588. Song, S., & Abbott, L. F. (2001). Cortical development and remapping through spike timing-dependent plasticity. Neuron, 32, 1–20. Song, S., Miller, K. D., & Abbott, L. F. (2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nature Neurosci., 3, 919– 926. Spencer, J. P., & Murphy, K. P. (2000). Bi-directional changes in synaptic plasticity induced at corticostriatal synapses in vitro. Exp. Brain Res., 135, 497–503. Sterratt, D. C., & van Ooyen, A. (2002). Does morphology influence temporal plasticity? In J. R. Dorronsoro, (Ed.), ICANN 2002 (Vol. LNCS 2415, pp. 186– 191). Berlin: Springer-Verlag. Stoof, J. C., & Kebabian, J. W. (1981). Opposing roles for D-1 and D-2 dopamine receptors in efflux of cyclic AMP from rat neostriatum. Nature, 294, 366–368. Stuart, G. J., & Sakmann, B. (1994). Active propagation of somatic action potentials into neocortical pyramidal cell dendrites. Nature, 367, 69–72. Stuart, G., Spruston, N., Sakmann, B., & H¨ausser, M. (1997). Action potential initiation and backpropagation in neurons of the mammalian central nervous system. Trends Neurosci., 20, 125–131. Suri, R. E., Bargas, J., & Arbib, M. A. (2001). Modeling functions of striatal dopamine modulation in learning and planning. Neurosci., 103(1), 65–85. Suri, R. E., & Schultz, W. (1998). Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp. Brain Res., 121, 350–354. Suri, R. E., & Schultz, W. (1999). A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neurosci., 91(3), 871–890. Suri, R. E., & Schultz, W. (2001). Temporal difference model reproduces anticipatory neural activity. Neural Comp., 13(4), 841–862.

Temporal Sequence, Prediction, and Control

317

Surmeier, D. J., Bargas, J., Hemmings, H. C., J., Nairn, A. C., & Greengard, P. (1995). Modulation of calcium currents by a D1 dopaminergic protein kinase/phosphatase cascade in rat neostriatal neurons. Neuron, 14, 385–397. Surmeier, D. J., & Kitai, S. T. (1993). D1 and D2 dopamine receptor modulation of sodium and potassium currents in rat neostriatal neurons. Prog. Brain Res., 99, 309–324. Sutton, R. S. (1984). Temporal credit assignment in reinforcement learning. Unpublished doctoral dissertation, University of Massachusetts. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Mach. Learn., 3, 9–44. Sutton, R. S. (1996). Generalization in reinforcement learning: Successful examples using sparse coarse coding. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 1038– 1044). Cambridge, MA: MIT Press. Sutton, R. S. (1999). Open theoretical questions in reinforcement learning. In EUROColt, (pp. 11–17). Sutton, R., & Barto, A. (1981). Towards a modern theory of adaptive networks: Expectation and prediction. Psychol. Review, 88, 135–170. Sutton, R. S., & Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel, & J. Moore (Eds.), Learning and computational neuroscience: Foundation of adaptive networks. Cambridge, MA: MIT Press. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press. Svenningsson, P., Lindskog, M., Rognoni, F., Fredholm, B. B., Greengard, P., & Fisone, G. (1998). Activation of adenosine A2A and dopamine D1 receptors stimulates cyclic AMP-dependent phosphorylation of DARPP-32 in distinct populations of striatal projection neurons. Neurosci., 84, 223–228. Svoboda, K., & Mainen, Z. F. (1999). Synaptic Ca2+ : intracellular stores spill their guts. Neuron, 22, 427–430. Tang, K., Low, M. J., Grandy, D. K., & Lovinger, D. M. (2001). Dopaminedependent synaptic plasticity in striatum during in vivo development. Proc. Natl. Acad. Sci. USA, 98, 1255–1260. Tesauro, G. (1992). Practical issues in temporal difference learning. Mach. Learn., 8, 257–277. Tingley, W. G., Ehlers, M. D., Kameyama, K., Doherty, C., Ptak, J. B., Riley, C. T., & Huganir, R. L. (1997). Characterization of the protein kinase A and protein kinase C phosphorylation of the N-methyl-D-aspartate receptor NR1 subunit using phosphorylation site-specific antibodies. J. Biol. Chem., 272, 5157–5166. Toan, D. L., & Schultz, W. (1985). Responses of rat pallidum cells to cortex stimulation and effects of altered dopaminergic activity. Neurosci., 15, 683– 694. Touzet, C. (1999). Neural networks and Q-learning for robotics. Available online at: http://www.sciences-cognitives.org/scico/annuaire/Touzet Claude/ Touzet IJCNN Tut.pdf. Tremblay, L., Hollerman, J. R., & Schultz, W. (1998). Modifications of reward expectation-related neuronal activity during learning in primate striatum. J. Neurophysiol., 80, 964–977.

318

F. Worg ¨ otter ¨ and B. Porr

Tremblay, L., & Schultz, W. (1999). Relative reward preference in primate orbitofrontal cortex. Nature, 398, 704–708. Umemiya, M., & Raymond, L. A. (1997). Dopaminergic modulation of excitatory postsynaptic currents in rat neostriatal neurons. J. Neurophysiol., 78, 1248– 1255. van Hemmen, J. L. (2001). Theory of synaptic plasticity. In F. Moss, & S. Gielen, (Eds.), Handbook of biological physics, neuro-informatics, neural modelling (Vol. 4, pp. 771–823. Amsterdam: Elsevier. van Rossum, M. C. W., Bi, G. Q., & Turrigiano, G. G. (2000). Stable Hebbian learning from spike timing-dependent plasticity. J. Neurosci., 20(23), 8812– 8821. Walsh, J. P. (1993). Depression of excitatory synaptic input in rat striatal neurons. Brain Res., 608, 123–128. Walsh, J. P., & Dunia, R. (1993). Synaptic activation of N-methyl-D-aspartate receptors induces short-term potentiation at excitatory synapses in the striatum of the rat. Neurosci., 57, 241–248. Watkins, C. J. C. H. (1989). Learning from delayed rewards. Unpublished doctoral dissertation, Cambridge University. Watkins, C. J. C. H., & Dayan, P. (1992). Technical note: Q-learning. Mach. Learn., 8, 279–292. West, A. E., Chen, W. G., Dalva, M. B., Dolmetsch, R. E., Kornhauser, J. M., Shaywitz, A. J., Takasu, M. A., Tao, X., & Greenberg, M. E. (2001). Calcium regulation of neuronal gene expression. Proc. Natl. Acad. Sci. USA, 98, 11024– 11031. Wickens, J. R., Begg, A. J., & Arbuthnott, G. W. (1996). Dopamine reverses the depression of rat corticostriatal synapses which normally follows highfrequency stimulation of cortex in vitro. Neurosci., 70, 1–5. Widrow, B., & Hoff, M. E. (1960). Adaptive switching circuits. In IRE WESCON Convention Record (Vol. 4, pp. 96–104). New York. Wiering, M., & Schmidhuber, J. (1998). Fast online Q(λ). Mach. Learn., 33, 105–115. Williams, S. M., & Goldman-Rakic, P. S. (1993). Characterization of the dopaminergic innervation of the primate frontal cortex using a dopamine-specific antibody. Cereb. Cortex, 3, 199–222. Witten, I. H. (1977). An adaptive optimal controller for discrete-time Markov environments. Information and Control, 34, 86–295. Xie, X., & Seung, S. (2000). Spike-based learning rules and stabilization of persistent neural activity. In S. A. Solla, T. K. Leen, & K. R. Muller, ¨ (Eds.), Advances in neural information processing systems, 12 (pp. 199–208). Cambridge, MA: MIT Press. Yamamoto, Y., Nakanishi, H., Takai, N., Shimazoe, T., Watanabe, S., & Kita, H. (1999). Expression of N-methyl-D-aspartate receptor-dependent longterm potentiation in the neostriatal neurons in an in vitro slice after ethanol withdrawal of the rat. Neurosci., 91, 59–68. Yang, S. N., Tang, Y. G., & Zucker, R. S. (1999). Selective induction of LTP and LTD by postsynaptic Ca2+ elevation. J. Neurophysiol., 81, 781–787. Yao, H., & Dan, Y. (2001). Stimulus timing-dependent plasticity in cortical processing of orientation. Neuron, 32, 315–323.

Temporal Sequence, Prediction, and Control

319

Yim, C. Y., & Mogenson, G. J. (1982). Response of nucleus accumbens neurons to amygdala stimulation and its modification by dopamine. Brain Res., 239, 401–415. Zhang, L. I., Tao, H. W., Holt, C. E., Harris, W. A., & Poo, M.-M. (1998). A critical window for cooperation and competition among developing retinotectal synapses. Nature, 395, 37–44. Zink, C. F., Pagnoni, G., Martin, M. E., Dhamala, M., & Berns, G. S. (2003). Human striatal response to salient nonrewarding stimuli. J. Neurosci., 23(22), 8092–8097. Zucker, R. S. (1999). Calcium- and activity-dependent synaptic plasticity. Curr. Opin. Neurobiol., 9, 305–313. Received September 4, 2003; accepted July 2, 2004.

NOTE

Communicated by James Stone

A Note on Stone’s Conjecture of Blind Signal Separation Shengli Xie [email protected]

Zhaoshui He he [email protected]

Yuli Fu [email protected] School of Electronics and Information Engineering, South China University of Technology, Guangzhou 510640, China

Stone’s method is one of the novel approaches to the blind source separation (BSS) problem and is based on Stone’s conjecture. However, this conjecture has not been proved. We present a simple simulation to demonstrate that Stone’s conjecture is incorrect. We then modify Stone’s conjecture and prove this modified conjecture as a theorem, which can be used a basis for BSS algorithms. 1 Problem Formulation Blind source separation (BSS) separates the source signals from the observed signals without any prior information of the source signals and the transfer channel. “Blind” means that both the source and the channel are unknown. The mathematical model of this problem is written as X(t) = AT S(t)

(1.1)

Y(t) = W X(t),

(1.2)

T

where equation 1.1 is the model of the mixture, while equation 1.2 is the model of the separation. The blind (unknown) source signal vector is denoted as S(t) = (s1 (t), s2 (t), · · · , sn (t))T , and the observed signal vector is denoted as X(t) = (x1 (t), x2 (t), · · · , xm (t))T . The vector Y(t) = (y1 (t), y2 (t), · · · , yn (t))T indicating the separated signals. A is the unknown n × m mixture matrix (indicating the unknown channel). W is the separating matrix. The aim of BSS is to get the expression Y(t) = PDS(t)

(1.3)

by adjusting the separation matrix W. In equation 1.3, P is a permutation matrix, while D is a diagonal matrix. Usually the source signals are only assumed to be statistically independent. Neural Computation 17, 321–330 (2005)

c 2005 Massachusetts Institute of Technology 

322

S. Xie, Z. He, and Y. Fu

Many methods are used to deal with BSS, for example, the neural network method (Amari, Cichocki, & Yang 1996; Bell & Sejnowski 1995), the minimal mutual information method (Pham, 2002; Xie & Zhang, 2002), the higherorder statistics method (Belouchrani, Karim, Cardoso, & Moulines, 1997; Yeredor, 2000), and the geometric method (Taro, Hirokewa, & Itoh, 2000; Xie & Zhang, in press). Recently, based on Stone’s conjecture, a novel method was proposed to deal with BSS problems for instantaneous mixture model (Stone, 2001), convolution mixture model (Stone, 2002), and nonlinear mixture model (Martinez, & Bray, 2003). Some fast algorithms were proposed in those articles. Thus, Stone’s conjecture-based method has gained attention. Generally, new methods and algorithms should be based on a reliable theoretical basis. If Stone’s conjecture could be proved, the new algorithms proposed in Stone (2001, 2002) and Martinez and Bray (2003) indeed would have a theoretical basis. However, in many cases, Stone’s conjecture is incorrect. The purpose of this note is to modify Stone’s conjecture and prove the modified conjecture theoretically. First, we present a simple simulation to demonstrate the problems with Stone’s conjecture. Then we modify Stone’s conjecture and prove the modified conjecture as a theorem. Thus, the theorem can be used as a basis of BSS algorithms. 2 The Modification of Stone’s conjecture Stone (2001) proposed temporal predictability and Stone’s conjecture. Some algorithms of BSS were presented based on Stone’s conjecture. Now, we cite the temporal predictability and the Stone’s conjecture from the original paper (Stone, 2001). 2.1 Definition of Temporal Predictability and Stone’s Conjecture. 2.1.1 Temporal Predictability. dictability is defined as n


rz = log

(zτ − zτ )2

τ =1 n 
τ =1

For a given signal series z = {zτ }, its pre-

zτ − z˜ τ

2

zτ = λL zτ −1 + (1 − λL )zτ −1 : 0 ≤ λL ≤ 1, z˜τ = λS z˜ τ −1 + (1 − λS )zτ −1 : 0 ≤ λS ≤ 1, where λL = 2−1/ hl , λS = 2−1/ hs , and hS , hl are the parameters will be selected.

A Note on Stone’s Conjecture of Blind Signal Separation

323

Figure 1: Waveforms of the three independent source signals s1 , s2 and s3 .

2.1.2 Stone’s Conjecture. The conjecture is, ”The temporal predictability of any signal mixture is less than (or equal to) that of any of its component source signals” (Stone, 2001). 2.2 The Faultiness of Stone’s Conjecture. Stone (2001) gave a counterexample to his conjecture and redefined the predictability. However, the source signals in this counterexample are not independent. Also, he did not prove the original conjecture in his early and later papers. By a simple example with independent source signals, we will point out the faultiness of the Stone’s conjecture. The three independent source signals (see Figure 1) are taken from Stone’s experiments (Stone, 2001). The supergaussian signal s1 is with the maximal temporal predictability rs1 = 5.2332, the sorted gaussian noise s2 is with the second maximal temporal predictability rs2 = 3.8966, and the subgaussian sine wave s3 is with the minimal temporal predictability rsnew = 1.8381. A mixed signal is taken as snew = s1 + s2 . Its temporal predictability can be computed by the definition rsnew = 5.1045. Obviously, we have rsnew > rs2 . Therefore, Stone’s conjecture is not always true for the mixture of the subgaussian signal, gaussian signal, and supergaussian signal. In fact, some counterexamples can also be given by taking some other type of signals.

324

S. Xie, Z. He, and Y. Fu

2.3 Modification of Stone’s Conjecture. For convenience, we define an equivalent index to replace temporal predictability. Definition 1 (predictive exponent). predicted exponent is defined as n


Vz =

(zτ − zτ )2

τ =1 n 
τ =1

For a given signal series z = {zτ }, its

zτ − z˜τ

2

(= exp(rz )).

(2.1)

Obviously, the predictive exponent is monotonic with respect to the temporal predictability. So the predictive exponent is equivalent to temporal predictability in Stone (2001). Thus, the statements in Stone’s conjecture can be rewritten in terms of predictive exponent. Although Stone’s conjecture is not true in general, it could be modified in the following way: Modified Conjecture. The temporal predictability (or equivalently, the predictive exponent) of any linear mixture of the source signals is between the greatest temporal predictability (or equivalently, the predictive exponent) and the smallest temporal predictability (or equivalently, the predictive exponent) of the source signals. The modified conjecture can be written as a theorem and can be proved exactly. Note that Stone’s method proposed some generalized eigenvalue-based algorithms. In the algorithms, the indices (temporal predictability or, equivalently, predictive exponent) should be sorted by values. Since the index temporal predictability (or, equivalently, predictive exponent) is time varying, it is not easily used in the algorithms, so we introduce a generalized index: Definition 2 (covariance rate). Consider a signal z(t) ∈ L(s1 (t), s2 (t), . . . , sk (t)), where L(s1 (t), s2 (t), . . . , sK (t)) denotes the linear space spanned by the signals s1 (t), s2 (t), . . . , sk (t). The index Rz =

cov( fza (t), fza (t)) cov(gbz (t), gbz (t))

(2.2)

is called a covariance rate of signal z(t), where fza = zk −zk , gaz = zk −zk , and zk , z˜ k are defined as in the definition of temporal predictability.

A Note on Stone’s Conjecture of Blind Signal Separation

325

The covariance rate implies the predictive exponent. The covariance rate is more general than the definitions of predictive exponent and temporal predictability, and is more easily used in both computation and theoretical analysis. From the independence of all the source signals, we have cov(s1 (t), sj (t)) = 0, (i = j). The covariance rate Rz of signal z(t) is independent of time t(time invariant). Therefore, for ∀z1 (t), z2 (t), . . . , zm (t) ∈ L(s1 (t), s2 (t), . . . , sk (t)), we can assume that Rz11 ≥ Rzl12 . . . , ≥ Rzlm > 0,

(2.3)

where {l1, 12, . . . , lm} is a permutation of {1, 2, . . . , m}. For any signal z(t) ∈ L(s1 (t), s2 (t), . . . , sk (t)), let z(t) = WT S(t), W ∈ Rk . Taking two constants λL , λs , 0 ≤ λs < λL ≤ 1, two estimates z(t) and z˜ (t) of the signal z(t) are defined by z(t) = λL z(t − 1) + (1 − λL )z(t − 1), t = 2, 3, . . . , N, z(1) = z(1)

(2.4)

z˜ (t) = λs z˜ (t − 1) + (1 − λs )z(t − 1), t = 2, 3, . . . , N, z˜ (1) = z(1).

(2.5)

Also, we set fza (t) = z(t) − z(t), gbz (t) = z(t) − z˜ (t), t = 1, 2, . . . , N

(2.6)

 T FaS (t) = fsa1 (t), fsa2 (t), . . . , fsak (t) ,  T Gbs (t) = gbs1 (t), gbs2 (t), . . . , gbsk (t) .

(2.7)

and

Some properties related to the generalized index covariance rate are obtained. Property 1. have

If the source signals s1 (t), s2 (t), . . . , sk (t) are independent, we

  cov Fas (t), Fas (t) =  0 cov( fsa1 (t), fsa1 (t)) a (t), f a (t))  0 cov( f s2 s2  = .. ..  . . 0 0

 ... 0  ... 0   .. ..  . . a a . . . cov( fsk (t), fsk (t))

326

S. Xie, Z. He, and Y. Fu

  cov Gbs (t), Gbs (t) =  cov(gbs1 (t), gbs1 (t)) 0 b (t), gb (t))  0 cov(g s2 s2  = .. ..  . . 0 0

Property 2. We have

 ... 0  ... 0   .. ..  . . b b . . . cov(gsk (t), gsk (t))

For z(t) ∈ L(s1 (t), s2 (t), . . . , sk (t)), set z(t) = WT S(t), W ∈ RK .

cov( fza (t), fza (t)) = WT cov(Fas (t), Fas (t))W, cov(gbz (t), gbz (t)) = WT cov(Gbs (t), Gbs (t))W. Property 3. For z1 (t), z2 (t) ∈ L(s1 (t), s2 (t), . . . , sk (t)), let z1 (t) = UT S(t), z2 (t) = VT S(t), U, V ∈ RK . We then have cov( fza1 (t), fza2 (t)) = UT cov(Fas (t), Fas (t))V, cov(gbz1 (t), gbz2 (t)) = UT cov(Gbs (t), Gbs (t))V. Property 4.

The covariance rate is homogeneous with zero order.

Property 1 tells us that the covariance of some independent signals can be calculated as some diagonal matrices. Properties 2 and 3 give the bilinear property of the covariance. And property 4 indicates that for any signal z(t) in the linear signal space and any constant c = 0, the covariance rate of z(t) is equal to that of cz(t). Property 4 will be helpful in computating the covariance rate. Property 3 is easy to understand. The proofs of properties 1, 2, and 4 are in the appendix. Now we can give the theorem to represent the modified conjecture: Theorem 1. In the linear signal space L(s1 (t), s2 (t), . . . , sK (t)), where the signals s1 (t), s2 (t), . . . , sk (t) are statistically independent, the covariance rate Rx of any signal x(t) ∈ L(s1 (t), s2 (t), . . . , sk (t)), is between the maximal and minimal covariance rates of the source signals. Under the following assumption, max

x(t)∈L(s1 (t),s2 (t),...,sk (t))

Rx = Rs1 ,

x(t) ∈ L(s1 (t), s2 (t), . . . , sk (t)),

min

x(t)∈L(s1 (t),s2 (t),...,sk (t))

Rx = Rsk , and

A Note on Stone’s Conjecture of Blind Signal Separation

327

Theorem 1 results in Rsk ≤ Rx ≤ Rs1 The proof of theorem 1 is given in the appendix. Therefore, theorem 1 can replace Stone’s conjecture as a theoretical basis of the BSS algorithms. 3 Conclusion This note elaborates on modifying Stone’s conjecture and establishing a theoretical basis of BSS. We present a simple example to point out that Stone’s conjecture is incorrect. Then we give a modified conjecture and prove it as a theorem. The theorem we present can be used as a theoretical basis for BSS algorithms. In future work, we will discuss BSS algorithms based on theorem 1. Appendix Proof of Property 1. Since the signals s1 (t), s2 (t), . . . , sk (t) are independent, by probability theory, the signals fsa1 (t), fsa2 (t), . . . , fsak (t) are so in  dependent. Thus, cov fsa1 (t), fsaj (t) = 0, when i = j, and the first equality is true. We can prove the second equality in same way. Proof of Property 2. and hence,

Since z(1) = WT S(1) , we have z(1) = WT S(1) ,

z(2) = λL z(1) + (1 − λL )z(1) = λL WT S(1) + (1 − λL )WT S(1) = WT (λL S(1) + (1 − λL )S(1)) = WT S(2).

(A.1)

Similarly, we have z(t) = WT S(t), t = 1, 2, . . . , N.

(A.2)

Then fza (t) = z(t) − z(t) = WT S(t) − WT S(t) = WT (s1 (t), s2 (t), . . . sk (t))T − (s1 (t), s2 (t), . . . , sk (t))T  = WT (s1 (t) − s1 (t), s2 (t), s2 (t), . . . , sk (t) − sk (t))T  T = WT fsa1 (t), fsa2 (t), . . . , fsak (t) = WT Fas (t).

(A.3)

By analogous argument, we have gbz (t) = WT Gbs (t).

(A.4)

328

S. Xie, Z. He, and Y. Fu

From the bilinear property of covariance, it follows that     cov fza (t), fza (t) = WT cov Fas (t), Fas (t)(t) W     cov gbz (t), gbz (t) = WT cov Gbs (t), Fas (t)(t) W.

(A.5)

The property is completed. Proof of Property 4. For the signal z(t) = WT S(t) and nonzero constant c, by the definition of covariance rate, we have   cov fcza (t), fcza (t)   = Rcz = cov gbcz (t), gbcz (t) =

(cW)T cov(Fas (t), Fas (t))(cW) (cW)T cov(Gbs (t), Gas (t))(cW)

= rz = c0 Rz .

(A.6)

This proves the property. Proof of Theorem 1. Let Rs1 ≥ Rs2 ≥ Rs1 · · · ≥ Rsk > 0, without loss of generality. Then     cov gbs1 (t), gbs1 (t) cov fsa1 (t), fsa1 (t)   ≥   ; i = 2, 3, · · · , k. (A.7) cov gbs1 (t), gbs1 (t) cov fsa1 (t), fsa1 (t) Define a matrix:



0 0  D1 =  .  .. 0 q22

0 q22 .. . 0

··· ··· .. . ···

 0 0  ..  .  qkk

    cov gbs2 (t), gbs2 (t) cov fsa2 (t), fsa2 (t)     = − cov gbs1 (t), gbs1 (t) cov fsa1 (t), fsa1 (t)

    cov gbsk (t), gbsk (t) cov fsak (t), fsak (t)  −  . qkk = cov gbs1 (t), gbs1 (t) cov fsa1 (t), fsa1 (t) By equation A.7, we find that D1 is nonnegative definite. Since x(t) is the linear combination of s1 (t), s2 (t), · · · , sk (t), we can write x(t) = AT1 S(t), where A1 is a k-dimensional column vector. So we have AT1 D1 A1 ≥ 0, that is,   0 0 ··· 0 0 q22 · · · 0    AT1  . .. ..  A1 ≥ 0. ..  .. . . .  0 0 · · · qkk

A Note on Stone’s Conjecture of Blind Signal Separation

It is equivalent to  0 v11  0 v22  AT1  . ..  .. . 0 0

··· ··· .. . ···

 0 0  ..  A1 .  vkk

cov(gbs1 (t), gbs1 (t))

 0 ··· 0 v˜ 11  0 v˜ 22 · · · 0    AT1  . .. ..  A1 ..  .. . . .  0 0 · · · v˜ kk ≥ cov( fsa1 (t), ( fsa1 (t))

329



(A.8)

  vii = cov(gbs1 (t), gbs1 (t)), v˜ ii = cov fsa1 (t), ( fsa1 (t) , i = 1, 2, . . . , k. From equation A.8 and property 2, we have   0 ··· 0 v˜ 11  0 v˜ 22 · · · 0    AT1  . .. ..  A1 ..  .. . . .   a  a cov fs1 (t), ( fs1 (t) 0 0 · · · v˜ kk   ≥   b b ˜ 0 ··· 0 v11 cov gs1 (t), (gs1 (t)  0 v˜ 22 · · · 0    AT1  . .. ..  A1 ..  .. . . .  0 0 · · · v˜ kk

=

    AT1 cov Fas (t), Fas (t) A1 cov fxa (t), fxa (t)   = Rx .   = cov gbx (t), gbx (t) AT1 cov Gbs (t), Gbs (t) A1

(A.9)

Hence, we have Rx ≤ Rs1 . By the same arguments, we have Rx ≥ Rsk . The proof of theorem 1 is complete. Acknowledgments The work is supported by Guang Dong Province Science Foundation for Program of Research Team (grant 04205783), the National Natural Science Foundation of China (grant 60274006), the Natural Science Key Fund of Guangdong Province, China (grant 020826), the National Natural Science Foundation of China for Excellent Youth (grant 60325310), and the TransCentury Training Program, Foundation for the Talents by the State Education Commission of China. We are also grateful to the reviewers for their helpful comments and suggestions. References Amari, S., Cichocki, A., & Yang, H. H. (1996). A new algorithm for blind source separation. In D. Touretsky, M. Mozer, & M. Wasselmo (Eds.), Advances in neural information processing, 8 (pp. 757–763). Cambridge, MA: MIT Press.

330

S. Xie, Z. He, and Y. Fu

Bell, A. J., & Sejnowski, T. J. (1995). An information-maximization approach to blind source separation and blind deconvolution. Neural Computation, 7, 1129–1159. Belouchrani, A., Karim, A. M., Cardoso, J. F., & Moulines, E. (1997). A blind source separation technique using second-order statistics. IEEE Trans. Signal Process, 45(2), 434–443. Martinez, D., & Bray, A. (2003). Nonlinear blind source separation using kernels. IEEE Trans. Neural Networks, 14(1), 228–235. Pham, D. T. (2002). Mutual information approach to blind separation for stationary of sources. IEEE Trans. Information Theory, 48, 1935–1946. Stone, J. V. (2001). Blind source separation using temporal predictability. Neural Computation, 13, 1559–1574. Stone, J. V. (2002). Blind deconvolution using temporal predictability. Neurocomputing, 49, 79–86. Taro, Y., Hirokawa, K., & Itoh, K. (2000). Independent component analysis by transforming a scatter diagram of mixtures of signal. Optics Communications, 173, 107–114. Xie, S., & Zhang, J. (2002). Blind separation algorithm of minimal mutual information based on rotating transform. Acta Electronica Sinica 30(5), 628–631. Xie, S., & Zhang, J. (in press). A geometric algorithm of blind source separation based on QR decomposition. Control Theory and Applications. Yeredor, A. (2000). Blind source separation via the second characteristic function. Signal Process, 80, 897–902. Received March 26, 2004; accepted July 2, 2004.

NOTE

Communicated by Barak Pearlmutter

A Further Result on the ICA One-Bit-Matching Conjecture Jinwen Ma [email protected] Department of Computer Science and Engineering, Chinese University of Hong Kong, Shatin, N.T., Hong Kong, and School of Mathematical Sciences and LMAM, Peking University, Beijing, 100871, China

Zhiyong Liu [email protected]

Lei Xu [email protected] Department of Computer Science and Engineering, Chinese University of Hong Kong, Shatin, N.T., Hong Kong

The one-bit-matching conjecture for independent component analysis (ICA) has been widely believed in the ICA community. Theoretically, it has been proved that under the assumption of zero skewness for the model probability density functions, the global maximum of a cost function derived from the typical objective function on the ICA problem with the one-bit-matching condition corresponds to a feasible solution of the ICA problem. In this note, we further prove that all the local maximums of the cost function correspond to the feasible solutions of the ICA problem in the two-source case under the same assumption. That is, as long as the one-bit-matching condition is satisfied, the two-source ICA problem can be successfully solved using any local descent algorithm of the typical objective function with the assumption of zero skewness for all the model probability density functions. The so-called one-bit-matching conjecture, which states that “all the sources can be separated as long as there is a one-to-one same-sign-correspondence between the kurtosis signs of all source probability density functions and the kurtosis signs of all model probability density functions,” was summarized in Xu, Cheung, and Amari (1998) and formally proved in Liu, Chiu, and Xu (2004) recently under the assumption of zero skewness for the model probability density functions. The one-bit-matching condition guarantees a successful solution of the ICA problem by globally maximizing the following cost function: J(R) =

n
n


r4ij νjs kim ,

(1)

i=1 j=1

Neural Computation 17, 331–334 (2005)

c 2005 Massachusetts Institute of Technology 

332

J. Ma, Z. Liu, and L. Xu

where the orthonormal matrix R = (rij )n×n = WA is to be estimated instead of W since A, i. e., the mixing matrix, is a constant one,1 νjs is the kurtosis of the source sj , and kim is a constant with the same sign as the kurtosis νim of the model probability density pi (yi ). As proved by Liu et al. (2004), under the assumption of zero skewness for the model probability density functions, this cost function is equivalent to the typical objective function of the ICA problem used by several researchers, such as Bell and Sejnowski (1995), Amari, Cichocki, and Yang (1996), and Cardoso (1999). However, in practice, typical gradient algorithms (e.g., Amari et al., 1996; Welling & Weber, 2001) cannot guarantee global optimization. Here we further analyze the local maxima of the cost function J(R) case of two  inthe n sources. For convenience, J(R) can be rewritten as ni=1 j=1 r4ij kij , where kij = νjs kim is the element of matrix K = (kij )n×n . Theorem. Assume that the one-bit-matching condition is satisfied for n = 2, the local maxima of J(R) are only the permutation matrices up to sign indeterminacy. Proof. For the second-order orthonormal matrix R, there exist two disjoint fields of R that can be expressed by  cos θ R1 = sin θ

 − sin θ , cos θ

 cos θ sin θ

 sin θ , − cos θ

and R2 =

respectively, with one variable θ ∈ [0, 2π). Actually, each R1 denotes a rotation transformation in R2 (i.e., the two-dimensional real Euclidean space), while each R2 denotes a reflection transformation in R2 . Since J(R1 ) = J(R2 ), for simplicity we consider just R1 . By differentiating J(R1 ) with respect to θ, we have dJ(R1 ) = 4 cos θ sin θ [−(k11 + k22 ) cos2 θ + (k12 + k21 ) sin2 θ ], dθ d2 J(R1 ) = 4(k11 + k22 )(3 sin2 θ cos2 θ − cos4 θ) + 4(k12 + k21 ) d2 θ × (3 sin2 θ cos2 θ − sin4 θ). 1

Note that here, we additionally assume that A is square and invertible.

(2)

(3)

One-Bit-Matching Conjecture

333

Assume that the one-bit-matching condition is satisfied for n = 2 and ν1s > ν2s and k1m > k2m ; we have the two following possibilities for the matrix K. Case I. All kij > 0, that is, both sources are subgaussians or supergaussians. The necessary condition for local maxima is obtained by setting equation 2 = 0, and we have sin θ cos θ = 0 k11 + k22 tan2 θ = . k12 + k21

(4) (5)

The roots for equation 5 correspond to local minima. Solving equation 4, we ˆ have θˆ = 0, π2 , π, 3π 2 which are local maxima as equation 3 < 0 at θ = θ. It can be readily verified that each local maximum θˆ leads R1 to a permutation matrix up to sign indeterminacy in this case. Case II. k11 , k22 > 0 while k12 , k21 < 0, that is, a mixture that consists of one subgaussian and one supergaussian source. In this case, no real roots exist for equation 5. Among the roots θˆ = 0, π2 , π, 3π 2 , only 0 and π are local maxima as equation 3 < 0 there. The remaining two roots correspond to local minima. Similarly, it can be readily verified that each local maximum leads R1 to a permutation matrix up to sign indeterminacy in this case. Summing the results of cases I and II, the proof is completed. According to this theorem, when the skewness of each model probability density function is set to be zero, the two-source ICA problem can be successfully solved using any local descent algorithm of the typical objective function as long as the one-bit-matching condition is satisfied. Acknowledgments This work was supported by a grant from the Research Grant Council of the Hong Kong SAR (Project CUHK 4184/03E), and also by the Natural Science Foundation of China for Project 60071004. References Amari, S. I., Cichocki, A., & Yang, H. (1996). A new learning algorithm for blind separation of sources. In D. Touretzky, M. Mozer, & M. Hasselmo (Eds.), Advances in Neural Information Processing, 8 (pp. 757–763). Cambridge, MA: MIT Press. Bell A., & Sejnowski, T. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7, 1129–1159.

334

J. Ma, Z. Liu, and L. Xu

Cardoso, J. F. (1999). Informax and maximum likelihood for source separation. IEEE Signal Processing Letters, 4, 112–114. Liu, Z. Y., Chiu, K. C., & Xu, L. (2004). One-bit-matching conjecture for independent component analysis. Neural Computation, 16, 383–399. Welling, M., & Weber, M. (2001). A constrained EM algorithm for independent component analysis. Neural Computation, 13, 677–689. Xu, L., Cheung, C. C., & Amari, S. I. (1998). Further results on nonlinearity and separation capability of a linear mixture ICA method and learned LPM. In C. Fyfe (Ed.), Proceedings of the I&ANN’98 (pp. 39–45). Tenerife, Spain: NAISO Academic Press. Received March 24, 2004; accepted June 22, 2004

LETTER

Communicated by Andrew Barto

Robust Reinforcement Learning Jun Morimoto [email protected] Computational Brain Project, ICORP, JST, Sora Ku-gun, Kyoto, 619-0288, Japan; ATR Computational Neuroscience Laboratories, Soraku-gun, Kyoto 619-0288, Japan

Kenji Doya [email protected] ATR Computational Neuroscience Laboratories, Soraku-gun, Kyoto 619-0288, Japan; Initial Research Project, OIST, Gushikawa, Okinawa, 904-2234, Japan; and CREST, JST, Soraku-gun, Kyoto 619-0288, Japan

This letter proposes a new reinforcement learning (RL) paradigm that explicitly takes into account input disturbance as well as modeling errors. The use of environmental models in RL is quite popular for both offline learning using simulations and for online action planning. However, the difference between the model and the real environment can lead to unpredictable, and often unwanted, results. Based on the theory of H∞ control, we consider a differential game in which a “disturbing” agent tries to make the worst possible disturbance while a “control” agent tries to make the best control input. The problem is formulated as finding a min-max solution of a value function that takes into account the amount of the reward and the norm of the disturbance. We derive online learning algorithms for estimating the value function and for calculating the worst disturbance and the best control in reference to the value function. We tested the paradigm, which we call robust reinforcement learning (RRL), on the control task of an inverted pendulum. In the linear domain, the policy and the value function learned by online algorithms coincided with those derived analytically by the linear H∞ control theory. For a fully nonlinear swing-up task, RRL achieved robust performance with changes in the pendulum weight and friction, while a standard reinforcement learning algorithm could not deal with these changes. We also applied RRL to the cart-pole swing-up task, and a robust swing-up policy was acquired. 1 Introduction In this letter, we propose a new reinforcement learning paradigm that we call robust reinforcement learning (RRL). Plain, model-free reinforcement learning (RL) requires a significant number of trials to learn policies for given tasks, and therefore it is difficult to use for online learning of realNeural Computation 17, 335–359 (2005)

c 2005 Massachusetts Institute of Technology 

336

J. Morimoto and K. Doya

world problems. Thus, the use of environmental models has been quite common for both online action planning (Doya, 2000) and off-line learning by simulation (Morimoto & Doya, 2000). However, no model is perfect, and modeling errors can cause unpredictable results—sometimes worse than with no model at all. In fact, robustness against model uncertainty has been the main subject of research in the control community for the past 20 years, and the result is formalized as the H∞ control theory (Zhou, Doyle & Glover, 1996). In general, a modeling error causes a deviation of the real system state from the state predicted by the model. This can be reinterpreted as a disturbance to the model. However, the problem is that the disturbances due to modeling error can be strongly correlated, and thus standard independence assumptions may not be valid. The basic strategy to achieve robustness is to keep the sensitivity γ of the feedback control loop against a disturbance input small enough so that any disturbance due to modeling error can be suppressed if the gain of mapping from the state error to the disturbance is bounded by 1/γ . In the H∞ paradigm, those disturbance-to-error and error-to-disturbance gains are measured by the max norms of the functional mappings in order to ensure stability for any mode of disturbance. In section 2, we briefly introduce the H∞ paradigm and show that design of a robust controller can be achieved by finding a min-max solution of a value function, which is given by the Hamilton-Jacobi-Isaacs (HJI) equation. In section 3, we formulate robust reinforcement learning. We first define an augmented value function that takes into account the amplitude of the disturbance and derive a corresponding HJI equation. We then derive an online algorithm for estimating the value function, the worst-case disturbances, and the optimal controller. In section 4, we test the validity of the algorithms first in a linear inverted pendulum task. It is verified that the value function as well as the disturbance and control policies derived by the online algorithms coincide with the analytical solution given by H∞ theory. We then compare the performance of the robust reinforcement learning algorithm with standard model-based RL in the nonlinear task of pendulum swing-up (Doya, 2000). It is shown that a robust RL controller can accommodate changes in physical parameters of the pendulum that a standard RL controller cannot cope with (Morimoto & Doya, 2001b). We also applied robust RL to the cart pole swing-up task (Doya, 2000) and tested the robustness of the learned controller. 2 H∞ Control Standard H∞ control (Zhou et al., 1996) deals with a system shown in Figure 1, where G is the plant, K is the controller, u is the control input, y is the measurement available to the controller, w is an unknown disturbance, and z is the error output that is desired to be kept small. In general, the controller K is designed to stabilize the closed-loop system based on a model of the plant

Robust Reinforcement Learning

w

G

u

337

z y

w

z G

u

y

K

K

(A)

(B)

Figure 1: (A) Generalized plant G and controller K. (B) Small gain theorem.

G. However, when there is a discrepancy between the model and the actual plant dynamics, the feedback loop could be unstable. The effect of modeling error can be equivalently represented as a disturbance w generated by an unknown mapping  of the plant output z, as shown in Figure 1b. The goal of the H∞ control problem is to design a controller K that brings the error z to zero while minimizing the H∞ norm of the closed-loop transfer function Tzw from the disturbance w to the output z z2 , Tzw ∞ = sup w 2 w

(2.1)

where  • 2 denotes L2 norm. The small gain theorem ensures that if Tzw ∞ ≤ γ , then the system shown in Figure 1b will be stable for any stable mapping : z → w with ∞ < γ1 (Zhou et al., 1996). 2.1 Min-Max Solution to H∞ Problem. We consider the plant G with dynamics given by x˙ = f (x, u, w),

(2.2)

where x ∈ X ⊂ Rn is the state, u ∈ U ⊂ Rm is the control input, and w ∈ W ⊂ Rl is the disturbance input. From equation 2.1 and the small gain theorem, the H∞ control problem can be considered as the problem to find a controller that satisfies a constraint z2 2 Tzw ∞ 2 = sup ≤ γ 2, 2 w w 2

(2.3)

where z is the error output. In other words, because the norm z2 and the norm w2 are defined as
∞ z2 2 = zT (t)z(t) dt (2.4) 0
∞ w2 2 = wT (t)w(t) dt, (2.5) 0

338

J. Morimoto and K. Doya

the H∞ control problem is equivalent to finding a control input u that satisfies a constraint
∞ V= (zT (t)z(t) − γ 2 wT (t)w(t)) dt ≤ 0 (2.6) 0

against all possible disturbances w with x(0) = 0. We can consider this problem as a differential game (Weiland, 1989) in which the best control output u that minimizes V is sought while the worst disturbance w that maximizes V is chosen. Thus, an optimal value function V ∗ is defined as
∞ ∗ V = min max (zT (t)z(t) − γ 2 wT (t)w(t)) dt. (2.7) u

w

0

The condition for the optimal value function is given by   ∂V ∗ T 2 T f (x, u, w) , 0 = min max z z − γ w w + u w ∂x

(2.8)

which is known as the Hamilton-Jacobi-Isaacs (HJI) equation. From equation 2.8, we can derive the optimal control output u and the worst disturbance w by solving ∂zT z ∂V ∗ ∂ f (x, u, w) + =0 ∂u ∂x ∂u ∗ ∂V ∂ f (x, u, w) −2γ 2 w + = 0. ∂x ∂w

(2.9) (2.10)

3 Robust Reinforcement Learning Here we consider a continuous-time formulation of reinforcement learning (Doya, 2000) with the system dynamics x˙ = f (x, u, w),

(3.1)

and the reward r(x, u), where x ∈ X ⊂ Rn is the state, u ∈ U ⊂ Rm is the control output, and w ∈ W ⊂ Rl is the disturbance input. The basic goal is to find a policy u = g(x) that maximizes the cumulative future reward,


∞

e−

s−t τ

r(x(s), u(s)) ds,

(3.2)

t

for any given state x(t), where τ is a time constant of evaluation. However, a particular policy that was optimized for a certain environment may perform

Robust Reinforcement Learning

339

badly when the environmental setting changes. In order to ensure robust performance under a changing environment or unknown disturbance, we introduce the notion of worst disturbance in H∞ control to the reinforcement learning paradigm. In this framework, we consider an augmented reward, q(t) = r(x(t), u(t)) + ω(w(t)),

(3.3)

where ω(w(t)) is an additional reward for withstanding a disturbing input; for example, we can use quadratic cost for the disturbance input to be consistent with equation 2.7, ω(w) = γ 2 wT w,

(3.4)

where γ is the robustness parameter. The augmented value function is then defined as


∞

V(x(t)) =

e−

s−t τ

q(x(s), u(s), w(s)) ds.

(3.5)

t

The optimal value function is given by the solution of a variant of HJI equation,   1 ∗ ∂V ∗ V (x) = max min r(x, u) + ω(w) + f (x, u, w) . u w τ ∂x

(3.6)

In the RRL paradigm, the value function is updated by using the temporal difference (TD) error (Doya, 2000), δ(t) = q(t) −

1 ˙ V(t) + V(t), τ

(3.7)

while the best action and the worst disturbance are generated by maximizing and minimizing, respectively, the right-hand side of the HJI equation, r(x, u) + ω(w) +

∂V ∗ f (x, u, w). ∂x

(3.8)

We use a function approximator to implement the value function V(x(t); v), where v = (v1 , . . . , vn ) is a parameter vector. As in the standard continuous-time RL, we use the learning rule for the value function appoximator as v˙ i = ηδ(t)ei (t),

(3.9)

340

J. Morimoto and K. Doya

J

K



L



N

L





P

Q

"





%



N

S

T

P

P

U

L

V

W

N

L

X

F

@

7

7

I

1



2

3

@

5

7

B

9

;

7

;

=

?































(

@

D

7







Y

A

0





R

0

E



*







,



(

 





































*







,










"









Figure 2: Actor-disturber-critic architecture.

where η denotes the learning rate and ei denotes eligibility trace for a parameter vi (Doya, 2000). The eligibility trace is updated by 1 ∂V(t) e˙i (t) = − ei (t) + , κ ∂vi

(3.10)

By comparing equations 2.8 and 3.6, we can see that the output error zT z in the H∞ framework is generalized as an arbitrary reward function r(x, u) in the RRL. Furthermore, the sign of the value function is flipped, and a discount factor is introduced in the RRL framework. 3.1 Model-Free Policy: Actor-Disturber-Critic. To implement robust RL in a model-free fashion, we propose the actor-disturber-critic architecture, shown in Figure 2. We define the policies of the actor and the disturber as u(t) = su (Au (x(t); vu ) + nu (t))

(3.11)

w(t) = sw (Aw (x(t); vw ) + nw (t)),

(3.12)

and

respectively, where Au (x(t); vu ) ∈ Rm and Aw (x(t); vw ) ∈ Rl are function approximators with parameter vectors vu and vw , and nu (t) ∈ Rm and nw (t) ∈ Rl are noise terms for exploration. su () and sw () are monotonically increasing output functions. For example, we can use sigmoidal functions for su () and sw () to saturate control output and disturbance input. In the actor-disturber-critic architecture, policies of the actor and disturber are

Robust Reinforcement Learning

341

represented by the parameter vectors vu and vw . The parameters of the actor and the disturber are updated by v˙ ui = ηu δ(t)nui (t)

∂Au (x(t); vu ) ∂vui

w w v˙ w i = −η δ(t)ni (t)

(3.13)

∂Aw (x(t); vw ) , ∂vw i

(3.14)

where ηu and ηw denote the learning rates. 3.2 Model-Based Policy: Using Value Gradient. When the augmented reward function q(x, u, w) in equation (3.3) is convex with respect to the action u and disturbance w, the HJI equation, 3.6, has a unique solution. Here, we assume that the augmented reward q(x, u, w) can be separated into three parts: the reward for the state L(x), the cost for the action S(u), and the cost for the disturbance (w). We specifically consider the case q(x, u, w) = L(x) −

m  i=1

Si (ui ) +

l 

j (wj ),

(3.15)

j=1

where Si () is a convex cost function for action variable ui and j () is a convex cost function for disturbance variable wj . In this case, the condition for the optimal action and the worst disturbance is given from the HJI equation, 3.6, as −Si (ui ) +

∂V(x) ∂ f (x, u, w) =0 ∂x ∂ui

(i = 1, . . . , m)

(3.16)

j (wj ) +

∂V(x) ∂ f (x, u, w) =0 ∂x ∂wj

(j = 1, . . . , l),

(3.17)

∂ f (x,u,w) are the ith and jth column vector of the n × m ∂wj ∂ f (x,u,w) ∂ f (x,u,w) input gain matrix and the n × l disturbance gain matrix , ∂u ∂w ∂ f (x,u,w) ∂ f (x,u,w) and ∂wj are respectively. We now assume that the input gains ∂ui

where

∂ f (x,u,w) ∂ui

and

not dependent on u and w; that is, the system is input-affine. Then the above equations have unique solutions,  ∂V(x) ∂ f (x, u, w) ∂x ∂ui   ∂V(x) ∂ f (x, u, w)  wj = j−1 − , ∂x ∂wj 

ui = Si−1



(3.18) (3.19)

342

J. Morimoto and K. Doya

where Si () and j () are monotonic functions. Accordingly, greedy policies for action and disturbance are represented in vector notation as 

 ∂ f (x, u, w)T ∂V(x)T ∂u ∂x   ∂ f (x, u, w)T ∂V(x)T  w =  −1 − , ∂w ∂x 

u = S −1

where

∂V(x) T ∂x

(3.20) (3.21)

represents the steepest ascent direction of the value func-

tion, which is then transformed by the “transpose” models ∂ f (x,u,w) T ∂w

∂ f (x,u,w) T ∂u

and

into a direction in the control output and disturbance input space, respectively (Doya, 2000). 3.3 Linear-Quadratic Case. We consider a special case in which a linear dynamic model and quadratic reward models are given or learned as x˙ = Ax + B1 u + B2 w q(x, u, w) = −xT Qx − uT Ru + γ 2 wT w.

(3.22) (3.23)

In this case, the value function is given by a quadratic form V(x) = −xT Px, where P is the solution of a Riccati equation:  AT P + PA + P ∂f

 1 1 T −1 T B B − B R B 1 1 2 2 P + Q = P. γ2 τ

(3.24)

∂f

T From ∂u = B1 , ∂w = B2 , S (u) = 2Ru,  (w) = 2γ 2 w, and ∂V ∂x = −2x P, the best action u in equation 3.20 and the worst disturbance w in equation 3.21 can be derived as

u = R−1 BT1 Px 1 w = − 2 BT2 Px. γ

(3.25) (3.26)

4 Simulation In this section, we show the performance of robust RL. In section 4.1, we consider the control problem of a linear inverted pendulum (see Figure 10 and appendix B) and test whether the parameters of the linear controller acquired by our robust RL algorithm converge to the analytical H∞ solution. In section 4.2, we apply robust RL to a nonlinear pendulum swing-up task (see appendix A) and compare the robustness of the controllers learned by robust RL and standard RL. We also test how the performance of the robust

Robust Reinforcement Learning

343

RL controllers change with different settings of the robustness parameter γ . In section 4.3, we apply robust RL to the cart pole swing-up task (Doya, 2000; see Figure 11 and appendix C). We show that robust RL is applicable to a challenging nonlinear control task with a higher-dimensional state space. (Refer to the appendixes for the outline of the three benchmark tasks.) Below are the function approximation methods we used for the value functions, the dynamics models, and the policies in the three tasks. Value function. We use normalized gaussian networks as the function approximator to represent the value function V(x; v) =

K 

vk bk (x),

(4.1)

k=1

where b() is the normalized gaussian basis function and v = (v1 , . . . , vk )T is the weight vector (see appendix D). In the case of linear dynamics and the quadratic reward, instead of using the normalized gaussian networks, we approximate the value function by the quadratic form so that V(x) = −xT Px,

(4.2)

where P is a symmetric matrix and the parameter of the function approximator. By using this quadratic form, we can exactly represent the actual value function. The update rules of these function approximators are given by equations 3.10 and 3.11. Dynamics model. We approximate the dynamics model without disturbance input by using the function approximator fˆ(x, u; vM ), x˙ −

∂ f (x, u, w) w  fˆ(x, u; vM ), ∂w

(4.3)

where vM is a parameter vector. Because we assume that the system is inputaffine as described in section 3.2, we can remove the disturbance term from ∂ f (x,u,w) the dynamics f (x, u, w) by subtracting ∂w w from x˙ . Then we can approximate input gain using the approximated model ˆf (x, u; vM ) as ∂ fˆ(x, u; vM ) ∂ f (x, u, w)  . ∂u ∂u

(4.4)

For linear dynamics, we use the linear approximation ˆ + Bˆ 1 u, x˙ − B2 w  Ax

(4.5)

where Aˆ and Bˆ 1 are parameter matrices. The update rule of the function approximator is given in appendix D.

344

J. Morimoto and K. Doya

Actor and disturber. Actor and disturber are represented by equations 3.12 and 3.13. Corresponding policies to the reward function (see equation A.3 in appendix A) are given by u=

1 −1 R (Au (x; vu ) + nu ), 2

(4.6)

w=

1 (Aw (x; vw ) + nw ). 2γ 2

(4.7)

We use normalized gaussian networks as the function approximator (see appendix D). The update rule of the function approximator is given in equations 3.14 and 3.15. In the linear quadratic case, the actor and the disturber in equations 3.12 and 3.13 are represented as linear controllers, Au (x; vu ) = vTu x and Aw (x; vw ) = vw x, respectively, where vu ∈ Rm and vw ∈ Rl . The output functions su () and sw () are identities. Value-gradient-based policies. Value-gradient-based policies for action and disturbance are given in equations 3.20 and 3.21. For the pendulum swing-up task, we use policies that correspond to the reward function (see equation A.3), 1 −1 ∂ f (x, u, w) T ∂V T R , 2 u ∂x 1 ∂ f (x, u, w) T ∂V T w=− 2 . 2γ w ∂x u=

(4.8) (4.9)

For the cart pole swing-up task, we use value-gradient-based policies which correspond to the reward function (see equation C.7),   1 ∂ f (x, u, w) T ∂V(x) T max u=u s , (4.10) c ∂u ∂x   1 ∂ f (x, u, w) T ∂V(x) T max (j = 1, 2). (4.11) wj = wj s − dj ∂wj ∂x Exploration strategy. In order to promote exploration, we incorporated a noise term in both policies, in equations 3.12 and 3.13 and equations 3.21 and 3.22. We used low-pass filtered noise τn n˙ u (t) = −nu (t) + σu Nu (t), and τn n˙ w (t) = −nw (t)+σw Nw (t), where Nu (t) and Nw (t) denote normal gaussian noise (Doya, 2000). Simulation setup. The physical systems were simulated by the fourthorder Runge-Kutta method, and the learning dynamics was simulated by the Euler method, both with the time step of 0.01 sec. Because calculating learning dynamics requires more computational resources and less accuracy than the physical dynamics of the pendulum and the cart pole, we used the faster numerical integration method (the Euler method) for the learning dynamics.

Robust Reinforcement Learning

345

4.1 Linear Inverted Pendulum. We first considered a linear problem in order to test if the value function and the policy learned by robust RL coincide with the analytic solution of H∞ control problem. Thus, we considered only the pendulum dynamics near the unstable equilibrium point x = (0, 0)T . The parameters for the value function in equation 4.2 were given as  P=

p11 p12

 p12 . p22

(4.12)

Each trial was started from an initial state x(0) = (θ (0), θ˙ (0)), where θ (0) and θ˙ (0) were selected from a uniform distribution that ranged over − π6 < ˙ θ (0) < π6 and − π6 < θ(0) < π6 , respectively. We terminated a trial after 2 seconds, or when the state of the pendulum reached |θ | > π6 . Parameters for the learning process were given as η = 1.0, ηu = 0.1, ηw = 0.1, τ = 1.0, κ = 0.1, R = 1.0, γ = 1.5, τn = 0.02, σu = 3.0, and σw = 2.0. 4.1.1 Actor-Disturber-Critic. Here, we used robust RL implemented by the actor-disturber-critic shown in equations 4.6 and 4.7. We initialized the parameters of the actor vu as vu = {−10.0, −1.0} and the disturber vw = {0.0, 0.0}, which made the system stable. The parameters of the value function P in equation 4.12 were initialized with {p11 , p12 , p22 } = {−1.0, −10.0, −1.0}. The initial parameters of the value function are consistent with the initial parameters of the actor (i.e., vu = R−1 BT1 P). Figures 3A and 3B show learning performance of the actor-disturbercritic architecture. The parameters of the actor vu and the disturber vw converged to the values close to those of policies in equations 3.25 and 3.26 derived by solving the Riccati equation, 3.25. 4.1.2 Value-Gradient-Based Robust Policy. Value-gradient-based policies for action and disturbance are given based on equations 3.25 and 3.26. The parameter matrix P was learned instead of using the solution of the Riccati equation, 3.25. Again, the parameters of the value function in equation 4.12 were initialized with {p11 , p12 , p22 } = {−1.0, −10.0, −1.0}. The parameter matrices Aˆ and Bˆ 1 for the dynamics model are initialized as Aˆ = 0 and Bˆ 1 = 0. Results in Figure 3C show that the parameters of the value function P nearly converged to the solution of the Riccati equation, 3.25. As shown in Figure 3D, the model parameter Bˆ 1 = (b1 , b2 )T used in the controller, equation 3.26, also converged to the exact model in equation A.2. 4.2 Nonlinear Pendulum Swing-Up. Now we consider the original nonlinear dynamics, equation A.2. We used the reward function defined

346

J. Morimoto and K. Doya 20

−20

p

10

v1

−40

p

0

vw

22 12

Gain

V

0

−60

2

−10 u

v2

−20

−80 −100 −120 0

w

−30

p11 200

400

600

800

u

−40 0

1000

v1 200

(A)

600

800

1000

(B) 1.2

0 −20

p

1

p

0.8

22

−40

b2

12

B_1

V

400 Trials

Trials

−60

0.6 0.4

−80

0.2

−100 −120 0

p

0

11

200

400

600 Trials

(C)

800

1000

−0.2 0

b1 50

100 Trials

150

200

(D)

Figure 3: Learning performance of (A) estimated value function for the actordisturber-critic, (B) the actor and the disturber, (C) estimated value function for the value-gradient-based robust policy, and (D) estimated dynamics model. The dash-dot lines show the exact values of the parameters.

in equation A.3 with the control cost R = 0.05. The value and policy functions were implemented by normalized gaussian networks with 21 × 21 ˙ A 4 × 4 × 2 basis network bases for each dimension of the state space {θ, θ}. ˙ u} was used for modeling the system dyfor three-dimensional space {θ, θ, namics. 4.2.1 Learning Performance. Each trial was started from an initial state x(0) = (θ(0), 0.0), where θ(0) was selected from a uniform distribution that ranged over −π < θ < π.

Robust Reinforcement Learning

347

We set the learning parameters as η = 10.0, ηu = 5.0, ηw = 5.0, τ = 1.5, κ = 0.1, and τn = 0.2. The sizes of the perturbation σu and σw were tapered off as the performance improved (Gullapalli, 1990). We took the modulation V(t)−V0 scheme σu = 2.0 min[1, max[0, VV11−V(t) −V0 ]] and σw = 1.5 min[1, max[0, V1 −V0 ]], where V0 = −2.0 and V1 = 0.0 are the minimal and maximal levels of the expected reward. We used a range of the robustness parameter γ , starting from 1.0 and gradually reducing by 0.05, and found the smallest value with which the learning method acquired the swing-up policy. As we make γ smaller, the policy becomes more robust and more conservative. For the actor-disturber-critic, we initialized the parameters of the function approximators as v = 0, vu = 0, and vw = 0. The robustness parameter was set to γ = 0.45. For the value-gradient-based robust policy, we initialized the parameter of the value function as v = 0, and the dynamics model as vM = 0. The robustness parameter was set to γ = 0.25. As suggested in Doya (2000), using the learned dynamics model and the value gradient has an advantage for learning the pendulum swing-up task. We found that the valuegradient method also has an advantage for learning a more robust policy compared to the actor-disturber-critic. Therefore, we can use a smaller robustness parameter (γ = 0.25) than that of the actor-disturber-critic method (γ = 0.45). As a measure of the swing-up performance, we defined the time for which the pendulum stayed up (|θ | < π4 ) as tup . A trial lasted for 20 seconds unless the pendulum was over-rotated (|θ | > 5π ). Upon such a failure, the trial was terminated with a reward r(t) = −2.0 for 0.5 second. We compared the learning performance of four learning schemes: actor-disturber-critic, actor-critic, value-gradient-based robust policy, and value-gradient-based standard policy. Figure 4 shows the results of learning performance. The actor-critic and the value-gradient-based standard policy learn the swing-up task faster than the actor-disturber-critic and the value-gradient-based robust policy, respectively. Results show that learning robust policies is slower than learning standard policies because the disturbance is explicitly considered during learning. 4.2.2 Value Functions and Policies. Figure 5 shows the value functions acquired by the value-gradient-based robust RL with robustness parameter γ = 0.25 and by the value-gradient-based standard RL (γ = ∞). The value function acquired by the value-gradient-based robust RL has a sharper ridge ˙ = (0.0, 0.0) and a smoother slope around the upright position, x = (θ, θ) ˙ around the bottom position, x = (θ, θ) = (π, 0.0) (see Fgiure 5A). The sharp ridge keep the pendulum at the upright position, and the smoother slope, which makes more swings around the bottom position, is suitable to cope with modeling error. Figure 6 shows the acquired actor and disturber. The

J. Morimoto and K. Doya

20

20

15

15

t_up [sec]

t_up [sec]

348

10

5

0 0

10

5

200

400

600

800

0 0

1000

200

400

Trials

800

1000

(B)

20

20

15

15

t_up [sec]

t_up [sec]

(A)

10

5

0 0

600 Trials

10

5

200

400

600 Trials

(C)

800

1000

0 0

200

400

600

800

1000

Trials

(D)

Figure 4: Comparison of the time course of learning with different control schemes: (A) actor-disturber-critic, (B) actor-critic, (C) value-gradient-based robust policy, and (D) value-gradient-based standard policy. tup : time in which the pendulum stayed up.

disturber learned the policy that mainly disturbs the actor’s policy around ˙ = (0.0, 0.0). the upright position, x = (θ, θ) 4.2.3 Robustness to Parameter Changes. In Figure 7, we compare the robustness of the value-gradient-based robust policy and the value-gradientbased standard policy to the change of physical parameters. Both policies learned to swing up and hold a pendulum that has the weight m = 1.0 kg and the coefficient of friction µ = 0.01 (see Figure 7A). The value-gradientbased robust policy took more swings, indicating its conservative control law.

349

0 −1 −2 −3

400

V

V

Robust Reinforcement Learning

200 −100

0 −1 −2 −3

0 0

400 200 −100

100 θ [deg]

−400

0 0

−200

−200 100

ω [deg/s]

θ [deg]

(A)

ω [deg/s]

−400

(B)

4 2 0 −2 −4

400

w

u

Figure 5: Shape of the value function acquired by the value-gradient-based RL after 1000 learning trials. (A) Robust RL (γ = 0.25). (B) Standard RL (γ = ∞).

1 0 −1

400

200 −100

200

0 0

−100

θ [deg]

−400

0 0

−200 100

−200 100

ω [deg/s]

θ [deg]

(A)

ω [deg/s]

−400

(B)

2.5

2.5

2

2

1.5

1.5

θ [rad/π]

θ [rad/π]

Figure 6: Shape of the control and disturbance function after 1000 learning trials. the robustness parameter was γ = 0.45 (A) Actor. (B) Disturber.

1 0.5 0 −0.5 0

1 0.5 0

Robust Standard 1

2

3 Time [sec]

(A)

4

5

6

−0.5 0

Robust Standard 5

10 Time [sec]

15

20

(B)

Figure 7: Swing-up trajectories with a pendulum with different weight and friction. The dash-dot lines show the upright position. The initial position was θ (0) = 56 π . (A) m = 1.0, µ = 0.01. (B) m = 3.0, µ = 0.5.

350

J. Morimoto and K. Doya

Table 1: Comparison with Different Robustness Parameter (Average Performance of 10 Learned Controllers). γ

0.25

0.5

1.0

∞

Max m[kg]

5.9

3.9

3.2

3.1

Next, we applied both robust and standard policies to a pendulum that has different physical parameters (m = 3.0 kg, µ = 0.5) from the originally learned environment (m = 1.0 kg, µ = 0.01). As shown in Figure 7B, the robust policy could successfully swing the pendulum up, while the standard policy could not. This result shows the robustness of the policy acquired by robust RL. We also tested how the robustness parameter γ affects robust performance of the learned controller. We compare the value-gradient-based robust policy with the three robustness parameters (γ = 0.25, 0.5, 1.0) and the value-gradient-based standard policy (γ = ∞). We trained these controllers with the pendulum mass m = 1.0 kg. Then we compared the maximum weight that each controller can successfully swing up, where the initial joint angle was θ (0) = 56 π and the coefficient of friction was the same as the originally learned environment, µ = 0.01. A trial was regarded as successful when tup > 5 seconds. We generated 10 controllers for each robustness parameter γ and then compared the average performance. The results of the comparison in Table 1 show a more robust performance with smaller γ . This results are consistent with H∞ control theory (Zhou et al., 1996). 4.3 Cart Pole Swing-Up. We compared the robust performance of the robust RL controller with the standard RL controller in the cart pole swingup task. The value and policy functions were implemented by normalized gaussian networks with 7 × 7 × 21 × 21 bases for each dimension of the state ˙ A 2 × 2 × 4 × 2 × 4 basis network for five-dimensional space space {x, v, θ, θ}. {x, v, θ, θ˙ , u} was used for modeling the system dynamics. When the cart bumped into the end of the track or when the pole overrotated (|θ| > 5π ), a terminal reward r(t) = −1.0 was given for 0.5 second. Otherwise a trial lasted for 20 seconds. We set the learning parameters as η = 20.0, τ = 1.0, κ = 0.3, ηM = 10.0, τn = V(t)−V0 0.2, σu = 2.0 min[1, max[0, VV11−V(t) −V0 ]], and σw = 2.0 min[1, max[0, V1 −V0 ]], where V0 = −1.0 and V1 = 0.0. Each trial was started from an initial state x(0) = (0.0, 0.0, θ(0), 0.0), where θ(0) were selected from a uniform distribution that ranged over −π < θ(0) < π.

Robust Reinforcement Learning

351

100

Success rate [%]

80

60

40

20 Robust RL Standard RL 0 0.1

0.15

0.2 0.25 Pole mass m_p [kg]

0.3

0.35

Figure 8: Success rate of the cart pole swing-up task (average success rate of 10 learned controllers).

Here, we compared the average swing-up performance after 1000 learning trials. We changed the pole mass from mp = 0.1 kg to mp = 0.35 kg by 0.05 kg. Each trial was started from an initial joint angle θ (0) selected from a uniform distribution that ranged over −π < θ (0) < π. Figure 8 shows the success rate of the cart pole swing-up task. The success rate was derived from the number of successful swing-up times in 100 trials. A trial was regarded as successful when tup > 5 seconds, where we defined the time for which the pole stayed up (|θ| < 12 deg) as tup . We generated 10 robust controllers (policies) and standard controllers (policies) and then compared average success rate. We used robustness parameters defined in equation C.9 in appendix C. The success rate using the robust policy with the pole mass mp = 0.35 kg was about 80%, while using standard policy was about 30%. This result shows the robustness of the acquired robust policy. Figure 9A shows that the robust policy could swing up the pole with the cart pole that has different physical parameters (pole mass mp = 0.35 kg, pole friction µp = 0.005, the cart mass mc = 1.5 kg) from the original environment (mp = 0.1 kg, µp = 2.0 × 10−6 , mc = 1.0 kg). Figure 9B shows that the standard policy failed to swing up the pole with the same cart pole used in Figure 9A. These results suggest that the robust RL can possibly be applied to a more difficult task with a higher-dimensional state space. 5 Discussion H∞ control theory gives an analytical solution only for linear systems. For nonlinear systems, there is no analytical way of solving the HJI equation.

352

J. Morimoto and K. Doya

1 0 10 −1

9 8 7

−3

6

−2

5

−1

4

0

3

1

2

2 3

1

Time [sec]

0

Position x [m]

(A)

1 0 10 −1

9 8 7

−3

6

−2

5

−1

4

0

3

1

2

2 3

1

Time [sec]

0

Position x [m]

(B) Figure 9: Cart pole swing-up trajectories with pole mass mp = 0.35 kg, pole friction µp = 0.005, and cart mass mc = 1.5 kg. The initial joint angle was θ(0) = π. (A) Value-gradiant-based robust policy. (B) Value-gradient-based standard policy.

Robust Reinforcement Learning

353

In order to derive a nonlinear H∞ controller, the value function is usually derived by using dynamic programming (Imafuku, 1999; Coraluppi & Marcus, 1999). A similar approach to the methods using dynamics programming was proposed by Nilim and Ghaoui (2004) to cope with uncertain state transition matrices. However, these methods need off-line calculation and an environmental model. Robust RL can derive a nonlinear H∞ controller by online calculation and without any prior knowledge of an environmental model. Robust RL provides a new way of using min-max solution in RL problems. Min-max RL was applied to games like backgammon (Tesauro, 1992) and Othello (Yoshioka & Ishii, 1998). Minimax Q-learning was proposed by Littman (1994, 2001) and applied to zero-sum games. However, in these studies, each player takes the same role. The min-max RL was also applied to a problem in which an airplane tries to avoid a missile and the missile tries to catch the airplane (Harmon, Baird, & Klopf, 1995). However, this study focuses on only linear control problems. We applied min-max RL in which each of two players (the control agent and the disturbing agent) has a different ability with the nonlinear problems of pendulum swing-up and cart pole swing-up. Risk-sensitive control studies are also related to our RRL framework and have an interesting application to the field of finance. Neuneirer and Mihatsch (1998) proposed risk-sensitive reinforcement learning and applied their method to acquire an optimal investment policy for an artificial stock price. Robustness during the learning process in RL framework was studied in Singh, Barto, Gruppen, and Connolly (1994) by using a constrained policy space and in Kretchmar et al. (2001) by combining standard RL with a linear robust controller. Although the target of our study is different from these studies, this kind of robustness is also important when we apply RL to real-world problems.

6 Conclusion In this study, we proposed a new RL paradigm called robust reinforcement learning (RRL). We showed that RRL can learn the analytic solution of the linear H∞ controller for the inverted pendulum dynamics around the upright position and also showed that RRL can deal with modeling error that standard RL cannot in the nonlinear inverted pendulum swing-up simulation example. We also applied RRL to the cart pole swing-up task, and a robust swing-up policy was acquired by RRL. We will apply RRL to more complex tasks like learning stand-up behaviors (Morimoto & Doya, 2000, 2001a) and biped locomotion (Atkeson & Morimoto, 2003; Morimoto & Atkeson, 2003) in future work.

354

J. Morimoto and K. Doya




Figure 10: Single-pendulum swing-up task. θ: joint angle, T: input torque, l: length of the pendulum.

Appendix A: Pendulum Swing-Up Task The task of swinging a pendulum up (Doya, 2000) is a simple but strongly nonlinear control task. The dynamics of the pendulum is given by ml2 θ¨ = −µθ˙ + mgl sin θ + T,

(A.1)

where θ is the angle from the upright position, T is the input torque, µ = 0.01 is the coefficient of friction, m = 1.0 kg is the weight of the pendulum, l = 1.0 m is the length of the pendulum, and g = 9.8 m/s2 is the gravity acceleration (see Figure 10). ˙ T and the action as u = T. We We define the state vector as x = (θ, θ) consider disturbance input w to the acceleration θ¨ . Thus the dynamics is given by an input-affine system: x˙ =

 θ˙



g µ ˙ + l sin θ − ml2 θ



0 1 ml2

 u+

  0 w. 1

(A.2)

The augmented reward function is given by q(x, u, w) = (cos θ − 1) − Ru2 + γ 2 w2 .

(A.3)

Appendix B: Linear Inverted Pendulum For comparison of robust RL with analytical linear H∞ control, we use a local linearization of the pendulum dynamics, equation A.2, near the unstable equilibrium point x = (0, 0)T . The coefficient matrices of the linear from

Robust Reinforcement Learning

355




Figure 11: Cart pole swing-up task. θ : joint angle, x: cart position F: input force.

equation 3.23 are given by  A=

0

g l

1 − mlµ2



 , B1 =

0



1 ml2

, B2 =

  0 . 1

(B.1)

The quadratic approximation of the augmented reward function, equation A.3, is given by q(t) = −xT Qx − Ru2 + γ 2 w2 ,

(B.2)

where Q = diag{0.5, 0}. Appendix C: Cart Pole Swing-Up Task The cart pole swing-up (see Figure 11; Doya, 2000) is a strongly nonlinear extension to the common cart pole balancing task (Barto, Sutton, & Anderson, 1983). The physical parameters of the cart pole were the same in Barto et al. (1983) and Doya (2000). Marked differences from the pendulum swing-up task are that the dimension of the state space is higher and that the disturbance input has two degrees of freedom. ˙ where x and v are, respectively, the The state vector is x = {x, v, θ, θ}, position and the velocity of the cart. The dynamics of the cart pole is modeled by the following nonlinear ordinary differential equations:

θ¨ = x¨ =

gsinθ +

2 ˙ cos θ(µc sign(x)−F−m p lθ˙ sin θ) mc +mp



l

4 3

−

mp cos2 θ mc +mp



−

µp θ˙ mp l

˙ F + mp l(θ˙ 2 sin θ − θ¨ cos θ) − µc sign(x) . mc + mp

,

(C.1)

(C.2)

356

J. Morimoto and K. Doya

The following specifications were used for the simulated pole and cart: the track ranged from −3.6 m to 3.6 m; mass of the cart: mc = 1.0 kg; mass of the pole: mp = 0.1 kg; half-length of the pole l = 0.5 m; gravity g = 9.8m/s2 ; coefficient of friction of cart on track: µc = 5.0 × 10−4 ; coefficient of friction of pole on cart: µp = 2.0 × 10−6 . F denotes applied force, x denotes the cart position, and θ denotes joint angle of the pole. The cart-pole dynamics can be written as an input-affine system, x˙ = f(x) + g(x)u + B2 w, where



(C.3) 

v

  mp l(θ˙ 2 sin θ −θ¨ cos θ)−µc sign(x) ˙   mc +mp    θ˙ f(x) =  , 2  ˙ cos θ(µc sign(x)−m µp θ˙  p lθ˙ sin θ) − mp l   g sin θ+ mc +mp

  2 l

(C.4)

mp cos θ 4 3 − mc +mp

 v 1  m +m  c p θ˙ g(x) =  cos θ  − mc +mp

m cos2 θ

    ,  

(C.5)

p 4 3 − mc +mp

l

and we used the input matrix for disturbance B2 :   0 0 1 0  B2 =  0 0 . 0 1

(C.6)

The augmented reward was given by q(x, u, w) = 0.5(cos θ − 1) − S(u) +

2 

j (wj ),

(C.7)

j=1

where


u

S(u) = c

s−1

0


j (wj ) = dj

a

da, umax 

wj

s 0

1

a wjmax

(C.8)

 da

(j = 1, 2),

(C.9)

Robust Reinforcement Learning

s(x) =

357

  2 π arctan x , 2 π

(C.10)

= 1.0 N, wmax = 10.0 N · m, c = 1.0, d1 = 1.0, and umax = 20.0 N, wmax 1 2 d2 = 1.0. Appendix D: Normalized Gaussian Network A value function is represented by V(x; v) =

K 

vk bk (x),

(D.1)

k=1

where the normalized gaussian basis function is represented by ak (x) bk (x) = K , l=1 al (x)

ak (x) = e−sk (x−ck ) . T

2

(D.2)

The vectors ck and sk define the center and the size of the kth basis function, respectively. Note that if there is no neighboring basis function, the shape of the basis functions extends like sigmoid functions by the effect of normalization. In the current simulations, the centers are fixed in a grid, which is analogous to the “boxes” approach (Barto et al., 1983) often used in discrete RL. Grid allocation of the basis functions enables efficient calculation of their activation as the outer product of the activation vectors for individual input variables. In the actor-disturber-critic method, the polices are implemented as    u u u(t) = su vk bk (x(t)) + n (t) , (D.3) k

w(t) = sw

 

 vw k bk (x(t))

w

+ n (t) ,

(D.4)

k

where sw () and sw () are monotonically increasing output function, and nu and nw are the noise. In the value-gradient-based methods, the control and disturbance policies are given by   ∂ f (x, u, w)T  ∂bk (x) u u(t) = su vk (D.5) + n (t) , ∂u ∂x k   ∂ f (x, u, w)T  ∂bk (x)T w w(t) = sw − + n (t) . vk (D.6) ∂w ∂x k

358

J. Morimoto and K. Doya

To implement the input gain model, a network is trained to predict the time derivative of the state from x and u, x˙ (t) −

 ∂ f (x, u, w) vkM bk (x(t), u(t)). w(t)  fˆ(x, u) = ∂w k

(D.7)

The weights are updated by v˙ kM (t) = η M

  ∂ f (x, u, w) ˆ x˙ (t) − w(t) − f (x(t), u(t)) bk (x(t), u(t)), (D.8) ∂w

where the learning rate was set to η M = 10.0 for all simulations, and the input gain of the system dynamics is given by  ∂ f (x, u)  M ∂bk (x, u)  vk   ∂u ∂u  k

.

(D.9)

u=0

Acknowledgments We thank Christopher G. Atkeson, Mitsuo Kawato, and the anonymous reviewers for their helpful comments. References Atkeson, C. G., & Morimoto, J. (2003). Nonparametric representation of policies and value functions: A trajectory-based approach. In S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in neural information processing systems, 15 (pp. 643–1650). Cambridge, MA: MIT Press. Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13, 834–846. Coraluppi, S. P., & Marcus, S. I. (1999). Risk-sensitive and minmax control of discrete-time finite-state Markov decision processes. Automatica, 35, 301–309. Doya, K. (2000). Reinforcement learning in continuous time and space. Neural Computation, 12(1), 219–245. Gullapalli, V. (1990). A stochastic reinforcement learning algorithm for learning real-valued functions. Neural Networks, 3, 671–192. Harmon, M. E., Baird III, L. C., & Klopf, A. H. (1995). Advantage updating applied to a differential game. In G. Tesauro, D.S. Touretzky, & T.K. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 353–360). Cambridge, MA: MIT Press. Imafuku, K. (1999). Singularities of nonlinear control systems designed by HamiltonJacobi equations. Unpublished doctoral dissertation, Nara Institute of Science and Technology.

Robust Reinforcement Learning

359

Kretchmar, R. M., Young, P. M., Anderson, C. W., Hittle, D. C., Anderson, M. L., & Delnero, C. C. (2001). Robust reinforcement learning control with static and dynamic stability. International Journal of Robust and Nonlinear Control, 11, 1469–1500. Littman, M. L. (1994). Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the Eleventh International Conference on Machine Learning (pp.157–163). San Francisco: Morgan Kaufmann. Littman, M. L. (2001). Value-function reinforcement learning in Markov games. Journal of Cognitive Systems Research, 2, 55–66. Morimoto, J., & Atkeson, C. G. (2003). Minimax differential dynamic programming: An application to robust biped walking. In S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in neural information processing systems, 15 (pp. 1563–1570). Cambridge, MA: MIT Press. Morimoto, J., & Doya, K. (2000). Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning. In Proceedings of Seventeenth International Conference on Machine Learning (pp. 623–630). San Francisco: Morgan Kaufmann. Morimoto, J., & Doya, K. (2001a). Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning. Robotics and Autonomous Systems, 36, 37–51. Morimoto, J., & Doya, K. (2001b). Robust reinforcement learning. In T.K. Leen, T. G., Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems, 13 (pp. 1061–1067). Cambridge, MA: MIT Press. Neuneier, R., & Mihatsch, O. (1998). Risk sensitive reinforcement learning. In M.S. Kearns, S A Solla, & D. A. Cohn (Eds.), Advances in neural information processing systems, 11 (pp. 1031–1037). Cambridge, MA: MIT Press. Nilim, A., & Ghaoui, L. E. (2004). Robustness in Markov decision problems with uncertain transition matrices. In S. Thrun, L. Saul, & B. Scholkopf, ¨ (Eds.), Advances in neural information processing systems, 16. Cambridge, MA: MIT Press. Singh, S. P., Barto, A. G., Grupen, R., & Connolly, C. (1994). Robust reinforcement learning in motion planning. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 655–662). San Francisco: Morgan Kaufmann. Tesauro, G. (1992). Practical issues in temporal difference learning. Machine Learning 8, 257–277. Weiland, S. (1989). Linear quadratic games, H∞ , and the Riccati equation. In Proceedings of the Workshop on the Riccati Equation in Control, Systems, and Signals (pp. 156–159) Como, Italy. Yoshioka, T., & Ishii, S. (1998). Strategy acquisition for the game “othello” based on reinforcement learning. In S. Usui & T. Omori (Eds.), International Conference on Neural Information Processing (pp. 841–844). London: ISO Press. Zhou, K., Doyle, J. C., & Glover, K. (1996). Robust optimal control. Englewood Cliffs, NJ: Prentice Hall. Received May 13, 2004; accepted July 2, 2004.

LETTER

Communicated by Daniel Durstewitz

A Computational Model of the Functional Role of the Ventral-Striatal D2 Receptor in the Expression of Previously Acquired Behaviors Andrew James Smith [email protected]

Suzanna Becker [email protected] Psychology Department, McMaster University, Hamilton, Ontario, Canada L8S 4K1

Shitij Kapur Shitij [email protected] Center for Addiction and Mental Health, Toronto, Ontario, Canada, M5R 1T8

The functional role of dopamine has attracted a great deal of interest ever since it was empirically discovered that dopamine-blocking drugs could be used to treat psychosis. Specifically, the D2 receptor and its expression in the ventral striatum have emerged as pivotal in our understanding of the complex role of the neuromodulator in schizophrenia, reward, and motivation. Our departure from the ubiquitous temporal difference (TD) model of dopamine neuron firing allows us to account for a range of experimental evidence suggesting that ventral striatal dopamine D2 receptor manipulation selectively modulates motivated behavior for distal versus proximal outcomes. Whether an internal model or the TD approach (or a mixture) is better suited to a comprehensive exposition of tonic and phasic dopamine will have important implications for our understanding of reward, motivation, schizophrenia, and impulsivity. We also use the model to help unite some of the leading cognitive hypotheses of dopamine function under a computational umbrella. We have used the model ourselves to stimulate and focus new rounds of experimental research. 1 Introduction Dopamine is a neuromodulator of great interest because of its central role in reward and motivation, as well as in a number of human disorders, including schizophrenia, attention deficit hyperactivity disorder (ADHD), drug addiction, and Parkinson’s disease. The precise role of dopamine in each of these processes is still a matter of debate, and a number of partially overlapping hypotheses exist. With a view to formalizing and uniting some of these hypotheses, we present a novel computational model of dopamine function that offers a consistent account of some apparently diverse results Neural Computation 17, 361–395 (2005)

c 2005 Massachusetts Institute of Technology


362

A. Smith, S. Becker, and S. Kapur

from the animal behavior literature. In particular, we concentrate on the effect of dopamine manipulation on the expression of previously acquired behaviors. There are several competing hypotheses of dopamine function, which we briefly review. Some are computational and others cognitive, with the computational approaches tending to be oriented around the phasic dopamine response and its role in learning, and the cognitive hypotheses based on behavioral data and possibly pertaining more to the tonic or constant background signal. The anhedonia hypothesis (Wise, 1982; Wise, Spindler, DeWit, & Gerber 1978) suggested that dopamine mediates the hedonia associated with rewarding environmental stimuli. Evidence was drawn from animal experiments in which neuroleptics (dopamine-blocking drugs) caused extinctionlike effects. Extinction refers to the slow disappearance of a conditioned behavior if the resulting reward is not forthcoming. Recent data have eroded the anhedonia hypothesis and supported a more subtle role for dopamine in reward, motivation, and salience. For example, Berridge and Robinson (2003) suggest that reward is a multidimensional construct that can be decomposed into (among others) liking (hedonic) and wanting (motivational) components and that dopamine selectively mediates the latter. This position has gained widespread recognition as the incentive salience hypothesis. Evidence is derived from an extreme experiment in which rats that are deprived of almost all dopamine in the ventral and neostriatum simply stop eating, even though they apparently maintain the motoric capability to do so and even though their life depends on the food that is right under their noses! An analysis of affective responses when artificially fed reveals that the hedonic impact of the food (”liking”) is apparently unaffected (Berridge & Robinson, 1998). The incentive salience hypothesis is not computational in nature, although McClure, Daw, and Montague (2003) have developed a computational instantiation to account for some basic experimental data. Salamone, Cousins, and Snyder (1997) suggest that “dopamine in the nucleus accumbens [part of the ventral striatum] is important for responding to conditioned stimuli and . . . to stimuli that are spatially and temporally distant from the organism” (p. 353). General support for the claim that dopamine is implicated in responding for rewards (or avoiding punishments) that are in some way distal from the organism is provided by a wide range of experiments pertaining to conditioned avoidance (Anisman, Irwin, Zacharko, & Tombaugh 1982; Beninger & Hahn, 1983; Blackburn & Phillips, 1989; Courvoisier, 1956a; Grilly, Johnson, Minardo, Jacoby, & LaRiccia, 1984; van der Heyden & Bradford, 1988; Maffii, 1959; Wadenberg, Soliman, Vanderspek, & Kapur, 2001; Stark, Bischof, & Scheich, 1999; Wilkinson et al., 1998), animal (Richards, Sabol, & de Wit, 1999; Wade, de Wit, & Richards, 2000; Cousins, Atherton, Turner, & Salamone, 1996; Salamone et al., 1991; Salamone, Cousins, & Bucher, 1994) and human (de Wit, Enggasser, & Richards, 2002) models of impulsivity, aphagia (Berridge &

Modeling the Functional Role of Dopamine in the Ventral Striatum

363

Robinson, 1998), and instrumental responding for food (Dickinson, Smith, & Mirenowicz, 2000; Evenden & Robbins, 1983; Fowler, LaCerra, & Ettenberg, 1986; Rolls et al., 1974; Wise & Schwartz, 1981; Wise et al., 1978), drugs (see Wise, 2002, for a review), conditioned reinforcers (Taylor & Robbins, 1984), and electrical brain stimulation (Ettenberg, 1989; Fibiger, Carter, & Phillips, 1976; Rolls et al., 1974; Salamone, Kurth, McCullough, Sokolowski, & Cousins, 1993). Those studies that directly target the ventral striatum suggest that this is the site where this particular effect of dopamine manipulation is occurring (Berridge & Robinson, 1998; Cardinal, Pennicott, Sugathapala, Robbins, & Everitt, 2001; Salamone et al., 1991, 1993, 1994; Salamone, Wisniecki, Carlson, & Correa, 2001). In contrast to these motivational perspectives, Horvitz (2002) suggests a more attentional role for dopamine in the gating of sensory, reward, and motor processes. Also within this attentional category, Redgrave, Prescott, and Gurney (1999) propose that dopamine plays a role in switching between different behaviors. For example, Phillips, Stuber, Helen, Wrightman, and Carelli (2003) report that rats could be made to stop what they were doing and press a lever for cocaine (a preconditioned behavior) simply by artificially stimulating dopamine release. From a computational perspective, a highly influential model is the prediction error hypothesis of dopamine function (Hollerman & Schultz, 1998; Schultz, Dayan, & Montague, 1997; Montague, Dayan, & Sejnowski, 1996; Houk, Adams, & Barto, 1995). Electrophysiological recordings from the primate midbrain suggest that the dopaminergic signal is somewhat analogous to the prediction error signal used to drive learning in the temporal difference (TD) learning algorithm (for TD, see Sutton, 1988; Sutton & Barto, 1998). Apart from being able to account for a range of data pertaining to the phasic firing of dopamine neurons, one of the strengths of this hypothesis is that it posits both a cause (error in predicted reward) and effect (update of reward prediction) of dopamine neuron firing within a concrete computational framework. However, a number of criticisms have been discussed (Berridge & Robinson, 1998; Horvitz, 2000, 2002; Redgrave et al., 1999). One problem for the prediction error hypothesis is that dopamine is released not just following rewarding stimuli and stimuli that predict reward, but also following novel stimuli, aversive stimuli, and even stimuli that predict aversive events (for reviews, see Ikemoto & Panksepp, 1999; Horvitz, 2000, 2002; Salamone et al., 1997; Joseph, Datla, & Young, 2003). Another problem is that dopamine neurons produce not only intermittent burst or phasic firing, but also a constant background tonic firing. Recently Fiorillo, Tobler, and Schultz (2003) have also found an intermediate sustained firing between a stimulus and a subsequent reward. A more general role for dopamine is therefore suggested. However, perhaps the major consideration is that the prediction error hypothesis primarily posits a role for dopamine in learning (i.e., the first derivative of behavior), and yet there is compelling evidence that dopamine

364

A. Smith, S. Becker, and S. Kapur

is also required for the expression (’‘zeroth derivative”) of previously acquired behaviors (Cousins et al., 1996; Rolls et al., 1974; Maffii, 1959; Berridge & Robinson, 1998; Wade et al., 2000; Richards et al., 1999; Wadenberg et al., 2001). Moreover, following dopamine manipulation, behaviors appear to be affected differently depending on their relationship to the rewarding (or punishing) outcome. Although Montague, Dayan, Person, and Sejnowski (1995), Montague et al., (1996), and McClure et al. (2003) have extended the TD model to include a role for dopamine in biasing action selection, this approach has not yet been used to account for the selectivity of dopamine manipulation on distal versus proximal rewards. It is this selectivity that is the focus of this letter. A brief note on the motivation of this work is now offered. One of our long-term goals is a better computational understanding of schizophrenia, and in particular psychosis. Since the ventral striatum and the dopamine D2-receptor subtype have been strongly implicated in the disorder, the first step was to look at behavioral studies that pertain to either the ventral striatum or the specific D2-receptor manipulation, or, where possible, both. The aim of this work is not to model the striatal D2 receptor at an anatomical or physiological level, but rather to induce its functional significance at the behavioral level based on the studies referred to above. Since TD methods provide the currently preeminent computational account of phasic dopamine, our first inclination was to adopt and adapt this approach. However, we were unsuccessful in matching the data to TD and will therefore outline an alternative internal model account of motivated behavior. That is not to say that TD cannot address the data considered below, but rather that we were unable to use it to do so in a parsimonious fashion. The internal model approach described below and TD use very different representational techniques with far-reaching implications, and it is important to know which forms the sounder basis for understanding schizophrenia. Behavioral data provide our current constraint, but future work must draw on additional constraints (including physiological and electrophysiological considerations) to resolve the issue to the satisfaction of behavioral, psychiatric, and computational communities.

2 Schizophrenia, Psychosis, and Conditioned Avoidance Schizophrenia occurs with a global incidence of around 1% and is considered one of the most debilitating human disorders (Kandel, Schwartz, & Jessell, 1991). The symptoms are broadly divided into two categories: the positive and the negative. The negative symptoms, characterized by a lack of motivation, flat affect, anhedonia, eccentric behavior, social isolation, poverty of speech, and a poor attention span, are chronic and effectively untreatable by pharmacological intervention. In contrast, the positive symptoms or psychosis, which consist of delusions, disordered thoughts,

Modeling the Functional Role of Dopamine in the Ventral Striatum

365

and hallucinations, often occur in acute phases and may be mitigated or prevented with antipsychotic drugs (APDs). Although psychosis is a major component of schizophrenia, it may also occur in nonschizophrenic individuals. For insight into the nature of delusions and hallucinations, see Maher and Ross (1984) and Beck and Rector (2003), respectively. All current APDs block dopamine; moreover, there is a striking correlation between the ability of these drugs to block the dopamine D2 receptor and the dose required to mitigate psychosis (Kandel et al., 1991). Interestingly, all drugs of abuse cause an increase in dopamine in the nucleus accumbens (for reviews, see DiChiara, 1999; Kauer, 2003; Ikemoto & Panksepp, 1999), and some of these (particularly cocaine and amphetamine) can also induce psychotic symptoms in users (Bell, 1973; Connell, 1958). The number of D2 receptors is increased in the striatum of (unmedicated) schizophrenia patients in postmortem examination, an effect that is particularly pronounced in patients with positive symptoms (Kandel et al., 1991, chap. 55). The nucleus accumbens in particular may be important as a convergence site for a number of brain regions that are implicated in schizophrenia, including PFC, amygdala, and hippocampus (Grace, 2000), not to mention dopamine projections of the VTA. (See also Grace, 1991, for discussion.) These, along with other data, have led to the hypothesis that psychosis is mediated, if not caused, by an excess of dopamine in the limbic system, of which the ventral striatum is a key component (Kandel et al., 1991, chap. 55). An important preclinical drug test for potential antipsychotic efficacy is a well-established animal experimental paradigm called conditioned avoidance (CA) (Kilts, 2001; Arnt, 1982; Janssen, Niemegeers, & Schellekens, 1965; Wadenberg & Hicks, 1999). The standard CA experiment finds that if a neutral stimulus, such as an auditory tone (the conditioned stimulus or CS), regularly precedes an electric shock (unconditioned stimulus, or US), an animal will learn to avoid the shock by taking appropriate evasive action in response to the tone (Kamin, 1954; Low & Low, 1962; Black, 1963). Appropriate evasive action often involves the animal’s running or jumping to another compartment in the cage. An avoidance response is recorded if the animal runs during the tone, and an escape response is recorded if the animal waits until the arrival of the shock. A failure is recorded if the animal fails to run even when shocked. It is well established that low (noncataleptic) doses of all APDs, administered after the avoidance behavior has been acquired, selectively disrupt that avoidance response yet leave the escape response intact (Ader & Clink, 1957; Arnt, 1982; Cook & Weidley, 1957; Cook & Catania, 1964; Courvoisier, 1956b; Davidson & Weidley, 1976; Ponsluns, 1962). As the drug wears off, the avoidance response is restored (Wadenberg et al., 2001; Smith, Li, Becker, & Kapur, 2004). It seems unlikely that these effects are due to the interaction of the drug with the learning processes of the animal because of the differences in the timescales involved. For example, a rat typically requires many trials over many days to acquire the avoidance response. If an APD is

366

A. Smith, S. Becker, and S. Kapur

then administered and the rat is tested 20 minutes later (allowing the drug to take effect), the rat will stop running in response to the tone almost immediately. Similarly, when the rat is tested the following day drug free, the avoidance response will reappear almost immediately. Also, in untreated rats, the extinction of the avoidance response when the shock is actually discontinued tends to be a relatively slow process (Kamin, 1954). Therefore, the immediate impact of the drug is again striking. However APDs do also retard acquisition of the conditioned response, and dissociating the role of dopamine in performance and learning is not straightforward (Beninger, 1989). The degree of APD-induced avoidance disruption has been correlated with D2-receptor blockade, leading to the suggestion that blockade of this dopamine receptor is the neurochemical link between conditioned avoidance disruption in rats and antipsychotic action in people (Wadenberg et al., 2001). Importantly, conditioned avoidance can also be disrupted by direct intra-accumbens injections of D2-receptor antagonists (blocking D2 receptors) and by accumbens 6-OHDA lesions, destroying the dopaminereleasing neurons themselves (see Ikemoto & Panksepp, 1999, for a review). However, the common behavioral or psychological processes are currently unknown. For example, existing hypotheses of why APDs disrupt avoidance include the inhibition of an internal “fear” or “anxiety” (Cook & Weidley, 1957; Miller, Murphy, & Mirsky, 1957; Davis, Capehart, & Llewellin, 1961; Hunt, 1956), motor impairment (Ponsluns, 1962; Cook & Catania, 1964; Morpurgo, 1965; Beninger, Mason, Phillips, & Fibiger, 1980a, 1980b; Grilly et al., 1984; Aguilar, Mari-Sanmillan, Mortant-Deusa, & Minarro, 2000; Ogren & Archer, 1994), reduced responsiveness to external stimuli (Dews & Morse, 1961), decrease in sensory stimulation (Irwin, 1958; Key, 1961), and loss of attention or arousal (Low, Eliasson, & Kornetsky, 1966). However, following the incentive salience hypothesis of dopamine function, we will propose a motivational explanation that can subsequently be used to account for additional data from other experimental paradigms.

3 The Model Our model is based on the assumption that an animal builds an explicit internal model of its environment. Internal models have played a significant role in the methodologies of a number of fields, including AI. Indeed, they have been used not only within formal reinforcement learning (reviewed in Sutton & Barto, 1998), but also to model animal conditioning (Schmajuk, 1988) and the dopamine system (Schmajuk, Cox, & Gray, 2001; Suri, Bargas, & Arbib, 2001; Suri, 2001). Although the types of representation used by animals are many and varied (Balleine, Garner, Gonzalez, &

Modeling the Functional Role of Dopamine in the Ventral Striatum

367

Dickinson, 1995; Berridge & Robinson, 1998; Cardinal, Parkinson, Hall, & Everitt, 2002; Dickinson, 1980), the evidence that animals internally represent action-outcome relationships (among others) is compelling (Dickinson, 1987; Dickinson, Nicholas, & Adams, 1983). For example, Dickinson (1980) reviews the sensory preconditioning paradigm (Nader & LeDoux, 1999; Rizley & Rescorla, 1972; Talk, Gandhi, & Matzel, 2002; Young, Ahier, Upton, Joseph, & Gray, 1998). During stage 1, a tone is paired with food. During stage 2, the food is paired with illness. During stage 3, appetitive response to just the tone on its own is tested. Rats trained on stages 1 and 2 show an attenuated response in stage 3 in comparison to rats that were trained on only stage 1. This demonstrates that rats are able to integrate the knowledge acquired in stages 1 and 2, and Dickinson interprets this (and other data) as evidence for the presence of declarative representations (i.e., an internal world model). Our approach is a type of model-based reinforcement learning in which an agent learns to maximize reward through trial-and-error interaction with its environment. This satisfactorily captures the problem faced by a real animal. First, we assume that our model can recognize the current environmental stimulus and the time since its onset, by activating one of a set of predefined and fixed internal states. This assumption is similar to the tapped-delay-line representation assumption of Schultz et al. (1997), Montague et al. (1996), and others. Second, these states are built into an explicit internal model of the environment that comprises a transition function and a reward function. Third, the internal model is used to evaluate alternative actions and generate motivation via the online calculation of expected future reward. We will then propose a role for dopamine in modulating the efficacy of the connections between the states of the internal model, effectively implementing an online version of the discount factor of reinforcement learning (Sutton & Barto, 1998). Figure 1 illustrates our tapped-delay-line assumption. The temporal resolution at which stimuli are represented is arbitrary, and we choose an interval of 1s. This interval will affect the quantitative but not the qualitative nature of the results. The availability of states is assumed to be sufficient for each task that is considered. Each state, si , is associated with an intrinsic reward value, r(si ), that is assumed to be supplied by the environment. For example, a state representing shock might be assumed to elicit a reward value of -1, while a state representing food might elicit a reward of 1. Neutral stimuli will always elicit a reward value of zero. We assume that a number of different actions are available to the agent, A = {a1 , a2 , ..., am }, where m is the total number of actions available. We also assume that the agent can take an action only when a new environmental stimulus is presented. For example, actions are not taken in s‘Any ,t>0 . The internal model is described with reference to the simple neural circuit shown in Figure 2. Each time a new internal state, snew , is activated, the

368

A. Smith, S. Becker, and S. Kapur

...

... US

SUS,0

SUS,1

SUS,2

CS2

SCS2,0

SCS2,1

SCS2,2

CS1

SCS1,0

SCS1,1

SCS1,2

0

1

2

...

Figure 1: We assume that the model is able to recognize and represent the current stimulus and the time since its onset by activating one appropriate internal state unit from a predefined and fixed set, S = {s1 , . . . , sn }, where n is the total number of states. We will find it convenient to index each unit by either a stimulus-offset pair (as in the figure) or a single generic index that uniquely labels each state. For example, si is simply the ith state.

following procedure is performed: 1. Update reward estimate of the new state:
 ˆ new ) + α r(snew ) − R(s ˆ new ) . ˆ new ) := R(s R(s 2. Update the transition connections: 

ˆ ˆ old , aold , y) := (1 − β)T(sold , aold , y) + β T(s ˆ old , aold , y) (1 − β)T(s

If y = snew , Otherwise.

(3.1)

for all states, y ∈ S. 3. Action selection: If snew represents the onset of an external stimulus, then select and take action, anew , as described below. In the above, sold is the previously active state and aold is the previously selected action. α and β are learning rates, which are arbitrarily defined by α = β = exp(− trial 100 ), where trial is the trial number. These learning rates start at 1 and are slowly reduced to 0 across trials. Some environmental states are marked as terminal states (see the environment descriptions later), and when a terminal state transition occurs, the

Modeling the Functional Role of Dopamine in the Ventral Striatum

369

Figure 2: The internal model can be interpreted using a simple neural metaphor. Each internal state unit is connected to each other state unit under each possible action by a unique transition connection, the strength of which is controlled ˆ i , a2 , sj ), will be by a transition weight. For example, the transition strength, T(s adapted during learning to reflect the probability that state sj follows state si under action a2 . Note that the direction of the transition connections is from the state-action units back to the state units. Additionally, each state maintains ˆ ∈ an estimation of the immediate reward value associated with that state, R(s S). Although this value will be immediately available in r(s ∈ S), the former represents the estimate learned by the internal model, while the latter represents the actual value coming in from the environment. In theory, r(s ∈ S) could be a distribution, for example, while Rˆ ( s ∈ S) is always a scalar value. The reward values and transition connections are adapted during learning so that an explicit internal model of the environment is approximated. This model will be used to estimate future reward and to drive motivation and action selection. From an anatomical perspective, we speculate that the states themselves are represented in cortical areas (possibly including the OFC and amygdala), while the transition connections are routed through the ventral striatum (see the discussion).

trial is ended. Over a number of trials, the agent learns to represent an explicit internal model of its environment that consists of an estimated reward function and an estimated transition function. Step 2 just redistributes the weights from the relevant state-action unit (see Figure 2) back to each state unit according to a learning rate, β. All transition strengths are initialized  ˆ y, si ) = 1 for all to 1/n, and the redistribution rule ensures that ni=1 T(x, x ∈ S and y ∈ A at all times.

370

A. Smith, S. Becker, and S. Kapur

Action selection is performed by using a look-ahead process to generate the expected future reward of each action in turn and then selecting the best action. The look-ahead process involves playing through the consequences of taking each action inside the internal model. We assume that this lookahead can be performed without disturbing the modeling process described above. If actually implemented in neural circuitry, it might be necessary to maintain two copies of the internal model: one for keeping track of the environment and one for performing look-ahead. We abstract over this detail. We now propose a role for dopamine in modulating the efficacy of the transition connections. First, let zij denote the state-action unit corresponding to taking action j in state i. Next, let the activation of si and zij be denoted by ξ(si ) and ξ(zij ) respectively. Also, let FutRew(a ∈ A) be an internal register used for accumulating the total future reward of taking action a in current state, snew . The action selection step is fleshed out as follows: 3. Action Selection: If snew represents the onset of an external stimulus: (a) For each action, ai ∈ A: i. FutRew(ai ) := 0 ii. For all j∈ {1...n}: 1 If sj = snew , ξ(sj ) := 0 Otherwise. iii. For some fixed number of iterations, q: A. Propagate activation from state units to stateaction units: For all j ∈ {1...n} and all k ∈ {1...m}: ξ(sj ) If k = i, ξ(zjk ) := 0 Otherwise. B. Generate hypothetical next state: For all j ∈ {1...n}:   ˆ ξ(sj ) := nk=1 m l=1 ξ(zkl ) × T(sk , al , sj ) × DAtonic C. Collect rewards for this hypothetical state: n ˆ j) FutRew(ai ) := FutRew(ai ) + j=1 ξ(sj ) × R(s D. Return to 3(a)iiiA. (b) Select and take action,   argmax FutRew(a) With prob. p, a∈A anew:=  Random action With prob. 1−p.

Modeling the Functional Role of Dopamine in the Ventral Striatum

371

In the above, 0 ≤ DAtonic ≤ 1 is the level of tonic dopamine (default = 1); q is the maximum depth of the look-ahead process, which we arbitrarily set at 50; and p controls exploration, where p = (1 + exp(5 − 0.07trial))−1 . Exploration starts at 1 and is reduced to 0 over successive trials. Exploration is an important part of behavior acquisition, since in general the initial state of the internal model will not accurately reflect the environmental contingencies. The approach we have taken is very simple. Activity cycles around the circuit of Figure 2 in order to simulate the environmental consequences of each action in sequence. The action that accumulates the greatest future reward during this process is actually selected, generating a new real sequence of environmental stimuli. A role for DAtonic as a modulator of the transition connections is proposed in step 3(a)iiiB. This will allow us to account for the finding that actions motivated by distal rewards are more vulnerable to dopamine manipulation than those motivated by proximal rewards. We have made the simplifying assumption that during the look-ahead process, the action currently being evaluated is always selected (step 3(a)iiiA). We consider a more flexible alternative to this look-ahead policy later, although the qualitative nature of the results are not contingent on this feature of the model because of the simple environments used. nProviding that DAtonic = 1, then at any stage in the look-ahead process, j=1 ξ(sj ) = 1. This is guaranteed by the starting condition in 3(a)ii and the  ˆ y, sj ) = 1 for all x ∈ S and y ∈ A. More important, the fact that n T(x, j=1

activation of each state unit corresponds to the probability of that state indeed being the current state at that future time, under the look-ahead policy. However, for DAtonic < 1, these activations will decay during the look-ahead process, although the activations will still be in proportion to these probabilities. As the look-ahead process continues, FutRew(a) converges on the estimated future reward of taking action a, modulated by the online discount factor, DAtonic . Expected future reward or the return is the standard quantity to maximize in reinforcement learning problems (Sutton & Barto, 1998; Kaelbling, Littman, & Moore, 1996). Note that the agent is naturally parallel in design and relies mainly on local learning and activation rules. 4 Modeling Conditioned Avoidance A prerequisite of any model of delusions or disordered thoughts is a model of ordered thoughts. Since a model of ordered thoughts represents the holy grail of a number of disciplines, it is as well that we have a relatively simple yet highly relevant animal model of dopaminergic action at our disposal. We present the model in conjunction with a generalized version of the CA paradigm presented in Maffii (1959) (reviewed with other classic APD studies in Dews & Morse, 1961). The standard CA finding that APDs disrupt avoidance before escape is observed within Maffii’s paradigm, but Maffii

372

A. Smith, S. Becker, and S. Kapur

also makes an interesting additional observation that we wish to model. He found that after sufficient training, the rats began producing the avoidance response as soon as they were placed in the cage and before they were even presented with the tone. He termed this response to environmental context the secondary avoidance response, and the standard response to the tone the primary avoidance response. He then found that not only was the primary avoidance response more vulnerable to dopamine blockade than the escape response itself (the standard finding), but that the secondary avoidance response was more vulnerable than the primary response (see Figure 5, left). Apparently, the more distal the cue from the shock, the more vulnerable that cue was to dopamine blockade in terms of its ability to elicit a response. We can describe the standard CA environment with the finite state model of Figure 3a (left) and Maffii’s experiment with Figure 3a (right). The finite state environments defined in Figure 3 are a necessary assumption required to formalize the problem, and they effectively take on the role of an animal’s environment as well as its sensory processing. These environment descriptions will be different for each experiment that we consider. In contrast, the learning rules described above are defined once and represent our model of the behavioral processes, which we believe to pertain to the ventral striatal D2 receptor. Figure 4(a) shows the internal model after 100 learning trials. The transition connections reflect the environmental contingencies of Figure 3a (right). This internal model can then be used to generate the expected future reward of taking different actions in each environment state. For example, Figure 4e shows how activation is propagated through the internal model during the look-ahead process of the ”do nothing” action at trial onset for DAtonic = 1 for the internal model of Figure 4c. The return is generated by summing the reward estimates of each active unit in proportion to its activation. Setting DAtonic < 1 after acquisition of the internal model results in the decay of this activation during the look-ahead process, and therefore also the attenuation of the impact of increasingly distal outcomes (not shown). Figure 5 compares expected future reward thus generated with the performance of Maffii’s rats. Here we are assuming that expected future reward is a suitable basis for motivation. In order to convert this value into the probabilistic behavior of the rats as shown in Figure 5 (right), a softmax action selection mechanism would be required in place of the -greedy mechanism of step 3b. Although not shown, the model acquires avoidance behavior within a number of trials that is consistent with the amount of experience required by an actual rat (Wadenberg et al., 2001). However, this behavior is rather trivial, and the real point of interest is the selective effect of DAtonic on secondary avoidance versus primary avoidance versus escape. For interest, Figures 4c and 4e give an example of the model’s behavior in noisy or stochastic environments.

Modeling the Functional Role of Dopamine in the Ventral Striatum

Shock

5s

Tone 0

Environment cue 5s

−1 0s

Safety 0s 0

Tone

0

5s

373

Shock

0

−1 0s

0s

5s

Safety 0s 0

Run Do Nothing

5s

(a) MDP for modeling Conditioned Avoidance

Tone Start

0s

0

4s

150 0s

0 0s

Fixed reward

Immediate reward 100

0s

Terminal 0

Press lever 1 Press lever 2

(b) MDP for discounting task of (Wade et al., 2000) Figure 3: (a, left) A simple and abstract environment model for the classic CA task. Each circle represents a different environment state an agent can be in, and the arrows denote the outcome of each of the two possible actions. The arrows are labeled with the time between the action being taken and the transition actually occurring. For example, if the Do Nothing action is selected when the tone is presented, the shock will be delivered after a delay of 5 s (see Wadenberg et al., 2001, for an experimental example). We make the simplifying assumption that the agent must take one of the available actions on entry into a state, and must then wait for the ensuing transition before another action may be taken. The number inside each state represents the reward, r, associated with that state. The Safety state is the terminal state, which always transitions to itself, and the trial is terminated when the first such transition is made. Note that if the agent selects Do Nothing in the Shock state, the shock lasts only for 5 s, after which the trial is automatically terminated. This is consistent with a typical experimental setup (Wadenberg et al., 2001). (a, right) A generalized version of (left) that represents the experimental setup of Maffii (1959). (b) An abstract definition of the environment for the discounting task of Wade et al., (2000; see section 5). Trials begin in the left-most state.

374

A. Smith, S. Becker, and S. Kapur

Figure 4: (a) The state of the internal model after learning the environment of Figure 3a (right). The secondary stimulus refers to the Environment cue in Figure 3a (right) and the primary stimulus to the Tone. All learned transition ˆ are shown. The relationship to Figure 2 is as follows: The units connections, T, of Figure 2 are plotted here in weight space rather than physical or anatomical space. Also, the state-action units are removed, and the transition connections from each of the state-action units are collapsed onto the relevant state so that the connections from state to state can be seen more clearly. As a result of visuˆ a, s ) function as a simpler T(s, ˆ s ) function, it is not possible in alizing the T(s, this figure to know which action causes a given transition. However, the transition connections associated with taking the Run action in response to the onset of each external stimulus always transition to the Safety unit. The remaining transitions show the consequences of Do Nothing. (b) The state of the internal model after learning the environment of Figure 3b. (c) As in a except that temporal random noise is added to the identification of the current state. The bolder connections denote a stronger transition weight. (d) The state of the internal model after learning the T-maze environment of Figure 8(a). (e) The propagation of activation through the internal model of c during a typical look-ahead process for the Do Nothing action at the beginning of a trial, with DAtonic = 1. The process is shown for every alternate iteration.

Modeling the Functional Role of Dopamine in the Ventral Striatum

375

Figure 5: A comparison of model performance with Maffii’s data. (a) Number of secondary avoidance responses, primary avoidance responses, and escape responses under increasing doses of the neuroleptic chlorpromazine (a drug with a particularly high affinity for the D2 receptor), as a percentage of the number of responses without the drug (adapted from Maffii, 1959). (b) The change in FutRew(Do Nothing) for each of the three important states (solid line = environment cue, dashed line = tone, gray line = shock) as DAtonic is decreased. Since we use DAtonic as an abstract representation of dopamine and the model does not attempt to address the underlying neurochemical processes, the relationship between the model parameter, DAtonic , and chlorpromazine dose is uncertain. However, it is the qualitative nature of the results that is of interest, and in particular, the selective effect of dopamine blockade on secondary avoidance versus primary avoidance versus escape. For the comparison to be meaningful, we assume that FutRew(Do Nothing) can be used as a direct analogy of motivation, since the alternative action, Run, always yields an estimated future reward of zero. The horizontal line suggests an example escape cost that could be used to threshold motivation. This could explain why many studies find that low doses of neuroleptics disrupt avoidance but not escape.

5 Impulsivity, Delayed Rewards, and ADHD Attention deficit/hyperactivity disorder (ADHD) is a developmental disorder affecting 3% to 7% of school-age children, characterized by inattention, hyperactivity, and impulsivity (Frances, 2000). The single most effective treatment for the disorder is medication with psychostimulants such as methylphenidate (Phares, 2003). Psychostimulants (of which amphetamine is one) are known to increase extracellular concentrations of dopamine (Seeman & Madras, 1998), and indeed ADHD is believed to be primarily a dopaminergic/noradrenergic disorder (see Phares, 2003). Decreased blood flow has also been observed in the striatum of ADHD patients, a deficit

376

A. Smith, S. Becker, and S. Kapur

reversible by treatment with methylphenidate (reviewed in Schneider, Sun, & Roeltgen, 1994). As with psychosis, striatal dopamine seems to be of importance to ADHD. However, unlike psychosis, it may be a deficit rather than an excess that is to blame. There are no laboratory tests, neurological assessments, or attentional assessments that have been established as a diagnostic in the clinical assessment of ADHD (Frances, 2000). However, Solanto et al., (2001) have suggested that a desire to avoid delay (the delay aversion hypothesis) is one of the best characterizations of impulsivity with respect to ADHD, and Catania (in press) has argued specifically that many of the symptoms of ADHD can be accounted for by assuming too steep a discounting gradient. Studies such as de Wit et al. (2002) have confirmed that certain measures of impulsivity are indeed reduced in healthy volunteers by amphetamine. For example, subjects were asked questions such as, “Would you prefer ten dollars in thirty days or two dollars at the end of the session?” The amphetaminetreated subjects showed an increased preference for the larger but delayed reward when compared with a placebo group. If blocking dopamine can disrupt avoidance responding in rats, enhancing dopamine seems to decrease impulsivity as measured by this kind of delay discounting (Wade et al., 2000; Richards et al., 1999; de Wit et al., 2002). Furthermore, both phenomena may be mediated by the same part of the brain: the striatum (see Phares, 2003, for more on striatal involvement in ADHD). However, as with psychosis, the underlying behavioral and psychological processes are unclear. In a bid to better understand the role of dopamine in impulsivity and ADHD, Richards et al. (1999) and Wade et al. (2000) have investigated the effects of both amphetamine- and dopamineblocking drugs on delay discounting in rats. Their experimental paradigm can be summarized as follows. A thirsty rat is trained to press one of two levers. One lever yields an immediate reward (for example, 100 µl of water), and the other yields a fixed reward (150 µl of water) but only after a delay of 4 s. If the animal selects the delayed reward, a tone is presented between the lever press and the water to make the task easier. The immediate reward is then adjusted from trial to trial depending on which lever the animal selected on the previous trial. If the rat chose the immediate reward, the immediate reward is reduced by 15%, and if the rat chose the delayed reward, the immediate reward is increased by 15%. In this way, the immediate reward is varied until the rat has no particular preference. The amount of immediate reward elicited at this indifference point is then interpreted as the animal’s ”value” of the fixed, delayed reward. The rats are trained over a period of many weeks, on a variety of different starting conditions, until they become familiar with and adept at exploring the two alternatives and achieving the indifference point that suits their preference. As an additional aid, if the rat chooses the same lever twice in a row, then the immediately following trial is a forced exploration trial in which only the other lever yields a reward.

Modeling the Functional Role of Dopamine in the Ventral Striatum

377

Figure 6: The impact of various doses of (a) a dopamine-enhancing drug (amphetamine) and (b) a D2-blocking drug (raclopride), on the indifference point. The indifference point indicates the value of the delayed 150 µL alternative. Error bars indicate SEM, and asterisks denote a significant difference from dose = 0. Adapted from Wade et al. (2000).

After the rats have been trained on this procedure, they are tested under various systemic doses of both amphetamine and raclopride (dopamine D2 receptor blocker; see Figure 6). A dose-dependent effect of both drugs is observed on the indifference point. They claim that amphetamine has effectively reduced impulsivity (by increasing the value of the delayed reward), and raclopride has increased impulsivity (by decreasing the value of the delayed reward). Because of the incrementally adjusting procedure used, it is impossible to rule out a learning effect completely. However, the effects of these drugs were very quick compared with the weeks of training required for acquisition of the basic task. In the case of raclopride in particular, the lower indifference point was reached as quickly as the paradigm allowed—after just a few trials under the drug. It therefore seems likely that a significant part of the change in the animals’ behavior was due to the effect of drug on performance (rather than or in addition to its effect on learning). Wade et al. (2000) conclude that “this pattern of results indicates that blocking D2-receptors may have a selective effect on the value of delayed rewards” (p. 197). We can now use the model described in the previous section to propose an explanation that is consistent with data from CA. Figure 4b shows the result of training the agent on the environment of Figure 3b. Acquisition is again trivial (for DAtonic = 1, the agent learns to select the greater, delayed reward), but Figure 7 shows the effect of manipulating dopamine after acquisition. The delayed reward is discounted more in

378

A. Smith, S. Becker, and S. Kapur

the look-ahead process as DAtonic is reduced. In order to capture the effects of both dopamine blockade and amphetamine, we assume a baseline DAtonic < 1. The model is then able to provide a qualitative account of all drug-induced changes in impulsivity. Note that the vertical bars shown in Figure 7 can be moved apart (or together) by reducing (or increasing) the temporal resolution in Figure 1, thereby modifying the steepness of discounting. For reference, the largest dose of raclopride used (120 µg) would reduce avoidance responding in CAR by around 20% (Wadenberg, Kapur, Soliman, Jones, & Vaccarino, 2000). Notionally, the model also captures five additional observations made by Wade et al. (2000). First, independent of drug treatments, they vary the delay to the fixed reward (2 s and 8 s). They find that a delay of 2 s increases the indifference point (value of the delayed alternative) and that a delay of 8 s decreases the indifference point. Second, they find that the initial amount of water on the immediate alternative does not affect the eventual indifference point. Third, they find that the deprivational state of the animal (i.e., its thirst) does not affect the indifference point. In our model, a logical role for thirst would affect the perception of the two rewards equally (i.e., by a common factor), which could be achieved, for example, by substituting step 3(a)iiiC with: 3(a)iiiC)

Collect rewards for this hypothetical state. n ˆ j ) × ”Thirst”. FutRew(ai ) := FutRew(ai ) + j=1 ξ(sj ) × R(s

This would not affect the relative indifference point. Fourth, our model predicts that blocking dopamine should generally reduce the motivation to press either lever, and enhancing dopamine should generally increase the motivation to press a lever. Wade et al (2000) examined the tendencies of the rats to complete each trial and found that raclopride reduced but amphetamine increased the mean number of trials completed. Admittedly, completed trials is only an informal measure of motivation. Also, in a related and recent study, J.B. Richards (personal communication, 2003) finds that if rats are trained on a task in which levers yield immediate rewards, but one is certain and one is uncertain, then amphetamine does not affect the indifference point. This behavior would be produced by the model too because uncertainty is represented by the transition connection strengths, and dopamine has an equal effect on all such connections. A complete analysis of the model’s performance under these five conditions is outside the scope of the account presented here. It should be noted that Wade et al. (2000) observed the effects described above only with dopamine D2-blocking drugs. D1 receptor blockers had no significant effect. Interestingly, Cardinal et al. (2001) demonstrate that accumbens (core) lesions in rats cause exactly the same kinds of effects, leading them to conclude that the accumbens is involved in the pathogenesis of impulsive choice, a finding they suggest can shed light on ADHD, addiction, and other impulsive control disorders. (Cardinal, Robbins, & Everitt (2000)

Modeling the Functional Role of Dopamine in the Ventral Striatum

379

Figure 7: The effect of varying the DAtonic parameter after learning on the estimated future reward associated with each of the two actions from the Start state. The solid curve shows the estimated future reward of pressing lever 1 (delayed reward =140 µL), while the horizontal dashed line shows the estimated future reward of pressing lever 2 (immediate reward = 100 µL). The gray curved line shows the hypothetical value of the immediate reward necessary to balance the estimated future reward of the immediate alternative with that of the delayed alternative (i.e., the indifference point). This gray line can be used to match the model performance with data from Wade et al. (2000). The thick vertical bar shows the simulated level of dopamine that corresponds to the performance of the undrugged rats in Wade et al. (2000). The vertical bars to the left show the simulated dopamine level corresponding to the indifference point of the amphetamine-treated rats, and the vertical bars to the right show the simulated dopamine level corresponding to the performance of the raclopride-treated rats (see Figure 6). In agreement with the animal study, the model predicts that reducing DAtonic reduces the indifference point, while increasing dopamine increases the indifference point.

also find the same kinds of impulsive behaviour with systemic administration of amphetamine and combined D1/D2 blockers. It has been suggested that accumbens dopamine is necessary for producing anticipatory responses (e.g., avoidance, or lever pressing for food or water), but not consummatory responses (e.g., escape, or feeding or drinking itself) (Ikemoto & Panksepp, 1999). It is therefore noteworthy that in the impulsivity studies discussed above, two equally anticipatory responses with

380

A. Smith, S. Becker, and S. Kapur

equal motoric requirements were dissociated by dopamine manipulation. Therefore, arguments that rely solely either on anticipatory-consummatory distinctions or on motor deficits may need to be adjusted to address delaydiscounting paradigms. Finally, two important caveats are considered. First, in contrast to the data discussed above, some studies have observed that amphetamine actually increases impulsivity rather than decreases it, leading to the suggestion that the finer experimental details, such as whether a cue is presented during the delay, may be important (Cardinal et al., 2000). Perhaps one problem is that amphetamine injections directly into the accumbens increase general locomotor activity (reviewed in Pennartz, Groenewegen, & Silva, 1994), as well as an animal’s motivation to work for reinforcers (Taylor & Robbins, 1984), as predicted by the internal model since expected future reward equals motivation). Therefore, it may be difficult for impulsivity studies to separate the absolute increases in motivation due to amphetamine from the relative decreases in choice tasks. The way in which the animal internally models its environment is likely to play a large role. Second, it must be acknowledged that ADHD is a complex and multidimensional disorder, and the delay aversion hypothesis (see Solanto et al., 2001) and delay discounting hypothesis (see Catania, in press), represent only one line of argument. However, we conclude by contrasting our proposed role for dopamine in gating a look-ahead process with one of the standard characterizations of impulsivity within ADHD offered by DSM-IV (emphasis our own): “Impulsivity may lead to accidents and to engagement in potentially dangerous activities without consideration of possible consequences” (Frances, 2000, p. 86). 6 A T-Maze Experiment In a fascinating study by (Cousins et al. (1996), rats were presented with a choice between two arms of a T-maze: one leading to four food pellets, and the other to two. However, the arm leading to four pellets was obstructed with a barrier that had to be climbed (see Figure 8b). Once trained, the rats chose the arm containing four pellets on almost 100% of trials. However, after bilateral intra-accumbens injections of 6-hydroxydopamine, a treatment that destroys dopaminergic projections to the accumbens, the rats changed their behavior, selecting the unobstructed but lesser reward instead (80% of the time). A second experiment was then performed in which different rats were trained (untreated, as before) on a different version of the T-maze in which there was no reward in the unobstructed arm. Subsequent dopamine lesions only slightly reduced responding for the obstructed arm (from 100% of the time to 80%). Therefore, Cousins et al. (1996) were able to rule out the possibility that the animals in the first experiment were unable to cross the barrier because of motoric deficits for example. Rather, it appears that dopamine lesions were influencing the motivational and choice processes of the animal.

Modeling the Functional Role of Dopamine in the Ventral Striatum

381

Figure 8: (a) A formal environment model capturing the salient features of the T-maze task. For simplicity we can assume negligible delays between all state transitions. This is not a necessary assumption. (b) The T-maze experimental setup used in Cousins et al. (1996). (c) The effect of varying DAtonic ,after acquisition, on the estimated future reward associated with taking the Left (solid) and Right (dotted) actions from the Start state. The point at which the graphs cross denotes the point at which the model will switch from seeking the obstructed reward to seeking the easily available alternative.

The internal model can be used to capture these results by training it on the environment of Figure 8a. Figure 4d illustrates the resulting internal model, and Figure 8c demonstrates the effect of modulating DAtonic , after acquisition, on the estimated future reward associated with moving left or right from the start state. The model accounts for the change in animal behavior from the obstructed to the unobstructed arm under dopamine disruption. It also notionally accounts for the observation that if the rats are trained with no food in the unobstructed arm, then they continue to select

382

A. Smith, S. Becker, and S. Kapur

the obstructed arm, although apparently with less motivation. The model would continue to select the obstructed arm in this case until the solid line failed to exceed the cost of climbing the wall. The results of this study are particularly striking because the nucleus accumbens is specifically targeted with a direct dopamine challenge, eliciting a qualitative change in behavior. This result is not an isolated experiment either. It is duplicated in essence in Salamone et al. (1991, 1994) with both direct accumbens dopamine depletion and systemic D2 blockers, and Salamone et al. (1997) provides a review of many other related experiments by Salamone and colleagues that adhere to the same general pattern. 7 Instrumental Responding for Rewards A generally accepted consequence of dopamine blockade in rats is a reduction in lever pressing (also other responses) for various types of reward (Ettenberg, Koob, & Bloom, 1981; Evenden & Robbins, 1983; Fibiger et al., 1976; Fowler et al., 1986; Rolls et al., 1974; Salamone et al., 1993; Wise & Schwartz, 1981; Wise et al., 1978). Moreover, lever pressing for food or water is often found to be reduced at doses that have little or no impact on free feeding or free drinking where no lever press is required (Rolls et al., 1974; Salamone et al., 1993; see Figure 9a). However, recall that Berridge and Robinson (1998) were able to disrupt free feeding, even if the food was under the rats’ noses, by reducing accumbens and neostriatal dopamine by around 95%. We can use the current model to account for these findings by assuming the environment model of Figure 9b. In order to estimate future reward (and therefore generate motivation) for lever pressing, all three of the transitions are required. In contrast, free feeding requires only Approach and Consume, and in Berridge’s rats’ case, only the Consume transition is required. We do not show the model’s performance on this task because of the qualitative similarity between Figures 9a and 5a. However, it should be evident that lever pressing is disrupted before free feeding and free feeding before consumption in the model under the assumption of Figure 9b. With respect to this and the T-maze experiment reviewed earlier, it is an open question as to whether it is the “instrumental” or temporal distance (or both) between an action and its rewarding outcome that renders that action vulnerable to dopamine blockade. 8 Uniting Existing Hypotheses and Related Work We have presented a computational model intended to address a corpus of experimental data that point to a selective role for dopamine in the expression of previously acquired behaviors. More specifically, the D2receptor subtype within the ventral striatum (particularly the accumbens) has emerged as a common neuroanatomical thread throughout the studies we have considered. If our model of the transition connections passing

Modeling the Functional Role of Dopamine in the Ventral Striatum

383

Figure 9: (a) Lever pressing for food is attenuated at neuroleptic doses that do not drastically affect free feeding. Free drinking may be even more robust than free feeding to dopamine challenge (results adapted from Rolls et al., 1974). Note that spiroperidol primarily blocks the D2 receptor. (b) An abstract environment for instrumental responding that, in combination with the model presented, accounts for the selective vulnerability of instrumental responding to dopamine challenge.

through the ventral striatum is plausible, then we would expect to see this area being activated in response to conditioned stimuli as the look-ahead process is invoked. A number of fMRI studies in human subjects confirm this for both appetitive (Gottfried, O’Doherty, & Dolan, 2002; O’Doherty, Deichmann, Critchley, & Dolan, 2002; O’Doherty, Dayan, Friston, Critchley, & Dolan, 2003) and aversive (Jenson et al., 2003) tasks. Also, using electrophysiological recording techniques, Schultz, Tremblay, & Hollerman (2000) have found a variety of activations in the ventral striatum that are consistent with its role in the expectation of reward across a delay. In addition to the ventral striatum, a range of fMRI, electrophysiological, and behavioral data points to a central role for the amygdala (Cador, Robbins, & Everitt, 1989; Everitt, Morris, O’Brien, & Robbins, 1991; Everitt & Robbins, 1989; Cardinal et al., 2002; Nishijo, Ono, & Nishijo, 1988; Robbins, Cador, Taylor, & Everitt, 1989; Tremblay & Schultz, 2000a, 2000b) and the orbito-frontal cortex (Arana et al., 2003; Bechara, Damasio, Tranel, & Anderson, 1998; Cardinal et al., 2002; Iversen & Mishkin, 1970; Masterman & Cummings, 1997; O’Doherty et al., 2003) in the representation of environmental reward contingencies of the kind relevant to the current discussion. It has been suggested that the OFC in particular is crucially involved in the motivational control of goal-directed behavior (Schultz et al., 2000) learning about rewards (Dias, Robbins, & Roberts, 1996), and evaluating alternatives

384

A. Smith, S. Becker, and S. Kapur

(Arana et al., 2003; Schultz et al., 2003) via a common neural currency (Montague & Berns, 2002). We therefore speculate that the states themselves may be represented cortically (originating presumably in the sensory cortex), ˆ along with the the return being represented with the reward estimates, R, in the OFC and amygdala. The proposed model can be used to unite a number of existing cognitive hypotheses of dopamine function within a formal framework. For example, Berridge and Robinson (1998) suggest that mesolimbic dopamine mediates the wanting component of reward as distinct from the liking, and in our model, following McClure et al. (2003), we interpret expected future reward as precisely this wanting. Salamone et al. (1997) suggest that “accumbens dopamine is important for responding to stimuli that are spatially and temporally distant from the organism” (p. 353). This statement precisely summarizes the psychological value of dopamine in our model. Ikemoto and Panksepp (1999) have also noted the relevance of the proximal-distal distinction to mesolimbic dopamine function. They argue that nucleus accumbens dopamine is important for invigorating flexible approach responses (distal), as distinct from consummatory responses (proximal). With respect to our model, and particularly section 7, it should be noted that consummatory behaviors such as free feeding may not be disrupted with only accumbens dopamine depletions. For example, Berridge and Robinson (1998) achieved aphagia with both accumbens and neostriatal dopamine reduction. The striatum may therefore be functionally differentiated, with the ventral region specializing in distal motivation (e.g., approach) and the dorsal region in proximal motivation (e.g., consumption). Modeling this distinction must constitute future work. Our approach is different from the TD prediction-error hypothesis (see Houk et al., 1995; Schultz et al., 1997; Montague et al., 1996), which suggests that the phasic dopamine response signals the difference (error) between the future reward predicted by the animal and the actual reward received. This error is then used to drive the learning process in a biologically plausible fashion (Waelti, Dickinson, & Schultz, 2001). In contrast, we have proposed a role for tonic dopamine in the generation of expected future reward that is independent of the acquisition process. The advantages of our approach are that we can model the effect of dopamine manipulation not only on the expression of previously acquired behaviors, but also the sensitivity of this effect to the relationship between action (or CS), and outcome (or US). A weakness of our internal model is that it fails to address the role of dopamine in the acquisition process or the phasic response of dopamine neurons themselves. We therefore suggest that future work should be oriented around hybrid approaches aimed at achieving a more comprehensive account of the neuromodulator. Toward this end, a number of discussions and models have been proffered that extend TD-based representations with

Modeling the Functional Role of Dopamine in the Ventral Striatum

385

explicit internal model representations (Daw, Courville, & Touretzky, 2004; Dayan, 2002; Dayan & Balleine, 2002; Suri & Schultz, 1998; Suri et al., 2001; Suri, 2001, 2002). However, a significant challenge remains in bridging the gap between models of dopamine neuron firing and models of behavioral and psychological phenomena in which dopamine may play a pivotal role. It is particularly important that we achieve a better understanding of whether explicit internal model representations or a cached value function (as in TD) is most appropriate for modeling the brain reward system. To conclude this section, we speculate as to why the brain might need DAtonic . Some suggestions include (1) constraint of the look-ahead process via its action as an online discount factor; (2) adaptation of the trade-off between proximal and distal rewards in response to environment cues (Wilson & Daly, 2004) or deprivational states (Giordano et al., 2002); or (3) global control of general motivated activity. 9 Model Predictions and Future Work Model is only as good as the useful predictions it makes, and so we are actively engaged in a program of experimental validation. In the account presented above, we allowed the model to make an action choice only in response to the onset of an external stimulus. However, within the context of conditioned avoidance, we have also looked at the predictions made by the model if an action can be selected in any state, including the internal timing states. These novel model-driven predictions were subsequently validated with experimental data (Smith et al., 2004), leading us to argue against the preeminent motor deficit hypothesis (Aguilar et al., 2000; Ogren & Archer, 1994) in favor of a motivational hypothesis of APD-induced avoidance disruption in rats. We are looking for interactions of dopamine manipulation with CS-US interval within CA, with a view to further testing the internal model account of motivated behavior. One of the primary motivations for undertaking this work is to create a computational dopamine hypothesis that can be used to shed light on schizophrenia. We argue that many of the negative symptoms of schizophrenia can be notionally captured by reducing DAtonic , including reduced motivation, flat affect, and anhedonia (given the suggestion that many of the rewards of life may actually be conditioned stimuli; Wise, (2002). Grace (1991) and Moore, West, and Grace (1999) have argued that a tonically hypo-dopaminergic state may be a key step to the development of psychosis, with the latter perceived as a hypersensitivity to phasic dopamine caused as result of homeostasis to the former. With respect to modeling psychosis, we suggest that an aberrant dopamine signal could lead to the construction or modulation of an aberrant internal model, and an aberrant internal model seems to be an excellent starting place to model delusions. Further research into combining a phasic (learning) role for dopamine with the internal model would be expected to flesh out this hypothesis.

386

A. Smith, S. Becker, and S. Kapur

One model assumption deserves a brief revisit. We enforced a very simple policy in the look-ahead process (step 3(a)iiiA), in which the current action being evaluated was always selected at each subsequent hypothetical state. This approach was simple and adequate given the way in which the problems were formulated. However, a more general and flexible look-ahead process would search different possible actions as one might in a game of chess, for example. However, since a branching search process is potentially costly, an alternative is to represent a Q-value at each state-action unit and then use this value to guide the search process during look-ahead. Such a value could be constructed using either a TD or Monte Carlo method (see Sutton & Barton, 1998). This would allow the agent to trade off the benefits of being able to explicitly simulate the consequences of actions (using the internal model) against the efficiency of simply using a precalculated value (an action-value function). The former is particularly important for being able to modulate behavior after acquisition based on an internal drive (e.g., salt deprivation) that was not present during conditioning. Berridge and Schulkin (1989) have demonstrated just such an ability in animals. There are already open lines of investigation into when and where internal model versus TD-like value functions influence motivated behavior (Dayan, 2002; Dayan & Balleine, 2002). However, using an action-value function to guide the online look-ahead process does not have a direct impact on the current discussion of dopamine function, which is kept as simple as possible. In conclusion, we have demonstrated that an internal model approach is able to account for a range of experimental evidence that suggests that ventral striatal dopamine D2-receptor manipulation selectively modulates motivated behavior for distal versus proximal outcomes. Whether an internal model or the cached values of the TD algorithm are better placed to model both a tonic and phasic dopamine response in this brain region is likely to have important implications for understanding a number of human disorders, including schizophrenia and ADHD. Acknowledgments This work was primarily supported by an OMHF Special Initiative grant and a NET grant from the Canadian Institutes of Health Research. S.K. is additionally supported by a Canada Research chair. Thanks also to Ming Li and Jimmy Jensen for assisting in the review of the conditioned avoidance and fMRI literature (respectively) and to Christopher Fiorillo for reviewing an early draft of this work. References Ader, R., & Clink, D. W. (1957). Effects of chlorpromazine on the acquisition extinction of an avoidance response in the rat. J. Pharmacol. Exp. Ther., 131, 144–148.

Modeling the Functional Role of Dopamine in the Ventral Striatum

387

Aguilar, M. A., Mari-Sanmillan, M. I., Morant-Deusa, J. J., & Minarro, J. (2000). Different inhibition of conditioned avoidance response by clozapine and Da D1 and D2 antagonists in male mice. Behav. Neurosci., 114(2), 389–400. Anisman, H., Irwin, J., Zacharko, R. M., & Tombaugh, T. N. (1982). Effects of dopamine receptor blockade on avoidance performance: Assessment of effects on cue-shock and response-outcome associations. Behavioral and Neural Biology, 36, 280–290. Arana, F. S., Parkinson, J. A., Hinton, E., Holland, A. J., Owen, A. M., & Roberts, A. C. (2003). Dissociable contributions of the human amygdala and orbitofrontal cortex to incentive motivation and goal selection. Journal of Neuroscience, 23(29), 9632–9638. Arnt, J. (1982). Pharmacological specificity of conditioned avoidance response Inhibition in rats: Inhibition by neuroleptics and correlation to dopamine receptor blockade. Acta Pharmacol. Toxicol. (Copenh.), 51(4), 321–329. Balleine, B. W., Garner, C., Gonzalez, F., & Dickinson, A. (1995). Motivational control of heterogeneous instrumental chains. Journal of Experimental Psychology: Animal Behaviour Processes, 21, 203–217. Bechara, A., Damasio, H., Tranel, D., & Anderson, S. W. (1998). Dissociation of working memory from decision making within the human prefrontal cortex. Journal of neuroscience, 18(1), 428–437. Beck, A. T., & Rector, N. A. (2003). A cognitive model of hallucinations. Cognitive Therapy and Research, 27(1), 19–52. Bell, D. S. (1973). The experimental reproduction of amphetamine psychosis. Archives of General Psychiatry, 29, 35–40. Beninger, R. J. (1989). Dissociating the effects of altered dopaminergic function on performance and learning. Brain Research Bulletin, 23, 365–371. Beninger, R. J., & Hahn, B. L. (1983). Pimozide blocks establishment but not expression of amphetamine-produced environment-specific conditioning. Science, 220, 1304–1306. Beninger, R. J., Mason, S. T., Phillips, A. G., & Fibiger, H. C. (1980a). The use of conditioned suppression to evaluate the nature of neuroleptic-induced avoidance deficits. J. Pharmacol. Exp. Ther., 213(3), 623-627. Beninger, R. J., Mason, S. T., Phillips, A. G., & Fibiger, H. C. (1980b). The use of extinction to investigate the nature of neuroleptic-induced avoidance deficits. Psychopharmacology (Berl.), 69(1), 11–18. Berridge, K. C., & Robinson, T. E. (1998). What is the role of dopamine in reward: Hedonic impact, reward learning, or incentive salience? Brain Research Reviews, 28, 309–369. Berridge, K., & Robinson, T. E. (2003). Parsing reward. Trends in Neurosciences, 26(9), 507–513. Berridge, K. C., & Schulkin, J. (1989). Palatability shift of a salt-associated incentive during sodium depletion. Quarterly Journal of Experimental Psychology, 41B(2), 121–138. Black, A. H. (1963). The effects of CS-US interval on avoidance conditioning in the rat. Canadian Journal of Psychology, 17(2), 174–182. Blackburn, J. R., & Phillips, A. G. (1989). Blockade of acquisition of one-way conditioned avoidance responding by haloperidol and metoclopramide but not

388

A. Smith, S. Becker, and S. Kapur

by thioridazine or clozapine: Implications for screening new antipsychotic drugs. Psychopharmacology, 98, 453–459. Cador, M., Robbins, T. W., & Everitt, B. J. (1989). Involvement of the amygdala in stimulus-reward associations: Interaction with the ventral striatum. Neuroscience, 30(1), 77–86. Cardinal, R. N., Parkinson, J. A., Hall, J., & Everitt, B. J. (2002). Emotion and motivation: The role of the amygdala, ventral striatum, and prefrontal coretx. Neuroscience and Biobehavioral Reviews, 26(3), 321–352. Cardinal, R. N., Pennicott, D. R., Sugathapala, C. L., Robbins, T. W., & Everitt, B. J. (2001). Impulsive choice induced in rats by lesions of the nucleus accumbens core. Science, 292(5526), 2499–2501. Cardinal, R. N., Robbins, T. W., & Everitt, B. J. (2000). The effects of damphetamine, chlordiazepoxide, alpha-flupenthixol and behavioural manipulations on choice of signalled and unsignalled delayed reinforcement in rats. Psychopharmacology, 152, 362–375. Catania, A. C. (in press). Attention-deficit/hyperactivity disorder (ADHD): Delay-of-reinforcement gradients and other behavioral mechanisms. Behavioral and Brain Sciences. Connell, P. H. (1958). Amphetamine psychosis. London: Chapman and Hall. Cook, L., & Catania, A. C. (1964). Effects of drugs on avoidance and escape behavior. Federation Proceedings, 23, 818–835. Cook, L., & Weidley, E. (1957). Behavioral effects of some psychopharmacological agents. Ann. N.Y. Acad. Sci., 66, 740–752. Courvoisier, S. (1956a). Pharmacodynamic basis for the use of chlorpromazine in psychiatry. Quarterly Review of Psychiatry and Neurology, 17(1), 25–37. Courvoisier, S. (1956b). Pharmacodynamic basis for the use of chlorpromazine in psychiatry. Journal of Clin. Exp. Psychophathol. Quart. Rev. Psychiat. Neurol., 17, 25–37. Cousins, M. S., Atherton, A., Turner, L., & Salamone, J. D. (1996). Nucleus accumbens dopamine depletions alter relative response allocation in a Tmaze cost/benefit task. Behavioural Brain Research, 74, 189–197. Davidson, A. B., & Weidley, E. (1976). Differential effects of neuroleptic and other psychotropic agents on acquisition of avoidance in rats. Life Sci., 18(11), 1279–1284. Davis, W. M., Capehart, J., & Llewellin, W. L. (1961). Mediated acquisition of a fear-motivated response and inhibitiory effects of chlorpromazine. Psychopharmacologia, 2, 268–276. Daw, N. D., Courville, A. C., & Touretzky, D. S. (2004). Timing and partial observability in the dopamine system. In S. Still, W. Bialek, & L. Botlou (Eds.), advances in neural information processing systems, 16. Cambridge, MA: MIT Press. Dayan, P. (2002). Motivated reinforcement learning. In T. G. Dietterich, S. Becker, & Z. Ghahramani, (Eds.) Advances in neural information processing systems, 14. Cambridge, MA: MIT Press. Dayan, P., & Balleine, B. W. (2002). Reward, motivation and reinforcement learning. Neuron, 36, 285–298.

Modeling the Functional Role of Dopamine in the Ventral Striatum

389

de Wit, H., Enggasser, J. L., & Richards, J. B. (2002). Acute administration of damphetamine decreases impulsivity in healthy volunteers. Neuropsychopharmacology, 27(5), 813–825. Dews, P. B., & Morse, W. H. (1961). Behavioral pharmacology. Annual Review of Pharmacology, 1, 145–174. Dias, R., Robbins, T. W., & Roberts, A. C. (1996). Disssociation in prefrontal cortex of affective and attentional shifts. Nature, 380(6569), 69–72. DiChiara, G. (1999). Drug addiction as dopamine-dependent associative learning disorder. European Journal of Pharmacology, 375, 13–30. Dickinson, A. (1980). Contemporary animal learning theory. Cambridge: Cambridge University Press. Dickinson, A. (1987). Instrumental performance following saccharin prefeeding. Behavioural Processes, 14, 147–154. Dickinson, A., Nicholas, D. J., & Adams, C. D. (1983). The effect of the instrumental training contingency on susceptibility to reinforcer devaluation. Quarterly Journal of Experimental Psychology, 35B, 35–51. Dickinson, A., Smith, J., & Mirenowicz, J. (2000). Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. Behavioral Neuroscience, 40, 468–483. Ettenberg, A. (1989). Dopamine, neuroleptics and reinforced behavior. Neuroscience and Biobehavioral Reviews, 13, 105–111. Ettenberg, A., Koob, G. F., & Bloom, F. E. (1981). Response artifact in the measurement of neuroleptic-induced anhedonia. Science, 213, 357–359. Evenden, J. L., & Robbins, T. W. (1983). Dissociable effects of d-amphetamine, chlordiazepoxide and alpha-flupenthixol on choice and rate measures of reinforcement in the rat. Psychopharmacology, 79, 180–186. Everitt, B. J., Morris, K. A., O’Brien, A., & Robbins, T. W. (1991). The basolateral amygdala-ventral striatal system and conditioned place preference: Further evidence of limbic-striatal interactions underlying reward-related processes. Neuroscience, 42, 1–18. Everitt, B. J., & Robbins, M. C. T. W. (1989). Interactions between the amygdala and ventral striatum in the stimulus-reward associations: Studies using a second-order schedule of sexual reinforcement. Neuroscience, 30(1), 63–75. Fibiger, H. C., Carter, D. A., & Phillips, A. G. (1976). Decreased intracranial self-stimulation after neuroleptics of 6-hydroxydopamine: Evidence for mediation by motor deficits rather than by reduced reward. Psychopharmacology, 47, 21–27. Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299, 1898–1902. Fowler, S. C., LaCerra, M. M., & Ettenberg, A. (1986). Effects of haloperidol on the biophysical characteristics of operant responding: Implications for motor and reinforcement processes. Pharmacology, Biochemistry and Behaviour, 25, 791–796. Frances, A., (Ed.). (2000). Diagnostic and statistical manual of mental disorders. Washington, DC: American Psychiatric Association. Giordano, L. A., Bickel, W. K., Loewenstein, G., Jacobs, E. A., Marsch, L., & Badger, G. J. (2002). Mild opioid deprivation increases the degree that opioid-

390

A. Smith, S. Becker, and S. Kapur

dependent outpatients discount delayed heroin and money. Psychopharmacology, 163, 174–182. Gottfried, J. A., O’Doherty, J., & Dolan, R. J. (2002). Appetitive and aversive olfactory learning in humans studied using event-related functional magnetic resonance imaging. Journal of Neuroscience, 22, 10829–10837. Grace, A. A. (1991). Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: A hypothesis for the etiology of schizophrenia. Neuroscience, 41(1), 1–24. Grace, A. A. (2000). Gating of information flow within the limbic system and the pathophysiology of schizophrenia. Brain Research Reviews, 31, 330–341. Grilly, D. M., Johnson, S. K., Minardo, R., Jacoby, D., & LaRiccia, J. (1984). How do tranquilizing agents selectively inhibit conditioned avoidance responding? Psychopharmacology, 84, 262–267. Hollerman, J., & Schultz, W. (1998). Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neuroscience, 1, 304– 309. Horvitz, J. C. (2000). Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience, 96(4), 651–656. Horvitz, J. C. (2002). Dopamine gating of glutamatergic sensorimotor and incentive motivational input signals to the striatum. Behavioural Brain Research, 137, 65–74. Houk, J. C., Adams, J. L., & Barto, A. G. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In J. C. Houk, J. L. Davis, & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 249–270). Cambridge, MA: MIT press. Hunt, H. F. (1956). Some effects of drugs on classical (type S) conditioning. Ann. N.Y. Acad. Sci., 65, 258–267. Ikemoto, S., & Panksepp, J. (1999). The role of nucleus accumbens dopamine in motivated behavior: A unifying interpretation with special reference to reward-seeking. Brain Research Reviews, 31(1), 6–41. Irwin, S. (1958). Factors influencing acquisition of avoidance behaviour and sensitivity to drugs. Fed. Proc., 17, 380. Iversen, S. D., & Mishkin, M. (1970). Perseverative interference in monkeys following selective lesions of the inferior prefrontal convexity. Experimental Brain Research, 11, 376–386. Janssen, P. A. J., Niemegeers, C. J. E., & Schellekens, K. H. L. (1965). Is it possible to predict the clinical effects of neuroleptic drugs (major tranquilizers) from animal data? Arzneimittelforschung, 15, 104–117. Jensen, J., McIntosh, A. R., Crawley, A. P., Mikulis, D. J., Remington, G., & Kapur, S. (2003). Direct activation of the ventral striatum in anticipation of aversive stimuli. Neuron, 40, 1251–1257. Joseph, M. H., Datla, K., & Young, A. M. J. (2003). The interpretation of the measurement of nucleus accumbens dopamine by vivo dialysis: The kick, the craving or the cognition. Neuroscience and Biobehavioral Reviews, 27, 527– 541. Kaelbling, L., Littman, M., & Moore, A. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence, 4, 237–285.

Modeling the Functional Role of Dopamine in the Ventral Striatum

391

Kamin, L. J. (1954). Traumatic avoidance learning: The effects of CS-US interval with a trace-conditioning procedure. Journal of Comparative and Physiological Psychology, 47, 65–72. Kandel, E. R., Schwartz, J. H., & Jessell, T. M. (1991). Principles of neural science. New York: Elsevier Science. Kauer, J. A. (2003). Addictive drugs and stress trigger a common change at VTA synapses. Neuron, 37, 549–550. Key, B. J. (1961). The effects of drugs on discrimination and sensory generalisation of auditory stimuli in cats. Psychopharmacologia, 2, 352–363. Kilts, C. D. (2001). The changing roles and targets for animal models of schizophrenia. Biological Psychiatry, 50(11), 845–855. Low, L. A., Eliasson, M., & Kornetsky, C. (1966). Effects of chlorpromazine on avoidance acquisition as a function of CS-US interval length. Psychopharmacologia, 10, 148–154. Low, L. A., & Low, H. L. (1962). Effects of CS-US interval upon avoidance responding. Journal of Comparative and Physiological Psychology, 55(6), 1059– 1061. Maffii, G. (1959). The secondary conditioned response of rats and effects of some psychopharmacological agents. Journal of Pharmacy and Pharmacology, 11, 129–139. Maher, B., & Ross, J. S. (1984). Delusions. In H. E. Adams, & P. Sutker (Eds.), Comprehensive handbook of psychopathology (pp. 383-409). New York: Plenum Press. Masterman, D. L., & Cummings, J. L. (1997). Frontal-subcortical circuits: The anatomical basis of executive, social and motivated behaviors. Journal of Psychopharmacology, 11(2), 107–114. McClure, S. M., Daw, N., & Montague, P. R. (2003). A computational substrate for incentive salience. Trends in Neuroscience, 26(8), 423–428. Miller, R. E., Murphy, J. V., & Mirsky, A. (1957). The effect of chlorpromazine on fear-motivated behavior in rats. Journal of Pharmacol. and Exper. Therap., 120, 379–387. Montague, P. R., & Berns, G. S. (2002). Neural economics and the biological substrates of valuation. Neuron, 36, 265–284. Montague, P. R., Dayan, P., Person, C., & Sejnowski, T. J. (1995). Bee foraging in uncertain environments using predictive hebbian learning. Nature, 377, 725–728. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience, 16(5), 1936–1947. Moore, H., West, A. R., & Grace, A. A. (1999). The regulation of forebrain dopamine transmission: Relevance to the pathophysiology and psychopathology of schizophrenia. Biological Psychiatry, 46, 40–55. Morpurgo, C. (1965). Drug-induced modifications of discriminated avoidance behavior in rats. Psychopharmacologia, 8, 90–99. Nader, K., & LeDoux, J. (1999). The dopaminergic modulation of fear: Quinpirole impairs the recall of emotional memories in rats. Behavioral Neuroscience, 113(1), 152–165.

392

A. Smith, S. Becker, and S. Kapur

Nishijo, H., Ono, T., & Nishino, H. (1988). Single neuron responses in amygdala of alert monkey during complex senosry stimulation with affective significance. Journal of Neuroscience, 8, 3570–3583. O’Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron, 28, 329–337. O’Doherty, J. P., Deichmann, R., Critchley, H. D., & Dolan, R. J. (2002). Neural responses during anticipation of primary taste reward. Neuron, 33, 815–826. Ogren, S. O., & Archer, T. (1994). Effects of typical and atypical antipsychotic drugs on two-way active avoidance: Relationship to DA receptor blocking profile. Psychopharmacology (Berl.), 114(3), 383–391. Pennartz, C. M. A., Groenewegen, H. J., & Silva, F. H. L. D. (1994). The nucleus accumbens as a complex of functionally distinct Neuronal ensembles: An integration of behavioural, electrophysiological and anatomical data. Progress in Neurobiology, 42, 719–761. Phares, V. (2003). Understanding abnormal child psychology. New York: Wiley. Phillips, P. E. M., Stuber, G. D., Helen, M. L. A. V., Wightman, R. M., & Carelli, R. M. (2003). Subsecond dopamine release promotes cocaine seeking. Nature, 422, 614–618. Ponsluns, D. (1962). An analysis of chlorpromazine-induced suppression of the avoidance response. Psychopharmacologia, 3, 361–373. Redgrave, P., Prescott, T. J., & Gurney, K. (1999). Is the short-latency dopamine response too short to signal reward error? Trends in Neurosciences, 22(4), 146– 151. Richards, J. B., Sabol, K. E., & de Wit, H. (1999). Effects of methamphetamine on the adjusting amount of procedure, a model of impulsive behavior in rats. Psychopharmacology, 146, 432–439. Rizley, R. C., & Rescorla, R. A. (1972). Associations in second-order conditioning and sensory preconditioning. Journal of Comparative and Physiological Psychology, 81(1), 1–11. Robbins, T. W., Cador, M., Taylor, J. R., & Everitt, B. J. (1989). Limbic-striatal interactions in reward-related processes. Neuroscience and Biobehavioural Reviews, 13, 155–162. Rolls, E. T., Rolls, B. J., Kelly, P. H., Shaw, S. G., Wood, R. J., & Dale, R. (1974). The relative attenuation of self-stimulation, eating and drinking produced by dopamine-receptor blockade. Psychopharmacology, 38, 219–230. Salamone, J. D., Cousins, M. S., & Bucher, S. (1994). Anhedonia or anergia? Effects of haloperidol and nucleus accumbens dopamine depletion on instrumental response selection in a T-maze cost/benefit procedure. Behavioural Brain Research, 65, 221–229. Salamone, J. D., Cousins, M. S., & Snyder, B. J. (1997). Behavioural functions of nucleus accumbens dopamine: Empirical and conceptual problems with the anhedonia hypothesis. Neuroscience and Biobehavioural Reviews, 21(3), 341– 359. Salamone, J. D., Kurth, P. A., McCullough, L. D., Sokolowski, J. D., & Cousins, M. S. (1993). The role of brain dopamine in response initiations: Effects of

Modeling the Functional Role of Dopamine in the Ventral Striatum

393

haloperidol and regionally-specific dopamine depletions on the local rate of instrumental responding. Brain Research, 628, 218–226. Salamone, J. D., Steinpreis, R. E., McCullough, L. D., Smith, P., Grebel, D., & Mahan, K. (1991). Haloperidol and nucleus accumbens dopamine depletion suppress lever pressing for food but increase free food consumption in a novel food-choice procedure. Psychopharmacology, 104, 515–521. Salamone, J. D., Wisniecki, A., Carlson, B. B., & Correa, M. (2001). Nucleus accumbens dopamine depletions make animals highly sensitive to high fixed ratio requirements but do not impair primary food reinforcement. Neuroscience, 105(4), 863–870. Schmajuk, N. A. (1988). The hippocampus and the classically conditioned nictitating membrane response: A real-time attentional-associative model. Psychobiology, 16(1), 20–35. Schmajuk, N. A., Cox, L., & Gray, J. A. (2001). Nucleus accumbens, entorhinal cortex and latent inhibition: A neural network model. Behavioural Brain Research, 118, 123–141. Schneider, J. S., Sun, Z. Q., & Roeltgen, D. P. (1994). Effects of dopamine agonists on delayed response performance in chronic low-dose MPTP-treated monkeys. Pharmacology, Biochemistry and Behaviour, 48(1), 235–240. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593–1599. Schultz, W., Tremblay, L., & Hollerman, J. R. (2000). Reward processing in primate orbitofrontal cortex and basal ganglia. Cerebral Cortex, 10, 272–283. Seeman, P., & Madras, B. K. (1998). Anti-hyperactivity medication: Methylphenidate and amphetamine. Molecular Psychiatry, 3(5), 386–396. Smith, A., Li, M., Becker, S., & Kapur, S. (2004). A model of antipsychotic action in conditioned avoidance: A computational approach. Neuropsychopharmacology, 29(6), 1040–1049. Solanto, M. V., Abikoff, H., Sonuga-Barke, E., Schachar, R., Logan, G. D., Wigal, T., Hechtman, L., Hinshaw, S., & Turkel, E. (2001). The ecological validity of delay aversion and response inhibition as measures of impulsivity in AD/HD: A supplement to the NIMH multimodal treatment study of AD/HD. Journal of Abnormal Child Psychology, 29(3), 215–228. Stark, H., Bischof, A., & Scheich, H. (1999). Increase of extracellular dopamine in prefrontal cortex of gerbils during acquisition of the avoidance strategy in the shuttle box. Neuroscience Letters, 264, 77–80. Suri, R. E. (2001). Anticipatory responses of dopamine neurons and cortical neurons reproduced by internal model. Experimental Brain Research, 140, 234–240. Suri, R. E. (2002). TD models of reward predictive responses in dopamine neurons. Neural Networks: Special Issue on Computational Models of Neuromodulation, 15(4-6), 523–533. Suri, R. E., Bargas, J., & Arbib, M. A. (2001). Modeling functions of striatal dopamine modulation in learning and planning. Neuroscience, 103(1), 65–85. Suri, R., & Schultz, W. (1998). Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Experimental Brain Research, 121, 350–354.

394

A. Smith, S. Becker, and S. Kapur

Sutton, R. S. (1988). Learning to predict by the method of temporal differences. Machine Learning, 3, 9–44. Sutton, R. S., & Barto, A. G. (1981). An adaptive network that constructs and uses an internal model of its world. Cognition and Brain Theory, 4(3), 217–246. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning. Cambridge, MA: MIT Press. Talk, A. C., Gandhi, C. C., & Matzel, L. D. (2002). Hippocampal function during behaviourally silent association learning: Dissociation of memory storage and expression. Hippocampus, 12, 648–656. Taylor, J. R., & Robbins, T. W. (1984). Enhanced behavioural control by conditioned reinforcers following microinjections of d-amphetamine into the nucleus accumbens. Psychopharmacology, 84, 405–412. Tremblay, L., & Schultz, W. (2000a). Modifications of reward expectation–related neuronal activity during learning in primate orbitofrontal cortex. Journal of Neurophysiology, 83, 1877–1885. Tremblay, L., & Schultz, W. (2000b). Reward-related neuronal activity during go–nogo task performance in primate orbitofrontal cortex. Journal of Neurophysiology, 83, 1864–1876. van der Heyden, J. A. M., & Bradford, L. D. (1988). A rapidly acquired oneway conditioned avoidance procedure in rats as a primary screening test for antipsychotics: Influence of shock intensity on avoidance performance. Behavioural Brain Research, 31, 61–67. Wade, T. R., de Wit, H., & Richards, J. B. (2000). Effects of dopaminergic drugs on delayed reward as a measure of impulsive behavior in rats. Psychopharmacology, 150, 90–101. Wadenberg, M. G., Soliman, A., Vanderspek, S. C., & Kapur, S. (2001). Dopamine D2 receptor occupancy is a common mechanism underlying animal models of antipsychotics and their clinical effects. Neuropsychopharmacology, 25(25), 633–641. Wadenberg, M. L., & Hicks, P. B. (1999). The conditioned avoidance response test reevaluated: Is it a sensitive test for the detection of potentially atypical antipsychotics? Neurosci. Biobehav. Rev., 23(6), 851–862. Wadenberg, M.-L. G., Kapur, S., Soliman, A., Jones, C., & Vaccarino, F. (2000). Dopamine D2 receptor occupancy predicts catalepsy and the supression of conditioned avoidance response behaviour in rats. Psychopharmacology, 150, 422–429. Waelti, P., Dickinson, A., & Schultz, W. (2001). Dopamine responses comply with basic assumptions of formal learning theory. Nature, 412, 43–48. Wilkinson, L. S., Humby, T., Killcross, A. S., Torres, E. M., Everitt, B. J., & Robbins, T. W. (1998). Dissociations in dopamine release in medial prefrontal cortex and ventral striatum during the acquisition and extinction of classical aversive conditioning in the rat. European Journal of Neuroscience, 10, 1019–1026. Wilson, M., & Daly, M. (2004). Do pretty women inspire men to discount the future? Biology letters, 271(S4), 177–179. Wise, R. A. (1982). Neuroleptics and operant behavior: The anhedonia hypothesis. Behavioural and Brain Sciences, 5, 39–87.

Modeling the Functional Role of Dopamine in the Ventral Striatum

395

Wise, R. A. (2002). Brain reward circuitry: Insights from unsensed incentives. Neuron, 36, 229–240. Wise, R. A., & Schwartz, H. V. (1981). Pimozide attenuates acquisition of leverpressing for food in rats. Pharmacology, Biochemistry and Behavior, 15, 655–656. Wise, R. A., Spindler, J., DeWit, H., & Gerber, G. J. (1978). Neuroleptic-induced ”anhedonia” in rats: Pimozide blocks reward quality of food. Science, 201, 262–264. Young, A. M. J., Ahier, R. G., Upton, R. L., Joseph, M. H., & Gray, J. A. (1998). Increased extracellular dopamine in the nucleus accumbens of the rat during associative learning of neutral stimuli. Neuroscience, 83(4), 1175–1183. Received January 9, 2004; accepted July 2, 2004.

LETTER

Communicated by Bruno Olshausen

A Hierarchical Bayesian Model for Learning Nonlinear Statistical Regularities in Nonstationary Natural Signals Yan Karklin [email protected]

Michael S. Lewicki [email protected] Computer Science Department and Center for the Neural Basis of Cognition, Carnegie Mellon University, Pittsburgh, PA 15213, U.S.A.

Capturing statistical regularities in complex, high-dimensional data is an important problem in machine learning and signal processing. Models such as principal component analysis (PCA) and independent component analysis (ICA) make few assumptions about the structure in the data and have good scaling properties, but they are limited to representing linear statistical regularities and assume that the distribution of the data is stationary. For many natural, complex signals, the latent variables often exhibit residual dependencies as well as nonstationary statistics. Here we present a hierarchical Bayesian model that is able to capture higher-order nonlinear structure and represent nonstationary data distributions. The model is a generalization of ICA in which the basis function coefficients are no longer assumed to be independent; instead, the dependencies in their magnitudes are captured by a set of density components. Each density component describes a common pattern of deviation from the marginal density of the pattern ensemble; in different combinations, they can describe nonstationary distributions. Adapting the model to image or audio data yields a nonlinear, distributed code for higher-order statistical regularities that reflect more abstract, invariant properties of the signal. 1 Introduction The goal of many algorithms in machine learning, signal processing, and computational perception is to discover and process intrinsic structures in the data. Extracting these from real signals is a difficult problem, because often the relationships among the observable variables are complex, and there is little a priori knowledge about the types of structures that exist. When some a priori knowledge is available, specialized algorithms can be designed, but this approach is generally less desirable, as it places restrictions on the type of structure that can be learned. Another difficulty is that the dimensionality of the data is often very high, and properties of interNeural Computation 17, 397–423 (2005)

c 2005 Massachusetts Institute of Technology


398

Y. Karklin and M. Lewicki

est lie in a relatively low-dimensional subspace. Because of the inherent variability of most real-world signals, intrinsic regularities are statistical in nature, which makes them that much more difficult to learn. One approach to learning statistical regularities is to formulate a probabilistic model of how the data are generated and adapt its parameters to fit the observed distribution. The adapted parameters reflect the statistics of the data ensemble, while internal representations encode individual data patterns. These models make minimal assumptions about the data and can result in more general representations than those in algorithms tailored for specific tasks or types of data. There are several ways in which data patterns are represented in probabilistic generative models. Distributed representations of linear componential models, such as those for principal component analysis (PCA) and independent component analysis (ICA), are particularly useful for modeling complex high-dimensional data because they can capture independent regularities with independent internal parameters (Bell & Sejnowski, 1995). This makes it possible to model a continuum of different statistical relationships and allows scaling of the algorithms to large numbers of dimensions. Current models, however, are limited in the type of structure they can represent; in order to understand these limitations, it is helpful to look at their mathematical formulation. Linear componential models achieve a distributed representation by describing the data as a combination of linear basis functions (for a review, see Hyv¨arinen, Karhunen, & Oja, 2001; Cichocki & Amari, 2002). This yields a probabilistic generative model in which the data (x) are generated as a linear combination of basis functions (A) weighted by coefficients (u), x = Au.

(1.1)

The likelihood of the observed data under this model is p(x) = p(u)/| det(A)|

(1.2)

(Pearlmutter & Parra, 1996; Cardoso, 1997), and the basis function matrix A is adapted to maximize the data likelihood. The coefficients u are the unknown (latent) variables. They are assumed to be independent and identically distributed (i.i.d.),
p(u) = p(ui ). (1.3) i

The priors p(ui ) are typically chosen to be fixed sparse distributions (although parameters of the prior may be adjusted to maximize data likelihood). Because basis function coefficients are assumed to be i.i.d., the dependence among the data is represented solely by the learned matrix of basis functions.

Learning Density Components

399

The obvious limitation of this model is that its inherent linearity restricts the type of structure it can capture. Even simple, low-dimensional data often exhibit statistical dependencies that cannot be captured by linear transformations. In many applications, data are complex and rich with statistical structure, and latent variables of linear models adapted to these data exhibit significant residual mutual dependence (Hyv¨arinen & Hoyer, 2000; Schwartz & Simoncelli, 2001; Karklin & Lewicki, 2003). Another shortcoming of these models is that they assume that the statistical regularities in the data do not change; they describe stationary probability distributions. For example, once model parameters are adapted in ICA, both the prior and the basis functions are fixed, leading to a stationary distribution over the data. This does not depend on the form of the prior and also applies to models with adaptive or entirely nonparametric priors. In many domains, however, the statistics of the data are known to change, as the physical properties of the environment or conditions for data acquisition vary. While the stationary prior assumption gives a valid approximation of true density over a large enough corpus of training data, it does not reflect the variation across contexts that is observed in many signals. Figure 1 illustrates nonstationary statistics observed in images of natural scenes. ICA basis functions were adapted to 20 × 20 patches taken from an ensemble of natural images. Over the full ensemble of the training data, the basis function coefficients have marginal distributions that are consistent with the prior assumed by the model (not shown). However, computing coefficient histograms over particular image regions reveals systematic deviations from the (globally valid) stationary distribution. Patterns in the histograms suggest that basis functions of certain orientations are more active in some parts of the image (e.g., textured, oriented surface of the log), while in other regions, different subsets tend to be activated. This is observed in other types of data as well: temporal basis functions adapted to speech also yield coefficients whose statistics vary greatly across local regions of the signal (see Figure 2). Figures 1 and 2 give just a few examples of patterns in latent variable distributions that depend on the local context. In fact, there is a wide range of statistical regularities in complex data, a continuum of contexts that is as multidimensional as the physical properties of the environment that give rise to it. Local representations, as employed by clustering or mixture model techniques, assume that the contexts are discrete and thus cannot describe regularities that arise from a combination of different contexts. A model that captures this variation must form flexible, distributed representations of higher-order structure. Moreover, because the dimensions of the contexts are not known a priori, the model must be able to automatically discover this underlying structure. Finally, many previous models of nonstationary distributions have relied on the assumption that data statistics vary smoothly from sample to sample (Everson & Roberts, 1999; Pham & Cardoso, 2001) and computed local estimates of context-dependent variation.

400

Y. Karklin and M. Lewicki

Figure 1: The distribution of ICA basis function coefficients exhibits nonstationary statistics that reflect local image structure. (a) A subset of image basis functions learned from an ensemble of natural images, ordered by orientation. The small black square on the image indicates the size, relative to the image, of the learned basis functions. (b) Coefficients of independent components were computed over two regions of an image. (c, d) Histograms of the coefficients for the two regions reveal patterns in the joint distributions. Each histogram in the 10 × 10 grid in c and d corresponds to a basis function at the same grid position in a and is normalized so that the filled area sums to 1. Different types of local image structure produce different patterns in the joint activities. For example, the image region containing the log yields higher coefficient variation for basis functions oriented along the grain and matching the approximate spatial frequency of the wood texture.

Learning Density Components a

401

256

128

1

b

R1

R2

R1

log(var(u))

c

R3

R2

R3

4

4

4

0

0

0

−4

−4

−4

−8

1

128

256

−8

1

128

256

−8

1

128

256

Figure 2: ICA basis functions adapted to speech data also exhibit nonstationary statistical dependencies. (a) A subset of 256 ICA-derived basis functions ordered by dominant frequency. (b) Each basis function was convolved with three different regions of a speech signal. The length of the basis function is indicated by the short bar above the start of the speech signal. (c) The variances of coefficients sampled over the three regions, with the 256 coefficients ordered by frequency as in a. Although all basis function coefficients have unit variance when sampled over the whole data ensemble, local regions show characteristic variance patterns that reflect local signal structure.

This assumption does not always hold; even spatially and temporally coherent data exhibit abrupt changes that cannot be modeled as slowly evolving processes. Here we address the limitations of previous models with a hierarchical Bayesian model that forms a distributed code of higher-order statistical regularities and captures nonstationarities in the data distribution. The model is a generalization of ICA; thus, we begin with a standard linear componential model in which the data are generated as a combination of linear basis functions. However, instead of assuming that the basis function coefficients are independent (and their joint prior distribution is factorable; see equation 1.3), we explicitly model the dependence among hyperparameters of their priors. In order to capture variable, context-dependent activation of

402

Y. Karklin and M. Lewicki

basis functions, the dependence is specified through the scale parameters governing the width of the prior (and hence the variance of the coefficients). This dependence is modeled with a set of density components, a distributed code that describes the shape of the joint density of the linear coefficients and captures patterns in the variances of the coefficients as observed in the motivating examples. Each density component describes a common underlying deviation from the standard assumption of independence (the i.i.d. joint prior) associated with a frequently encountered context. Using a weighted combination of density components, the model is able to represent a continuum of contextdependent changes in probability distributions. Adapting the set of density components and modeling their activation with a sparse prior yields a compact description of higher-order statistical regularities of the data ensemble. Unlike other recent methods, the model makes no assumptions of temporal or spatial coherence; it is able to infer, independently for each data sample, the higher-order code that describes the generating distribution. Below we present the probabilistic framework for the model and describe the associated learning algorithms. Previously, we have used this model to discover higher-order structure in natural images (Karklin & Lewicki, 2003). Here, we describe the algorithm in more detail and frame it as a general method of statistical density estimation for high-dimensional nonstationary data. We verify the recovery of correct model parameters using a toy data set, apply the learning algorithm to a wider range of data types, and show how the learned higher-order code accounts for observed dependencies. We provide results and analysis for photographs of natural scenes, scanned images of newspapers, and speech waveforms. However, the model is not tailored specifically to images or audio data, and can be used to automatically learn the nonlinear statistical dependencies in any data set with sufficiently rich structure. 2 A Hierarchical Model for Nonstationary Distributions Our model is a generalization of previous linear models. Hence, we begin by assuming that each data vector is generated as a combination of linear basis functions, x = Au. As in standard ICA models (e.g., Cichocki & Amari, 2002), basis function coefficients are assumed to be sparsely distributed. Here we use a generalized gaussian distribution with zero mean: p(ui ) = N (0, λi , qi )    qi   ui  = zi exp −   , λ

(2.1) (2.2)

i

where zi = qi /(2λi [1/qi ]) is a normalizing constant. The parameter qi determines the weight of the distribution’s tails and can be estimated from the data; in many ICA applications, the coefficients tend to be sparse, making

Learning Density Components

403

their distributions supergaussian (qi < 2). Typically, the scale parameter λi is fixed to a constant, since the basis functions in A can themselves scale to fit the data. In order to capture residual dependence among coefficients u, we must abandon the assumption of fixed, independent priors. The motivating examples suggested that intrinsic structures in the data give rise to patterns in the scales of the coefficients (similar dependencies have been observed previously in wavelet coefficients; Simoncelli, 1997). A natural way to model this is through the scale parameters of the prior, which we model as a nonlinear transformation of latent higher-order variables. Specifically, we use a matrix of density components B and density component coefficients v to describe the logarithm of the scale parameter, log(λ/c) = Bv.

(2.3)

 If we define the constant c = (1/q)/ (3/q), the variance of the coefficients becomes 1 when the right side of the equation is 0 (this becomes convenient when a zero-centered prior is selected for the distribution of v; see below). The joint prior distribution of coefficients u can now be expressed as − log p(u|B, v) ∝

   [Bv]i +  i

qi  ui  , c exp([Bv]i ) 

(2.4)

where [Bv]i represents the ith element of the vector Bv (see the appendix for the derivation). Basis function coefficients are assumed to be independent conditional on  the higher-order variables, p(u|v) = p(ui |v). This accounts for the dependence in the magnitudes of basis function coefficients. The new form of the prior (2.4) implies that if v is 0, the model reduces to standard ICA in which the linear coefficients are independent and identically distributed with variance equal to 1. Nonzero values of v scale and combine density components (columns of B) that define patterns in the distributions of u. Because each vi can be positive or negative, each density component represents contrast in the magnitudes of coefficients u (see Figure 3). We place a nongaussian, sparse prior on the latent variables v and infer their values for each data sample.1 This means that a priori, we assume that the activity of density component coefficients is sparse, and relatively few components are needed to describe how the generating distribution associated with each data sample differs from the i.i.d. ICA model. Using this parameterization, we adapt the density components to the entire data ensemble, which produces a compact description of higher-order statistical regularities. 1 A Laplacian prior was used in the simulations, but other distributions may be more appropriate.

404

Y. Karklin and M. Lewicki

Bj

p(u|v j = 1) p(u|v j = 0) p(u|v j = −1) Figure 3: Each density component defines a pattern in the joint distribution p(u). The plot at the top shows an example nine-dimensional density component Bj . The distributions of coefficients u1,...,9 are shown for different values of vj . Here we show only a single density component Bj , whereas the model adapts a set of them B = {B1 , B2 , . . . , BM } to obtain a compact description for common scale patterns in the data.

The full generative model is shown in graphical form in Figure 4. There are two sets of random variables that give rise to the data, v and u, and two sets of parameters adapted to the data: the linear basis functions A and the density components B. A crucial difference between this generative form and several other models that account for higher-order dependence is that here, the density components specify a distribution over the coefficients, as opposed to exact values or pooled magnitudes, which have been used in other models (Hoyer & Hyv¨arinen, 2002; Welling, Hinton, & Osindero, 2003). Thus, the model forms a hierarchical representation in which the lower-level codes data values precisely and the higher level represents more abstract properties associated with the shape of the data distribution. 3 Inference of Density Component Coefficients For each data sample, it is necessary to compute the higher-order representation v that best describes the pattern in the scale of coefficients u. This transformation is nonlinear and cannot be expressed in closed form. Here, we compute the best value of v by maximizing the posterior distribution, vˆ = arg max p(v|u, B), v

= arg max p(u|B, v)p(v). v

(3.1) (3.2)

Learning Density Components

405

vi ∼ N (0, 1, qi) λ j = c exp[Bv] j u j |λ j ∼ N (0, λ j , q j ) xk =

∑ Ak j u j j

Figure 4: Schematic of the hierarchical generative model. Sparsely distributed random variables v specify (through a nonlinear transformation) the scale hyperparameters λ for the distribution of coefficients u. The data x are a linear combination of coefficients u. Matrices A and B are parameters that are adapted to the statistical distribution of the data.

 We assume that vi ’s are independent (p(v) = i p(vi )) and sparsely distributed (log p(vi ) ∝ −|vi |). For the simulations below, vˆ was derived by gradient ascent. We used second-order methods (LeCun, Bottou, Orr, & Muller, ¨ 1998) to stabilize and speed up convergence to optimal estimates. Because the prior is zero centered and sparse, only a few nonzero values will contribute to the representation of each data sample. The inference of optimal density component coefficients is analogous to estimating sample variance based on a single observation, but the problem is further constrained by the structure of the learned density components. Because the model is constrained to describe the pattern of variance with a sparse combination of density components, the value of v for a typical pattern is usually well determined. In addition, the high dimensionality of the input facilitates the inference process, as it provides more directions of variation that make up the variance pattern. 4 Adapting Model Parameters to the Data The linear basis functions and the density components are adapted to the data ensemble by maximizing the posterior p(A, B|X). We assume that samples in the data ensemble X = {x1 , . . . , xN } are independent, so that p(X|A, B) =

N


p(xn |A, B).

(4.1)

n=1

For each data sample x, the posterior distribution is p(A, B|x) ∝ p(x|A, B)p(A, B) = p(u|B)p(B)/| det(A)|.

(4.2) (4.3)

406

Y. Karklin and M. Lewicki

Ideally, the marginal distribution p(u|B) would be computed by integrating over v, but evaluating this integral for equation 2.4 is intractable. Here we approximate it using the maximum a posteriori estimate v: ˆ  p(u|B) =

p(u|B, v)p(v)dv,

≈ p(u|B, v)p( ˆ v). ˆ

(4.4) (4.5)

Substituting this approximation into the posterior gives p(A, B|x) ∝ p(u|B, v)p( ˆ v)p(B)/| ˆ det A|.

(4.6)

The prior on B places a small a priori bias for small values of Bi,j and eliminates the problem of a degenerate case in which B grows without bounds while v’s rescale to be smaller. For the results here, we assumed Bi,j followed a gaussian distribution. The matrices A and B can be optimized iteratively by maximizing p(A|X, B) and then maximizing p(B|X, A). In this case, the first step amounts to performing ICA in which the priors incorporate the scale estimates v. ˆ Alternatively, we can assume that optimal linear basis functions are largely independent of the set of density components, and optimize B using a fixed A. For computational efficiency, A and B were assumed to be independent and were adapted separately in the simulations described below. We confirmed the validity of this approach by training a model on data of reduced dimensionality and with fewer density components; results were qualitatively similar to optimizing the parameters independently. In order to verify that the learning algorithm produces a valid solution, we adapted model parameters to an artificial data set for which the optimal solution was known. The data were generated by constructing a set of density components and then sampling basis function coefficients according to p(u|B). An illustration of the process and the obtained results is shown in Figure 5. Optimizing density components from random initial values produced a matrix that was identical (up to a permutation of its columns) to the true model parameters (see Figures 5a and 5b). The patterns in the learned density components specify nonlinear dependencies among coefficient magnitudes; in fact, there are no linear correlations among basis function coefficients sampled from the model (even when the same v is used to generate the coefficients). Linear models like ICA are unable to recover these statistical regularities. As a control, we adapted the density component model to a pure noise data set in which coefficients u were random samples from independent sparse distributions. In this case, no regularities in the magnitudes of coefficients existed, and the resulting density components consisted of small, random values.

Learning Density Components

407

5 Discovering Structure in Complex Data 5.1 Learned Density Components. We optimized model parameters on several data sets and analyzed the learned density components. For computational simplicity, the model was optimized in two stages in all the simulations. First, a complete linear basis A was adapted to the data using standard methods; next, the density component matrix B was optimized on the coefficients of the fixed A. Since the linear basis functions were learned using standard ICA methods, our analysis and discussion here is limited to the recovered matrix of density components. The density components were initialized to small random values, and gradient ascent was performed on stochastically sampled batches of data. The maximum a posteriori estimate vˆ was obtained using 20 steps of gradient ascent. Convergence of the gradient procedures for the optimization of B and estimation of vˆ was tested in a number of ways, including varying the step size, the number of iterations, and the initial conditions. The given optimization parameters yielded reasonable speed and accuracy, as well as consistent solutions for different random initial conditions. We first applied the learning algorithm to small (20 × 20) image patches sampled from a standard set of 10 gray-scale images of natural scenes (Olshausen & Field, 1996; Karklin & Lewicki, 2003). We used a complete set of 400 linear basis functions. The number of density components was set to 100 (although the algorithm is able to recover any number that yield a sparse distribution for coefficients v). We used batches of 1000 samples for 35,000 iterations of gradient ascent with a fixed step size of 0.3. Statistical regularities of the data ensemble are captured in the matrix of density components. In order to analyze the structure described by this matrix, we need to examine its weights as they relate to the basis functions whose distributions they affect. (Recall that each weight in a density component vector specifies how a particular p(ui ) is rescaled). The initial ordering of basis functions in the learned matrix A is arbitrary; hence, weights in B also appear random in their original ordering. However, we can rearrange the weights in B according to some property of the linear basis functions and examine whether the learned density components capture structure related to the chosen property. For example, ICA basis functions adapted to natural images are spatially localized; arranging density component weights according to the location of corresponding basis functions within the image patch reveals patterns in their organization (see Figure 6). Thus, density components that appear structured in this arrangement specify dependence among spatially related linear basis functions. As parameters in the generative model, they describe common data distributions that reflect localized image structure. Some density components also appear random when arranged spatially, but these often show organization along other dimensions of the lower-order representation, such as orientation or spatial frequency (Karklin & Lewicki, 2003). Changing the number of density components

408

Y. Karklin and M. Lewicki

does not affect the type of structure captured by the hierarchical model. A larger number of density components allows the model to represent finerscale spatial regularities, as well as other statistical structure that is not as obvious to interpret. We also applied the model to speech data from the TIMIT database. Linear basis functions were adapted to bandpass filtered speech segments of 256 samples (16 msec of 16 kHz sound). The number of density components was set to 100, and the parameters were optimized using stochastic learning on data batches of 1000 for 10,000 iterations. A representative set of the learned density components is shown in Figure 7. In order to display the weights in the density components as they relate to the linear code, we first computed the Wigner distributions (WD) of the linear basis functions using the DiscreteTFDs Matlab package (O’Neill, 1999). The Wigner distribution of a basis function is a surface in the time-frequency space; we took a contour at 95% peak value for each basis function and drew all these contours on a single time-frequency plot (time on the horizontal axis, 0 to 16 msec, and frequency on the vertical axis, 0 to 8 kHz). Because the linear basis functions adapted to speech tile most of the the time-frequency space, the contours also exhibit relatively even tiling of the plots. In Figure 7, nine WD plots show the weights in nine density components to the same set of linear basis functions. Here, as in image density components, the shading of each patch corresponds to the value of the weight. Some density components

Figure 5: Facing page. The model correctly recovers the density components used to generate synthetic data. We constructed a 50 × 10 matrix B composed of 10 cosine-shaped density components (a). After 3000 iterations, the model recovers (up to a permutation) the correct density components (b). (c) The generative and inference steps of the algorithm. (1) Three 10-dimensional density component coefficients are drawn from a sparse distribution; (2) each v(i) specifies a vector of (i) (i) scaling variables λ through the nonlinear transformation λ = c exp[Bv](i) . (3) The scaling variables are hyperparameters for nonstationary distributions p(u), from which data samples u are drawn. In order to emphasize that each (i) vector of scaling variables λ specifies a distribution, not fixed values of u, we (i) plotted several u’s drawn from the distribution p(u|λ ). In actual simulation, each data point was generated independently. Using the learned density comˆ were obtained for each data sample. Because ponents, estimates of (4) vˆ and (5) λ the inference problem involves the estimation of density parameters from sinˆ only approximately match true parameters. Although gle data points, vˆ and λ the complete hierarchical model includes another transformation x = Au, the projection to data space x is linear and is not necessary for inference of vˆ when coefficients u are known. The scatter plot of 1000 samples of u1 and u2 drawn from the model (d) shows that there is no linear dependence among basis function coefficients.

Learning Density Components

true B

a

c (1)

(5)

v ∼ p(v)

vˆ

409

b

(2)

(4)

λ = c exp(Bv)

learned B

(3)

λ) u ∼ p(u|λ

λˆ d

uj

ui

410

Y. Karklin and M. Lewicki

Figure 6: Density components optimized on an ensemble of 20 × 20 image patches drawn from natural scenes. Each column of B is represented here as a square; its weights to 400 image basis functions are plotted as dots, placed in locations corresponding to the center of each image basis function in the image patch. Each dot is colored according to the value of the weight, with white indicating positive weights, black negative weights, and gray weights that are close to zero. Most density components describe spatial relationships and capture coactivation of linear basis functions localized to a particular area of the image patch. For example, the density component in the second row, second column indicates whether contrast in the image patch is localized to the top or the bottom half. While most density components represent location, orientation, or spatial frequency regularities, the organization of some is not obvious.

describe coactivation of linear basis functions of adjacent frequency bands, while others are localized in time within the sample window. Most density components capture periodic higher-order structure and regularities across

Learning Density Components

411

Figure 7: A subset of density components of speech. The weights in a column of B are plotted as shaded patches in one of the nine panels. Each patch is placed according to the temporal and frequency distribution of the associated linear basis function and shaded according to the value of the weight, with white indicating positive weights, black negative weights, and gray weights that are close to zero. The axes represent time, 0 to 16 msec, horizontally, and frequency, 0 to 8 kHz, vertically. The density components form a distributed representation of the frequency of the signal and the location of energy within the sample window. Density components coding for multiple frequencies might capture harmonic regularities in the speech signal (see the text for details).

412

Y. Karklin and M. Lewicki

multiple frequencies or time intervals, and a few are tuned specifically to subtle shifts in dominant frequency over the sample window. 5.2 Higher-Order Code. In order to better understand the type of structure captured by the model, it is informative to look at the higher-order code—the coefficients of density components—and the statistical regularities it represents. Individual density component coefficients indicate the presence, in each data sample, of the type of structure represented in Figure 6. As a distributed code, their joint activity describes the data density whose shape reflects underlying structure in the data. Figure 6 shows that among other statistical regularities, the higher-order code captures spatial relationships in the data. How does this representation compare to the lower-level, linear code for image structure? The activity of density component coefficients over contiguous regions of the data suggests that the higher-order representation captures more abstract properties of the data (see Figure 8). When a sliding window is applied to a natural scene image, the resulting lower-level representation changes rapidly from sample to sample, as would be expected from what are essentially outputs of linear filters. The higher-order representation varies more slowly over the image and captures more invariant properties of the data, such as overall image contrast or the dominance of certain spatial frequencies. Also shown in Figure 8 are the values of the linear and the density component coefficients for a model trained on images of newspaper text. Here too the density component coefficients describe more abstract properties: several combine to form a distributed representation of text line position in the image patch (the activity of one such coefficient is shown in the first panel of Figure 8f), while others represent commonly observed structures in the data, such as recurring shapes of letters or blank spaces between words. Applied to audio data, the model also captures more abstract properties of the stimulus. In Figure 9a, we plot an example audio signal, along with the activities of three linear coefficients in Figure 9b and three density component coefficients in Figure 9c. We emphasize that, as for the images, the model is trained on segments drawn randomly from the data set, and the values of the coefficients for each sample position in the signal shown in the figure are determined independently. The higher-order representation varies more slowly than responses of the linear filters and captures structural elements that extend well beyond the small sampling window. This may reflect a general property of natural signals—fast fluctuations in their exact values are caused by interactions of underlying physical properties, which themselves change more slowly. 5.3 Modeling Residual Dependencies. The motivating examples (see Figures 1 and 2) showed specific types of residual dependencies among the “independent” linear coefficients, such as the dependence among the scale of coefficients, which formed patterns that changed from context to context.

Learning Density Components

413

Figure 8: The higher-order code captures more abstract properties of image data and therefore forms a more invariant representation than the coefficients of linear basis functions. We trained the model on natural images (a–c) and scanned newspaper clippings (d–f) and analyzed the representation formed by the model as it varied over the images. A sliding window (represented as white squares in the images) was applied over contiguous sections of the training data (a,d), and values of three linear coefficients ui (b,e) and three higher-order coefficients vj (c,f) were plotted as they varied over the signal. White represents large pos values, black large negative values, and gray zeros. Although the model is trained on image patches selected randomly from the data set, the higher-order code forms a representation that changes more slowly over space and captures properties of the data that extend beyond the sampling window, such as the overall contrast in natural images or the position of the text-line in newspaper images.

The adapted hierarchical density component model is able to capture these dependencies. First, drawing from the model generates data with similar

414

Y. Karklin and M. Lewicki

a b

c

Figure 9: The higher-order representation formed by the hierarchical model trained on speech data is more invariant than simple outputs of linear filters. A sliding window was applied to a speech signal (a; size of window indicated by a short bar). At each point, the linear basis function coefficients u were computed (b) and the higher-order coefficients v were inferred (c). Values of v change slowly and represent more abstract properties, such as the presence of silence or the onset of vocalization. Only three examples for u and v are shown.

statistical regularities. Furthermore, the higher-order representation in the model defines an implicit normalization of the linear code, and the residual dependencies are no longer observed in the normalized code. Figure 10a shows the empirical joint distributions (top row) of two linear coefficients when sampled from the image regions R1 or R2 of Figure 1. In the two contexts, the shape of the distribution is different: the coefficients have high variance in one context but not in the other. The statistical properties in the two contexts are captured by the inferred density component coefficients. Fixing the density component coefficient to the empirical distributions and sampling the linear coefficients reveals the same type of statistical structure (middle row). At the same time, it is possible to use the estimated parameters of the generating distribution to normalize the data. ˆ results Dividing the linear coefficients by the estimated scale parameters λ in joint distributions that are symmetric with uniform variance across different contexts and image regions (bottom row). Another way to observe dependence among coefficient magnitudes is to draw a conditional histogram that plots distributions of one coefficient conditional on different values of another (Simoncelli, 1997; Schwartz & Simoncelli, 2001). While the joint histograms show that coefficient magnitudes are dependent on the sampling context, conditional histograms reveal pair-wise dependencies between coefficients across all contexts. For natural images, most linear coefficients show a positive magnitude dependence; the

Learning Density Components

415

Figure 10: Dependence in the magnitudes of linear basis function coefficients is captured by the density component model. (a) The joint distributions of linear coefficients are different in the two image regions from Figure 1, that is, the data distribution is not stationary. Sampling from the model under the estimated higher-order representation of each context results in similar distributions. Normalizing the image data by the estimated scale parameters, u¯i = ui /λi , eliminates the non-stationarity. (b) Over the full data ensemble, empirical conditional histograms for pairs of coefficients show statistical dependencies in the magnitude. Sampling from the model adapted to this data ensemble produces similar dependencies, and normalizing by the estimated scale parameters removes the magnitude correlations. See the text for more details.

magnitude of one coefficient is positively correlated with the magnitude of another, (e.g., the left pair in Figure 10b), but some exhibit the reverse pattern. Sampling from the model produces data with the same statistical dependencies (see Figure 10b, middle row), while normalized linear coefficients show no conditional magnitude dependence (see Figure 10b, bottom row). Joint and conditional histograms illustrate pair-wise structure in the linear coefficients; global patterns in coefficients, such as those observed in Figures 1 and 2, are also captured by the model. In the top row of Figure 11, we replot the statistics from Figure 2 that show variance patterns in different regions of the speech signal. In the bottom row, we plot the same statistics for the coefficients normalized by the estimated scale parameters; after nor-

416

Y. Karklin and M. Lewicki R

R

log(var(u))

1

3

4

4

0

0

0

−4

−4

−4

−8 1 4

log(var(u)) ¯

R

2

4

128

256

−8 1 4

128

256

−8 1 4

0

0

0

−4

−4

−4

−8

1

128

256

−8

1

128

256

−8

1

128

256

128

256

Figure 11: The model accounts for nonstationary statistics of coefficients. (Top row) Log variance of u for the three regions in the speech signal from Figure 2b. Each plot shows the log variance of 256 basis functions, sorted by dominant frequency (replotted from Figure 2c). (Bottom row) Log variance of the normalized basis function coefficients u¯i = ui /λˆ i .

malization, the statistics are stationary, and the coefficients are identically distributed. The same global normalization effect is observed for natural images (plots not shown). 6 Discussion Some previous work has focused on extending linear probabilistic models. Mixtures of linear ICA models have been used to describe high-dimensional, nongaussian data drawn from distinct classes (Lee, Lewicki, & Sejnowski, 2000; Lee & Lewicki, 2002). In this approach, the number of classes is specified in advance, and an optimal linear basis is learned for each class. This nonlinear generative model describes different data distributions for different classes, but its higher-order representation is fundamentally local and does not scale well in domains where the variation in higher-order structure is continuous and high-dimensional. A key problem addressed by the model presented here is the presence and interaction of multiple instrinsic structures, and this is achieved by a continuous, distributed higher-order code. Other models have extended ICA to handle nonstationary data distributions. Everson and Roberts (1999) proposed a model in which ICA basis functions evolve with time as a first-order Markov diffusion process. Similarly, Pham and Cardoso (2001) developed and Choi, Cichocki, and Belouchrani (2002) extended algorithms for non-stationary models in which

Learning Density Components

417

the variances of the sources modulate slowly in time. These are also related to models of time-varying mean and variance in economics (Bollerslev, Engle, & Nelson, 1994), and typically model data whose statistics change slowly over time or space. Alternatively, one can describe the variance with a sparse but temporally coherent latent variable (Hyv¨arinen, Hurri, & V¨ayrynen, 2003). However, in many cases, real-world data are subject to both smooth and abrupt changes that do not follow diffusion dynamics or smooth amplitude modulation. In contrast to these approaches, the density component model makes no assumptions of temporal or spatial smoothness. It infers an optimal generating distribution for each data sample based on only the values of that sample, though the inference process is constrained by parameters adapted to the statistical regularities of the entire data ensemble. Thus, it is able to capture both smooth and abrupt changes in the underlying structure. Another approach to capturing intrinsic structures in the data has been to incorporate a specific nonlinearity, such as the sum of squares (Kruger, ¨ 1998; Hoyer & Hyv¨arinen, 2002) or sigmoid functions (Lee, Koehler, & Orglmeister, 1997). The drawback to these models is that the type of structure learned is limited by the specific choice of the nonlinearity. Most of these methods also assume a fixed linear representation (e.g., a set of oriented, localized 2D basis functions for image models), and those that adapt the linear representation assume a more constrained form of the nonlinear dependence (see below). In the model presented here, the linear basis is adapted to the data and maximizes the statistical independence of the linear representation. This ensures that the statistical regularities captured by the higher-order code represent fundamentally nonlinear dependencies rather than residual dependence resulting from the choice of a suboptimal linear basis. Furthermore, in some applications, there is no clear choice of linear representation (such as Gabor filters or wavelets in image processing); in such cases, it is sensible to derive the linear code from the statistics of the data. Several earlier models have explicitly represented the dependence among coefficients of linear basis functions. In the subspace ICA model (Hyv¨arinen & Hoyer, 2000), the linear basis functions are grouped into neighborhoods and adapted to maximize the independence of the vector norms of the neighborhoods. Basis functions within a neighborhood are no longer assumed to be independent; in fact, the energies of their coefficients are correlated. In the more generalized form of the model, called topographic ICA, the disjoint sets of dependent basis functions are replaced by a topographic arrangement that defines magnitude dependencies among basis functions (Hyv¨arinen, Hoyer, & Inki, 2001). The generative forms of subspace ICA and topographic ICA can be interpreted as more constrained versions of the density component model presented here. Neighborhood or topographic dependencies can be equivalently represented by density components whose weights are specified in advance to reflect tree-dependent or topographic relationships. The density component model, however, places no such constraints on the

418

Y. Karklin and M. Lewicki

higher-order representation; thus, density components adapted to the data can capture nontopographic dependencies as well. A related set of work has attempted to model the dependence among coefficients of a fixed linear transform, such as a multiscale wavelet decomposition. Romberg, Choi, and Baraniuk (2001) used a set of discrete latent variables, propagated along a multiscale wavelet tree, to describe the distribution of each wavelet coefficient. The transition probabilities of the latent states were adapted to match the scale dependencies between adjacent nodes in the tree. Buccigrossi and Simoncelli (1999) computed a linear predictor of scale for each coefficient as a function of the magnitudes of its neighbors. Wainwright, Simoncelli, and Willsky (2001) extended this approach by modeling the wavelet coefficients as observed variables in a gaussian scale mixture, in which random gaussian variables are multiplied by latent scaling variables. Dependence among coefficients adjacent on the wavelet tree is captured through the structure of a gaussian process defined on the scaling variables. In addition to its reliance on a fixed linear representation (the drawbacks of this are outlined above), this model is limited in that it can only describe pairwise dependencies between variables adjacent on the wavelet tree. Adapting a model to learn global statistical regularities, as opposed to local representations of class structure or pairwise dependence, allows it to capture a wider range of intrinsic structures. Also, learning an efficient basis to describe these dependencies facilitates their interpretability and provides a better fit to the underlying structure. 7 Conclusion We have introduced a hierarchical, generative Bayesian model that can be considered a nonlinear extension to ICA. It uses parametric density estimation to learn statistical regularities from the data and makes no assumptions about the type of structure it expects to find. The model is general, it is not specific to any domain and can be applied to any data set with rich statistical structure. Because the model forms distributed representations at all levels of its hierarchy, it scales well to large-dimensional data. Adapted to patches from natural images or samples from speech data, the density component model was able to learn nonlinear statistical regularities. It yielded a distributed representation of context, which included higher-order spatial relationships for image data and frequency and harmonic structure for audio data. Sampling from the model produced data with the same statistical regularities observed in the training data sets and the model’s implicit normalization of the lower-order code accounted for the residual dependencies observed in various data sets. Recently, it has been argued that higher-order properties of natural signals change slowly across time or space and that this spatial and temporal coherence can be used to extract higher-order structure from the data (Foldiak, 1991; Kayser, Einh¨auser, Dummer, ¨ Konig, ¨ & Kording, ¨ 2001; Wiskott & Se-

Learning Density Components

419

jnowski, 2002; Hurri & Hyv¨arinen, 2003). We show that in some cases, simply learning higher-order statistical regularities in the data leads the model to recover more abstract properties that tend to vary slowly with time or space. This raises the possibility that the explicit computational goal of extracting coherent (slowly changing) parameters is helpful, but not necessary to learning intrinsic structures that underlie the variation in the data. One result of learning global statistical regularities is that the learned structure is not necessarily obvious; for example, density components adapted to natural images describe a variety of statistical regularities, some of which are not easily interpreted. This is true for many unsupervised learning models that do not specify in advance the structure to be learned. For example, ICA applied to natural images yields a matrix of basis functions whose functional interpretation has ranged from edge detectors (Bell & Sejnowski, 1997) to models of biological sensory systems (van Hateren & van der Schaaf, 1998). The work presented here suggests that as more powerful unsupervised learning models are developed, the analysis of learned parameters and data representations will gain in importance. The approach taken in this work is to attack a difficult problem—capturing intrinsic regularities in complex high-dimensional data—incrementally. Although the model is able to capture some nonlinear statistical regularities, the structure it learns is still quite low level. This step-wise approach stands in contrast to other computational schemes that solve specific problems, such as perceptual invariance or scene segmentation. This may prove more tractable and robust because it does not rely on preconceived notions of intrinsic structures but learns them from the data. This approach might also give more insight into the organization of biological perceptual systems, where each processing unit performs a relatively simple computational task, and many computational goals might be achieved incrementally and in parallel. Appendix The value of vˆ for a given u was obtained by maximizing the log posterior distribution L = log p(v|u, B) ∝ log p(u|B, v)p(v).

(A.1)

We use the Laplace distribution for the prior on v and a generalized gaussian distribution with the scale parameters λ for the likelihood p(u|B, v), so that L ∝ log ∝

  qi 
  qj  M  ui   vj  zi exp −   zj exp −   λi c i=1 j=1

N


M  log i=1

 qi   qj

M  ui   vj  qj qi log −   + −   2λi (1/qi ) λi 2(1/qj ) c j=1

(A.2)

(A.3)

420

Y. Karklin and M. Lewicki

∝

 qi  N M  qj   ui  v  j , − log λi −   −  λi c i=1 j=1

(A.4)

where z = q/(2λ(1/q)) is the normalization term, λi = ce[Bv]i , and c =  (1/q)/ (3/q). For a given data sample, u is the N × 1 vector of linear basis function coefficients and v the M × 1 vector of density component coefficients. A is the N × N matrix of linear basis functions, and B is the N × M matrix of density components. We use [Bv]i to denote the ith element of the vector Bv, and Bi to denote the ith row of the matrix B. The MAP estimate vˆ was obtained by gradient ascent,    qi  N M  qj  ui  vj ∂  ∂L    − log λi −   − = c ∂vj ∂vj i=1 λi j=1 =

N  u qi

 |vj |qj −1  i  . −Bij + qi Bij  [Bv]  − sign(vj )qj cqj ce i i=1

(A.5)

(A.6)

The gradient ascent procedure was sensitive to initial conditions and in some cases did not converge to a solution. We tried several alternatives, including a closed-form approximation to the MAP estimate. Ultimately, the most effective learning method was to adjust the step size by the stochastic estimate of the Hessian over each batch of data (LeCun et al., 1998): ηj =

2

∂∂vL2  j

+µ

,

(A.7)

where µ is a small constant that improves stability when the second derivative is very small. The second derivative for a data sample is given by N  u qi  |vj |qj −2 ∂ 2L  i  =− q2i B2ij  [Bv]  − qj (qj − 1) qj . 2 c ce i ∂vj i=1

(A.8)

The density component matrix B was estimated by maximizing the posterior over the data ensemble, log p(B|x1 , . . . , xN , A) ∝ log p(x1 , . . . , xN |A, B)p(B)  log p(xn |A, B)p(B) ∝

(A.9) (A.10)

n

∝



log p(un |B, vˆ n )p(vˆ n )p(B)/| det A|. (A.11)

n

Let Lˆ n = log p(un |B, vˆ n )p(vˆ n )p(B). We place a gaussian prior on B and im plement gradient ascent B = N1 n ∂ Lˆ n /∂Bij , where the posterior for each

Learning Density Components

421

data sample xn is  ∂  ∂ Lˆ log p(un |B, vˆ n ) + log p(vˆ n ) + log p(B) = ∂Bij ∂Bij     qi

N,M N M  qj   B2ij  ui  vj ∂    −  = − log λi −   − c ∂Bij i=1 λi 2 j=1 i=1,j=1  u  qi  i  = −vj + vj qi  [Bv]  − Bij . ce i

(A.12)

(A.13) (A.14)

Acknowledgments We thank Bruno Olshausen and Eero Simoncelli for helpful discussions. This work was supported by a Department of Energy Computational Science Graduate Fellowship to Y.K. and National Science Foundation grant no. 0238351 to M.S.L. References Bell, A. J., & Sejnowski, T. J. (1995). An information maximization approach to blind separation and blind deconvolution. Neural Computation, 7(6), 1129– 1159. Bell, A. J., & Sejnowski, T. J. (1997). The “independent components” of natural scenes are edge filters. Vision Res., 37(23), 3327–3338. Bollerslev, T., Engle, R. F., & Nelson, D. B. (1994). ARCH models. In R. F. Engle & D. L. McFadden (Eds.), Handbook of Econometrics. Amsterdam: Elsevier. Buccigrossi, R. W., & Simoncelli, E. P. (1999). Image compression via joint statistical characterization in the wavelet domain. IEEE Transactions on Image Processing, 8(12), 1688–1701. Cardoso, J.-F. (1997). Infomax and maximum likelihood for blind source separation. IEEE Signal Processing Letters, 4, 109–111. Choi, S., Cichocki, A., & Belouchrani, A. (2002). Second order nonstationary source separation. Journal of VLSI Signal Processing, 32(1–2), 93–104. Cichocki, A., & Amari, S.-I. (2002). Adaptive blind signal and image processing: Learning algorithms and applications. New York, Wiley. Everson, R., & Roberts, S. (1999). Non-stationary independent component analysis. In Proceedings of the 8th International Conference on Artificial Neural Networks (pp. 503–508). Berlin: Springer-Verlag. Foldiak, P. (1991). Learning invariance from transformation sequences. Neural Computation, 3(2), 194–200. Hoyer, P. O., & Hyv¨arinen, A. (2002). A multi-layer sparse coding network learns contour coding from natural images. Vision Res., 42, 1593–1605. Hurri, J., & Hyv¨arinen, A. (2003). Simple-cell-like receptive fields maximize temporal coherence in natural video. Neural Computation, 15(3), 663–691.

422

Y. Karklin and M. Lewicki

Hyv¨arinen, A., & Hoyer, P. (2000). Emergence of phase- and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural Computation, 12, 1705–1720. Hyv¨arinen, A., Hoyer, P. O., & Inki, M. (2001). Topographic independent component analysis. Neural Comput., 13, 1527–1558. Hyv¨arinen, A., Hurri, J., & V¨ayrynen, J. (2003). Bubbles: A unifying framework for low-level statistical properties of natural image sequences. Journal of the Optical Society of America A, 20(7), 1237–1252. Hyv¨arinen, A., Karhunen, J., & Oja, E. (2001). Independent component analysis. New York: Wiley. Karklin, Y., & Lewicki, M. (2003). Learning higher-order structures in natural images. Network: Computation in Neural Systems, 14, 483–499. Kayser, C., Einh¨auser, W., Dummer, ¨ O., Konig, ¨ P., & Kording, ¨ K. (2001). Extracting slow subspaces from natural videos leads to complex cells. Artificial Neural Networks, 2130, 1075–1080. Kruger, ¨ N. (1998). Collinearity and parallelism are statistically significant second order relations of complex cell responses. Neural Processing Letters, 8, 117–129. LeCun, Y., Bottou, L., Orr, G., & Muller, ¨ K. (1998). Efficient backprop. In G. Orr & K. Muller ¨ (Eds.), Neural networks: tricks of the trade. New York: Springer. Lee, T.-W., Koehler, B., & Orglmeister, R. (1997). Blind source separation of nonlinear mixing models. In Proceedings of IEEE International Workshop on Neural Networks for Signal Processing. Lee, T.-W., & Lewicki, M. S. (2002). Unsupervised classification, segmentation and de-noising of images using ICA mixture models. IEEE Trans. Image Proc., 11(3), 270–279. Lee, T.-W., Lewicki, M. S., & Sejnowski, T. J. (2000). ICA mixture models for unsupervised classification of non-Gaussian sources and automatic context switching in blind signal separation. IEEE Trans. on Pattern Analysis and Machine Intelligence, 22(10), 1078–1089. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive-field properties by learning a sparse code for natural images. Nature, 381, 607–609. O’Neill, J. C. (1999). DiscreteTFDs time-frequency analysis software. Available online at: http://tfd.sourceforge.net/. Pearlmutter, B. A., & Parra, L. C. (1996). A context-sensitive generalization of ICA. In International Conference on Neural Information Processing (pp. 151–157). New York: Springer. Pham, D.-T., & Cardoso, J.-F. (2001). Blind separation of instantaneous mixtures of non stationary sources. IEEE Trans. Signal Processing, 49(9), 1837–1848. Romberg, J., Choi, H., & Baraniuk, R. (2001). Bayesian tree-structured image modeling using wavelet domain Hidden Markov models. IEEE Transactions on Image Processing, 10(7), 1056–1068. Schwartz, O., & Simoncelli, E. P. (2001). Natural signal statistics and sensory gain control. Nat. Neurosci., 4, 819–825. Simoncelli, E. P. (1997). Statistical models for images: Compression, restoration and synthesis. In Proc. 31st Asilomar Conference on Signals, Systems and Computers. Pacific Grove, CA.

Learning Density Components

423

van Hateren, J. H., & van der Schaaf, A. (1998). Independent component filters of natural images compared with simple cells in primary visual cortex. Proc. Royal Soc. Lond. B, 265, 359–366. Wainwright, M. J., Simoncelli, E. P., & Willsky, A. S. (2001). Random cascades on wavelet trees and their use in analyzing and modeling natural images. Applied Computational and Harmonic Analysis, 11, 89–123. Welling, M., Hinton, G. E., & Osindero, S. (2003). Learning sparse topographic representations with products of student-t distributions. In S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in neural information processing systems, 15, Cambridge, MA: MIT Press. Wiskott, L., & Sejnowski, T. J. (2002). Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14(4), 715–770. Received December 23, 2003; accepted July 2, 2004.

LETTER

Communicated by Aapo Hyvarinen

Extended Gaussianization Method for Blind Separation of Post-Nonlinear Mixtures Kun Zhang [email protected]

Lai-Wan Chan [email protected] Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong

The linear mixture model has been investigated in most articles tackling the problem of blind source separation. Recently, several articles have addressed a more complex model: blind source separation (BSS) of postnonlinear (PNL) mixtures. These mixtures are assumed to be generated by applying an unknown invertible nonlinear distortion to linear instantaneous mixtures of some independent sources. The gaussianization technique for BSS of PNL mixtures emerged based on the assumption that the distribution of the linear mixture of independent sources is gaussian. In this letter, we review the gaussianization method and then extend it to apply to PNL mixture in which the linear mixture is close to gaussian. Our proposed method approximates the linear mixture using the CornishFisher expansion. We choose the mutual information as the independence measurement to develop a learning algorithm to separate PNL mixtures. This method provides better applicability and accuracy. We then discuss the sufficient condition for the method to be valid. The characteristics of the nonlinearity do not affect the performance of this method. With only a few parameters to tune, our algorithm has a comparatively low computation. Finally, we present experiments to illustrate the efficiency of our method. 1 Introduction Blind source separation (BSS) aims at identifying and separating signals from their mixtures by linear or nonlinear transformation. It has received more and more attention due to its potential applications in signal processing areas, such as speech recognition, telecommunications, and medical signal processing. Most BSS algorithms are based on the theory of independent component analysis (ICA). As a generative model, ICA aims to find the independent components from the mixture of statistically independent sources by optimizing different criteria (for review, see Hyv¨arinen, 1999b; Hyv¨arinen & Oja, 2000; Cardoso, 1998; Mansour, Barros, & Ohnishi, 2000; Neural Computation 17, 425–452 (2005)

c 2005 Massachusetts Institute of Technology


426

K. Zhang and L.-W. Chan

Jutten & Taleb, 2000). In many cases, the terms independent component analysis and blind source separation mean essentially the same thing. ICA for linear instantaneous mixtures has been solved in many different ways (Jutten & H´erault, 1991; Amari & Cardoso, 1997; Oja, 1997; Hyv¨arinen, 1999a; Cardoso & Souloumiac, 1993; Pham & Garat, 1997; Comon, 1994; Pham, 2002; Bell & Sejnowski, 1995). Recently some extensions of classical ICA have been investigated. These extensions are more realistic for some situations in the real world. One important extension is nonlinear ICA. As shown in Hyv¨arinen and Pajunen (1999), for general nonlinear ICA, there always exist an infinite number of solutions if the space of the nonlinear mixing functions is unlimited. The independent components extracted from the observations are not necessarily the true source signals. Furthermore, in general, nonlinear ICA suffers from high computational complexity. Solving the nonlinear BSS problem appropriately with only the independence assumption of the sources is possible only in some special cases, for example, when the mixtures are post-nonlinear (PNL) with some weak assumptions (Taleb & Jutten, 1999b). Due to the practicability and relative uniqueness of the solutions, blind separation of PNL mixtures plays an important role in the nonlinear ICA area. The nonlinear model in Yang, Amari, and Cichocki (1998) is similar to the PNL mixing model. Lee, Koehler, and Orglmeister (1997) proposed a set of algorithms for BSS of PNL mixtures using parametric nonlinear functions. They focused on a parametric sigmoidal nonlinearity and on high-order polynomials. In a series of papers (Taleb & Jutten, 1997, 1999a, 1999b; Jutten & Taleb 2000), Taleb and Jutten approximated the inverse nonlinear function with multilayer perceptrons (MLP). Taleb and Jutten (1999b) exhibited their results on this topic and gave a general learning procedure based on the minimization of mutual information, given any parametric model for the inverse nonlinear functions. In Ziehe, Kawanabe, Harmeling, and Mueller (2001), alternating conditional expectation (ACE), a technique from nonparametric statistics, is used to approximately invert the nonlinear function. Due to the large residual nonlinear distortion that cannot be eliminated using this technique in some cases, the performance of this method is sensitive to the ICA algorithm in the linear stage. In fact, in this method, the linear mixture of independent sources is assumed to be gaussian (Breiman & Friedman, 1985). Under this assumption, the inverse nonlinear function can be constructed directly using the gaussianization technique (Ziehe, Kawanabe, Hermeling, & Mueller, 2003). Almost all existing methods for BSS of PNL mixtures impose a parametric model, such as an MLP or a given functional space directly on the inverse of the nonlinear function generating observations. Although multilayer feedforward networks are universal approximators (Hornik, Stinchcombe, & White, 1989) they may behave poorly for some peculiar nonlinearities. These methods are sensitive to actual nonlinear functions used. In principle, when we use a parametric model to fit an arbitrary nonlinear function on which

Extended Gaussianization for Separation of PNL Mixtures

427

we have very little prior knowledge, either many parameters are needed or the constraint imposed on the actual function must be restrictive. In this letter, we consider the BSS problem of PNL mixtures when the distributions of latent linear mixtures are close to gaussian. We do not construct the inverse nonlinearity directly, but recover the nonlinearity from the distributions of linear mixtures. In the gaussianization method for BSS of PNL mixtures (Sol-Casals, Babaie-Zadeh, Jutten, & Pham, 2003; Ziehe et al., 2003), the distributions of the linear mixtures are simply approximated by the gaussian distribution. We first briefly review this method by discussing its foundation, its limitation, and some practical considerations in its implementation. Next, we propose an extended version of gaussianization to tackle the case when the distributions of the linear mixtures are not exactly gaussian but close to gaussian. Assuming the distributions of the linear mixtures are close to gaussian, this method models the linear mixtures by the Cornish-Fisher expansion and gives their distributions freedom to adapt in the vicinity of the standard gaussian distribution. In noiseless cases, its performance is not affected by the actual nonlinearities. It depends only on how close the estimated distributions of latent linear mixtures are to their actual distributions. With few parameters to tune, this method is comparatively practical—it should be less prone to getting stuck at local optima, and it avoids the demanding computation other nonlinear ICA algorithms require in applications. The rest of this letter is organized as follows. In section 2, we give a brief introduction to the PNL mixing model and discuss its identifiability. The gaussianization method is reviewed in section 3. In section 4, we develop an extended version of the gaussianization. In section 5, experimental results are shown to illustrate the performance of our method. Section 6 concludes the article. 2 Post-Nonlinear Mixing Model Blind separation of PNL mixtures, which was introduced by Taleb and Jutten (1997), is a special case of nonlinear ICA. In this model, the sources si are first mixed linearly according to the basic ICA model, and after that, a nonlinear function fi is applied to the linear mixture ti to generate the observation xi ,  xi = fi (ti ) = fi 

n 

 aij sj  ,

(2.1)

j=1

where si are statistically independent sources, fi are unknown invertible nonlinear functions, aij denote entries of a regular m × n mixing matrix A, and ti are latent linear mixtures. The PNL assumption can be used to model a nonlinear sensor distortion in some signal processing applications.

428

K. Zhang and L.-W. Chan

PNL mixing system s1

t1

s2

t2

. . . sn

A linear mixing

. . . tn

f1 f2 . . . fn nonlinear distortion

PNL de-mixing system x1 x2 . . . xn

g1 g2 . . . gn

z1

y1

z2

y2

. . . zn

inverse nonlinear transform

W

. . . yn

linear de-mixing

Figure 1: PNL mixing system and PNL demixing system. The mixing system consists of a linear mixing stage (with the mixing matrix A) and a nonlinear transform stage (applying fi to linear mixtures ti ). The demixing system does an inverse operation, which is a nonlinear transform stage followed by a linear demixing stage.

For simplicity, we have assumed there is no additive noise in this system. And in the following discussion, the number of independent sources, n, is assumed to be equal to that of observations, m. As a counterpart of the mixing model 2.1, the separation of PNL mixtures is also a two-stage procedure: a nonlinear stage followed by a linear demixing stage. Figure 1 shows the PNL mixing system and the demixing system, where gi are elements of g used to invert the nonlinear mapping f, z has n elements zi as an estimate of the latent linear mixtures ti , and W is a linear demixing matrix transforming zi to yi , the estimate of independent sources si . Taleb and Jutten (1999b), showed that when A has at least two nonzero entries per row or per column and si accepts a density function that vanishes at one point at least, one can affirm that the output y has mutually independent components if and only if gi ◦ fi is linear and W is a linear separating matrix for z. In other words, under these assumptions, the sources can be separated up to the same indeterminacies as in the linear ICA model, which are permutation and scaling indeterminacies. (In some work, for instance, Hyv¨arinen, Karhunen, & Oja, 2001, it is said there is one more indeterminacy, named sign indeterminacy, which can be considered as a special case of the scaling indeterminacy.) In fact, if the mean of si is unknown, one more indeterminacy emerges in the BSS problem of PNL mixtures, which we call mean indeterminacy.

Extended Gaussianization for Separation of PNL Mixtures

429

If the sources si are not zero mean, let si = snew + E(si ) and ti = tnew + E(ti ). i i We have   n  new xi = fi (ti ) = fi  aij (sj + E(sj )) = fi (tnew + E(ti )) = finew (tnew i i ), (2.2) j=1

where finew is a new function obtained by shifting the nonlinear function fi (to the left if E(ti ) > 0, and to the right if E(ti ) < 0) by |E(ti )| units. We can change the mean of ti and at the same time shift the nonlinear function fi with corresponding units, while the observations xi do not change. In other words, in the problem of blind separation of PNL mixtures, the mean of the latent random vector s, as well as the actual functions fi , cannot be recovered without any additional prior knowledge. In some cases, this indeterminacy may be trivial. Moreover, in practice, if the actual nonlinear distortion functions fi or the actual values of si are required for further investigation, we can sometimes find some hints, such as the intersection of the nonlinear function fi with the x-axis, or prior knowledge in distributions of si , to eliminate this indeterminacy. Even if si are not zero mean while yi are zero mean, each gi ◦ fi is also affine. This is because the nonlinear block g in the demixing system is used to invert the unknown nonlinear function f new , new = k · ti − k · E(ti ), zi = (gi ◦ fi )(ti ) = (gi ◦ finew )(tnew i ) = k · ti

(2.3)

where k is a constant. In the following, in order to eliminate the scaling indeterminacy, we assume each function gi is strictly monotonically increasing, and zi and yi are of unit variance. In order to eliminate the mean indeterminacy, zi are assumed to be zero mean. 3 Gaussianization for BSS of PNL Mixtures Gaussianization (Chen & Gopinath, 2001) transforms a random variable into a standard gaussian random variable. In BSS of PNL mixtures, by approximating the distributions of linear mixtures as the gaussian distribution, gaussianization can be used to recover the linear mixtures directly from the observations. 3.1 Review of the Central Limit Theorem. The central limit theorem (CLT), as one of the fundamentals in probability theory, provides an imprecise but intuitive way of measuring independence between nongaussian variables using their individual nonnormality. This has been investigated in the ICA literature. The CLT also inspires the application of gaussianization in BSS of PNL mixtures.

430

K. Zhang and L.-W. Chan

The CLT provides sufficient conditions for asymptotic normality of normalized sums of random variables (Neuts, 1995). Assume Xi , i = 1, 2, ... are independent variables with zero mean and variance σi2 , respectively, and let Sn = X1 + X2 + ... + Xn . The classical CLT says that if Xi are independent and identically distributed, then Sn /(σ1 n1/2 ) is asymptotically normally distributed as n → ∞ (note that σ1 = σ2 = ... = σn ). In another form of CLT, it is claimed that Sn /sn is asymptotically normally distributed under n  E(|Xi |3 ) → 0 as Lyapunov’s condition that E(|Xi |3 ) are finite and that s13 n

i=1 n→ = σ12 +σ22 +...+σn2 . Lindeberg’s condition is much weaker; n   1 it states that Sn /sn is asymptotically normal if s2 x2 dFi (x) → 0 for n j=1 |x|≥εsn

∞, where s2n

any ε > 0, where Fi (x) is the cumulative distribution function (cdf) of Xi (Neuts, 1995). In the latter two conditions, the constraint that Xi are identically distributed is not necessary. We can roughly say that in general, sums of independent variables tend to be close to gaussian. Of course, there are many exceptions, for instance, when the independent variables are very nongaussian and the number of independent variables is very small, or when one strong nongaussian source dominates the sum. Here, we focus on the case that linear mixtures of independent sources are comparatively close to gaussian.

3.2 BSS of PNL Mixtures with Gaussianization. Now we approximate the distributions of the recovered linear mixtures zi by the standard gaussian distribution. Then the nonlinearity gi is used to transform the variable xi into the standard gaussian variable zi . Assume the cdf of xi is Fxi (xi ), which can be estimated from observed samples. Let G be the cdf of the standard  gaussian distribution: G(z) =

√1 2π

z

−∞

e−t

2

/2

dt. BSS of PNL mixtures with

gaussianization is illustrated as a two-stage procedure (Sol-Casals et al., 2003; Ziehe et al., 2003): 1. Construct each inverse nonlinear function by gaussianization: gi = G−1 ◦ Fxi . The latent linear mixtures are estimated by zi = gi (xi ). 2. Use a linear ICA algorithm to extract independent components yi from z: y = Wz. y is the estimate of s. We address the following practical issues: • The estimation of the cdf Fxi can be done in a variety of ways. For instance, it can be constructed using the pdf estimated by kernel estimation. For simplicity, we use the empirical cumulative distribution function (ecdf) Fˆxi (x) instead, which is the proportion of samples less

Extended Gaussianization for Separation of PNL Mixtures

ˆ than or equal to x, that is, F(x) =

1 N+1

N 

431

I(x(i) ≤ x), where x(i) is the ith

i=1

sample of variable x, N is the total number of samples, and I(x(i) ≤ x) is an indicator function with value 1 if x(i) ≤ x and with value 0 otherwise. • The inverse of the cdf of the standard gaussian distribution G−1 can be calculated numerically. Or we can use an analytical form to approximate it (Abramowitz & Stegun, 1968). For 0.5 ≤ p < 1, G−1 (p) can be approximated by c0 + c1 u + c2 u z = G−1 (p) ∼ (0.5 ≤ p < 1), where =u− 1 + d1 u + d2 u2 + d3 u3  u = ln[1/(1 − p)2 ], and c0 = 2.515517, c1 = 0.802853, c2 = 0.010328, d1 = 1.432788, d2 = 0.189269, d3 = 0.001308. The absolute error in this approximation is less than 4.5 × 10−4 . For 0 < p ≤ 0.5, z = G−1 (p) can first be rewritten as z = −G−1 (1 − p) and then evaluated using the same approximation. • If necessary, parametric estimates of gi can be computed using a least(k) N −1 ˆ square method on points {x(k) i , G (Fxi (xi ))}k=1 , with a LevenbergMarquardt search algorithm (Gill, Murray, & Wright, 1981). • In principle, any linear ICA algorithm is valid in the linear demixing stage, such as JADE (Cardoso & Souloumiac, 1993), FastICA (Hyv¨arinen, 1999a), Extended INFOMAX algorithm (Lee, Girolemi, & Sejnowski, 1999), and others. However, in some cases, the latent linear mixtures are far from gaussian, such that the residual distortion in the recovered linear mixtures is considerable, and hence these methods may provide different results. We found that methods that do not ensure perfect decorrelation between independent components, such as the INFOMAX algorithm (Bell & Sejnowski, 1995; Lee et al., 1999), and the natural gradient algorithm (Amari, Cichocki, & Yang, 1996), exhibit better separability. And if there is a time correlation in the independent sources, methods using time structure, such as SOBI (Belouchrani, Abed-Meraim, Cardoso, & Moulines, 1997), are more robust to the residual distortion. In general, the separation result with the gaussianization method is not accurate because this method involves a rough approximation—it simply approximates the distribution of linear mixtures by the gaussian distribution. However, its computational cost is very low. Sol-Casals et al. (2003) describe this method as a technique providing a pertinent initialization of the demixing system and improving the learning speed. We aim to find

432

K. Zhang and L.-W. Chan

a reasonable parametric model for the linear mixtures to provide a better estimate for them, and the gaussianization method will be used as an initialization step. 4 Extended Gaussianization When the linear mixtures ti are not so close to gaussian, we denote by g(1) the nonlinear mapping constructed by gaussianization and introduce another nonlinear mapping g(2) to the demixing system, as shown in Figure 2B. In this system, v = (v1 , v2 , ..., vn )T and vi follow the standard gaussian distribution. In other words, in the extended gaussianization method, the nonlinear stage in the demixing system consists of two parts. First, we use an invertible nonlinear mapping g(1) to obtain v from the observed vector x by gaussianization: −1 vi = g(1) i (xi ) = G (Fxi (xi )).

(4.1)

After v has been obtained, we consider v as a new observation. The latent linear mixture z is then recovered from v by another invertible nonlinear mapping g(2) , (1) gi = g(2) i ◦ gi .

(4.2)

(2) Similar to gi , g(1) i and gi are strictly monotonically increasing. When the (2) nonlinear mapping g is estimated, we can construct the overall inverse nonlinear functions by equation 4.2.

4.1 Parametric Model for Function g(2) i Using Cornish-Fisher Expansion. The functions g(2) transform the standard gaussian variables vi into i the variables zi . Since the distributions of zi are assumed to be close to gaussian, the Cornish-Fisher (C-F) expansion helps to construct a simple and reasonable parametric model for g(2) i . 4.1.1 Cornish-Fisher Expansion. We aim to find how to express quantiles for a distribution close to gaussian using those of the gaussian distribution. In fact, it is possible to obtain explicit polynomial expansions for standardized quantiles of a general distribution in terms of its normalized cumulants and the corresponding quantiles of the standard gaussian distribution (Johnson & Kotz, 1970). The basis behind this kind of expansion is that if p(x) is a pdf with cumulants κ1 , κ2 , ..., then the function i   ∞ (−D) j ∞ ∞ j   j=1 j j! (−D)  p(x) = j p(x) g(x) = exp  j! i! i=1 j=1

Extended Gaussianization for Separation of PNL Mixtures

433

A

x

g

z

W

inverse nonlinear transform

y

linear de-mixing

B

x

g(1)

v

g(2)

inverse nonlinear transform

z

W

y

linear de-mixing

Figure 2: (A) The separating system with gaussianization. (B) The separating system used in the extended gaussianization method. In A, components of z and the nonlinear transform g are constructed by gaussianization. In B, we do not construct the nonlinear transform g directly, but construct g(2) ◦ g(1) instead. g(1) and v are constructed by gaussianization. g(2) and W are tuned to minimize the mutual information between yi .

will have cumulants κ1 + 1 , κ2 + 2 , ..., where D is the differentiation operator and D j p(x) = d j p(x)/dx j . Using this principle, we can obtain useful approximate representation of a distribution with known cumulants in terms of a known distribution. The Edgeworth expansion and the Gram-Charlier expansion arise from choosing the standard gaussian distribution as this known distribution. Furthermore, quantiles for a distribution are totally determined by the pdf of this distribution, and hence one can approximate the quantiles of a distribution close to gaussian using those of the standard gaussian distribution and known cumulants. Cornish and Fisher (1937) propounded a form of expansion in which the terms are polynomial functions of the appropriate standard gaussian quantile, with functions of known cumulants as coefficients. The four-term C-F expansion for α-quantile is 1 1 1 x(vα ) ∼ = vα + (v2α − 1)κ3 + (v3α − 3vα )κ4 − (2v3α − 5vα )κ32 , (4.3) 6 24 36 where κ3 and κ4 are the third-order and fourth-order cumulants (or skewness and kurtosis) of x, respectively, vα is the α-quantile for the standard gaussian distribution, and x(vα ) is zero mean and of unit variance. Figure 3 shows the pdf’s of x(vα ) obtained by equation 4.3 and the mapping functions from quantiles for the standard gaussian distribution to those for distributions determined by equation 4.3 with different values of κ3 and

434

K. Zhang and L.-W. Chan

κ4 . Jaschke (2002) showed that the C-F approximation is a competitive technique if the target distribution is relatively close to normal. According to the CLT, the higher-order statistics of the sum of independent variables tend to vanish, so the sum can be modeled well with only the third- and fourth-order statistics. And the most prominent use and motivation of this expansion is to model the distribution of the normalized sum of independent variables. 4.1.2 Parametric Model for g(2) i . Assume the distributions of the linear mixtures ti are close to gaussian and ti can be modeled by the four-term C-F expansion. According to equation 4.3, the recovered linear mixtures zi can be expressed in terms of the gaussian signals vi : zi = g(2) i (vi |θi ) 1 1 1 2 = vi + (v2i − 1)κ3,i + (v3i − 3vi )κ4,i − (2v3i − 5vi )κ3,i . 6 24 36

(4.4)

Parameters in the nonlinear stage are θi = {κ3,i , κ4,i }, i = 1, 2, ...n. 4.2 Learning Rule for Extended Gaussianization. Now we derive the learning rules for the unknown parameters in Figure 2B based on the minimization of the mutual information between yi . 4.2.1 Mutual Information: An Independence Criterion. Mutual information is a measure of the information that members of a set of random variables have on the other random variables in the set.1 In information theory, the mutual information I between n random variables yi , i = 1, 2, ..., n is defined as follows: I(y1 , y2 , ..., yn ) =

n 

H(yi ) − H(y),

(4.5)

i=1

where y = (y1 , y2 , ..., yn )T , and H(•) denotes the differential entropy.2 Alternatively, mutual information can be interpreted as the Kullback-Leibler (KL) divergence between the joint pdf py (y) and the product of marginal  densities ni=1 pyi (yi ):  n  py (u) δ(py (y), du. pyi (yi )) = py (u) log n i=1 pyi (ui ) i=1 Mutual information is always nonnegative, and it is zero if and only if the variables are independent (Cover & Thomas, 1991). 1 Mutual information was originally defined for two variables. It has more than one multivariate generalizations. For details, see Bell (2003). 2 Shannon entropy is defined for a discrete variable. Differential entropy is the extension of Shannon entropy for continuous random variables.

Extended Gaussianization for Separation of PNL Mixtures

435

A 0.7

standard normal skew:0, kurt:2 skew:0.6, kurt:0 skew:−1, kurt:2 skew:−0.3, kurt:−0.7

0.6

0.5

p(x)

0.4

0.3

0.2

0.1

0 −4

−3

−2

−1

0

1

2

3

4

x (obtained by the C−F expansion)

B 4

3

2

standard normal skew:0, kurt:2 skew:0.6, kurt:0 skew:−1, kurt:2 skew:−0.3, kurt:−0.7

α

x(v )

1

0

−1

−2

−3

−4 −3

−2

−1

0

1

2

3

vα Figure 3: Quantiles obtained by the C-F expansion with different cumulants κ3 and κ4 . (A) Distributions of quantiles obtained. (B) Quantiles obtained versus quantiles for the standard gaussian distribution—the mapping x(·) defined in equation 4.3.

436

K. Zhang and L.-W. Chan

Mutual information is a reliable measurement for independence between random variables. It has been widely used for constructing contrast functions in the ICA problem (Comon, 1994; Pham, 2000; Taleb & Jutten, 1999b). 4.2.2 Blind Separation of PNL Mixtures by Minimization of Mutual Information: A General Procedure. Let θi be the parameter set in the nonlinear function gi in Figure 1. According to Figure 1, we have zi = gi (xi |θi ),

(4.6)

y = Wz.

(4.7)

Since the demixing system for PNL mixtures is of cascade architecture and the linear demixing stage is to minimize the mutual information between yi given variables zi as inputs, the learning rule for W will not involve parameters in θi explicitly. It is just the same as in the linear ICA model except that the inputs are not the original observations but the estimated linear mixtures. Therefore, any learning rule for the linear ICA model based on minimization of mutual information can be used, for instance, the BellSejnowski algorithm (Bell & Sejnowski, 1995; Cardoso, 1997), W ∝ [W T ]−1 + E[ψ(y)zT ],

(4.8)

where ψ is a component-wise vector function that consists of the so-called score functions ψyi of the distributions of yi = wTi z (wTi is the ith row of W), defined as ψyi (u) = (log pyi (u)) =

pyi (u) pyi (u)

.

(4.9)

Multiply the right-hand side of equation 4.8 with W T W; we obtain W ∝ (I + E[ψ(y)yT ])W,

(4.10)

which is known as the natural gradient proposed by Amari (1998) and Cardoso and Laheld (1996). In order to make yi of unit variance when the algorithm converges, we modify equation 4.10 as follows: W ∝ (I−diag{E(y21 ), ..., E(y2n )}+E[ψ(y)yT ]−diag{E[ψ(y)yT ]})W,(4.11) where diag{E(y21 ), ..., E(y2n )} denotes the diagonal matrix with E(y21 ), ..., E(y2n ) as its diagonal entries, and diag{E{ψ(y)yT }} denotes the diagonal matrix whose diagonal entries are those on the diagonal of E{ψ(y)yT }. Taleb and Jutten (1999b) derived the mutual information-based learning rule for the parameters θi in the nonlinear stage:      n  ∂ log |gi (xi |θi )| ∂gi (xi |θi ) ψyk (yk )wki . (4.12) +E θi ∝ E ∂θi ∂θi k=1

Extended Gaussianization for Separation of PNL Mixtures

437

4.2.3 Learning Rule for Extended Gaussianization. The learning rule for extended gaussianization is obtained by applying the general learning rule for separating PNL mixtures (Taleb & Jutten, 1999b) to the specific demixing (2) procedure (g(2) i , W). Let Di be the derivative of gi (vi |θi ) with respect to vi : 1 1 2 1 2 Di = g(2) κ (6v2 − 5). (4.13) i (vi |θi ) = 1 + κ3,i vi + (vi − 1)κ4,i − 3 8 36 3,i i Let φ = (φ1 , φ2 , ..., φn )T = W T ψ. equation 4.12 becomes  1  2 κ3,i ∝ E vi − 6 κ3,i (6vi −5) + φi v2 − 1 − 3Di 6 i κ4,i ∝ E v2i −1 + φi (v3 − 3vi ) . 8Di 24 i

κ3,i (2v3i −5vi ) 3



,

(4.14)

These two rules, together with equation 4.11, are the update rules for all the unknown parameters in the demixing system of PNL mixtures. Of course, suitable learning rates are also essential for the convergence speed and stability of the learning procedure. Estimation of the score function ψ will be treated later. 4.2.4 Performance Analysis. The gaussianization technique is very simple, and its computational cost is very low. It is considered a preprocessing step in the extended gaussianization method. After this step, the inverse of remaining nonlinearity in each channel is modeled using only two unknown parameters (κ3,i and κ4,i ). When the linear mixtures are not far from gaussian, our method is more stable and computationally lighter compared to ordinary methods, which simply exploit some generic structures, such as MLPs and polynomials, to model the nonlinearity (Taleb & Jutten, 1999b; Lee et al., 1997). In the extended gaussianization method, the component-wise function, defined as g(1) ◦f, has elements that are strictly increasing functions mapping ti to standard gaussian variables vi . Given the same latent mixture t, different nonlinear functions f do not affect the realization value of v. Therefore, the performance of the extended gaussianization method is not affected by the choice of nonlinearities fi .3 The extended gaussianization method can be considered as a specific application of the general learning rule derived in Taleb and Jutten (1999b) with gaussianization as a preprocessing step and the C-F expansion for representing the inverse of remaining nonlinearities. The performance of this method depends on how well the C-F expansion can fit the actual distributions of linear mixtures. Hence, in order to analyze the performance of our method, we first investigate the property of the C-F expansion. 3 This is true only in the noiseless situation, and only in the training set. We thank one of the reviewers for pointing this out.

438

K. Zhang and L.-W. Chan

To ensure that the C-F expansion corresponds to the quantile of a distribution and that the mapping x(vα ) (see equation 4.3) is one-to-one, equation 4.3 should be a strictly monotonic function of vα . However, it is not always satisfied for all possible values of (κ3 , κ4 ). We can identify the region in the (κ3 , κ4 )-plane guaranteeing the strict monotonicity of the mapping x(vα ). For values of (κ3 , κ4 ) in this region, the derivative 1 1 1 dx(vα ) = 1 + κ3 vα + (v2α − 1)κ4 − κ32 (6v2α − 5) dvα 3 8 36

(4.15)

should be positive definite. Since equation 4.15 is a simple function of vα with the highest order no more than 2, we can easily find the domain for (κ3 , κ4 ) α) in which dx(v dvα is always positive for any value of vα , as shown in Figure 4A. Values in this region always result in super-gaussian distributions of x(vα ). In practice, we can concentrate on a bounded range of vα because the value of vα is bounded. In this way, the monotonicity constraint is relaxed. For instance, we can implement the monotonicity constraint for vα between the 0.001 quantile (−3.09) and the 0.999 quantile (3.09) for the standard gaussian distribution. After some straightforward but tedious derivations, the domain for κ3 and κ4 is obtained, as sketched in Figure 4B. The region in this figure shows us the feasibility of approximating zi with the C-F expansion when zi is close to gaussian, especially when it is super-gaussian. When (κ3 .κ4 ) lies in the lower part of Figure 4B, the C-F expansion may become less and less reliable for α → 0 (or α → 1). Due to the limitation of approximation of a distribution by the C-F expansion, the performance of the extended gaussianization method varies for different characteristics of the distribution of the linear mixture: • When the linear mixtures ti are close to gaussian and their statistics (κ3,i , κ4,i ) lie in the domain shown in Figure 4A, the C-F expansion models the linear mixtures ti well, and hence good performance is achieved. • When ti are close to gaussian, {κ3,i , κ4,i } lie in the domain indicated by the lower part of Figure 4B, and the monotonicity of the C-F expansion is guaranteed for all samples of vi , the linear mixtures ti can still be modeled well by the C-F expansion, which also results in a good performance. • When ti are close to gaussian, (κ3,i , κ4,i ) lie in the domain indicated by the lower part of Figure 4B, and some samples of vi are so large as to violate the monotonicity of the C-F expansion, we can drop these samples, and the extended gaussianization method still works. This is because mutual information as an independence measure is robust to outliers. But inevitably the performance declines compared to the first two cases.

Extended Gaussianization for Separation of PNL Mixtures

439

A

B

Figure 4: The domain of (κ3 , κ4 ) guaranteeing monotonicity of the four-term C-F expansion, equation 4.3. (A) Theoretical solution. (B) The domain guaranteeing monotonicity for vα ∈ [−3.093.09]. It consists of two parts. The upper part is the same as the domain in A.

440

K. Zhang and L.-W. Chan

• When the distributions of linear mixtures are far from the gaussian distribution, for instance, when they are multimodal, this method may fail. 4.3 Estimation of the Score Function. In the ICA algorithms for linear mixtures, estimation of the score functions (see equation 4.9) does not have to be very accurate. In practice, only two nonlinearities (one for supergaussian signals and the other for subgaussian signals) used to approximate the score functions are sufficient (Lee et al., 1999). However, separation performance in nonlinear mixtures depends greatly on the estimation performance of score functions (Taleb & Jutten, 1999b). Currently, there are three main ways to estimate the score function. The first is to estimate the pdf pyi directly and find its derivative. Finally, the score function can be obtained. Many ways such as the kernel estimation can be used to estimate the pdf. Although the kernel estimation method may provide a noisy estimation, its simplicity makes it popular when the number of samples is small. However, since this method has a heavy memory demand, it is not suitable when the number of samples is large. The second way is to use the Gram-Charlier expansion to approximate the pdf pyi and to find the score function based on this expansion. The performance of this method is good when the nonlinearities are not strong, but it is poor in the case of strong nonlinearities due to the limitation of accuracy of the Gram-Charlier expansion (Taleb & Jutten, 1999b). The third way is to use some nonlinear models, such as nonlinear regressors and MLPs, to estimate the score function directly. But for nonlinear regressors, it is hard to choose suitable projection basis functions, and for MLPs, many parameters need to be learned. Here we propose to use a mixture of gaussians to model each output density. The mixture density model is widely used to model an arbitrary density. In Xu, Cheung, Yang, and Amari (1997), the cdf of a mixture of densities is used to model the nonlinearity in Bell and Sejnowski’s informationtheoretic ICA framework. In the independent factor analysis model (Attias, 1999), each source density is described by a mixture of gaussians. In our experiments, the density of output yi is modeled as the mixture of ni gaussians, and the qi th gaussian has the means mi,qi , variances vi,qi , and mixing weights πi,qi : pyi (u) =

ni 

πi,qi G (u|mi,qi , vi,qi ).

(4.16)

qi =1

Then the score function can be obtained analytically, ψyi (u) =

pyi (u) pyi (u)

=

ni  qi =1

p(qi |u)

u − mi,qi , vi,qi

(4.17)

Extended Gaussianization for Separation of PNL Mixtures

441

where p(qi |u) denotes the posterior probability that u, as a realization of yi , is generated by the qi th gaussian. During the learning process, all parameters in the mixture model are tuned to trace current outputs based on maximum likelihood. The expectation-maximization (EM) algorithm is used to maximize the likelihood. After each update of the demixing system, the parameters in the mixture model are updated according to the EM algorithm (N denotes the number of samples, and the subscript t in yi,t denotes the tth sample of yi ): (k) (k) πi,q G (yi,t |m(k) i,qi , vi,qi ) i p(k) (qi |yi,t ) = n , (k) (k) (k) i qi =1 πi,qi G (yi,t |mi,qi , vi,qi ) N 1  p(k) (qi |yi,t ), N t=1

N (k) t=1 p (qi |yi,t )yi,t = , N (k) t=1 p (qi |yi,t )

(4.18)

(k+1) πi,q = i

(4.19)

m(k+1) i,qi

(4.20)

= N v(k+1) i,qi

1

(k) t=1 p (qi |yi,t )

N 

2 p(k) (qi |yi,t )(yi,t − m(k+1) i,qi ) .

(4.21)

t=1

When the nonlinear stage (with parameters θi , i = 1, ..., n), the linear stage (with parameter W), and the EM stage for modeling the output density (with parameters πi,qi , mi,qi , vi,qi , i = 1, ..., n and qi = 1, ..., ni ) converge, the learning process terminates. We observe that using this method, all parameters exhibit a small oscillation in the learning process. The reason is that parameters in the nonlinear stage are very sensitive to the estimated output densities, and parameters in the linear stage and the EM stage depend crucially on those in the nonlinear stage. We have two ways to eliminate the oscillation. The first is to let the EM algorithm iterate several times to get a stable distribution, instead of just once, after each update of the demixing system. The second way is to reduce the learning rate in the nonlinear stage as training goes on. 5 Experiments In this section, we present some experiments to show the validity of our methods. When the number of independent sources increases, generally their linear mixtures are closer to gaussian, so that the accuracy of our methods is improved. In our experiments, we use only two sources for illustration purpose. In order to compare the performance of different methods quantitatively, the classical residual cross talk C(y, s) = 10 log10 E[(y − s)2 ] is used as the performance index, in which y is an estimate of s, and by multiplying by a constant, y and s are made to be of unit variance and with positive correlation.

442

K. Zhang and L.-W. Chan

In the first experiment, two independent sources are an amplitudemodulated signal, that is, s1 (t) = cos(0.0157t) sin(0.2t), and a gamma(3,2) distributed noise. They have been made zero mean and of unit variance. The linear mixing matrix is chosen randomly:  A=

0.0691 0.8821

 0.1257 . 0.9395

In the first experiment, the nonlinearities chosen are said to be “hard” in Taleb and Jutten’s work: 

f1 (t1 ) = (5t1 )3 , f2 (t2 ) = t2 + 2 tanh(t2 ).

Figure 5 shows the independent sources si , their linear mixtures ti , and the distributions of PNL observations xi . We can see that ti are skewed and supergaussian. The learning rule for the nonlinear stage is described in equation 4.14. Parameters in the nonlinear stage are {κ3,i , κ4,i }, i = 1, 2. We initialized all of these parameters as 0. The demixing matrix W in the linear demixing stage was initialized as the identity matrix. The learning rate in the nonlinear stage is 0.5, and that in the linear demixing stage is 0.2. For simplicity, the ecdf Fˆxi is used as the estimate of the cdf Fxi . In estimation of the score function, the gaussian mixture model consists of five gaussians. After about 800 iterations, the learning process converges. At convergence, residual distortion in each channel, independent components yi , and their joint distributions are given in Figure 6. The performance index is C(s1 , y1 ) = −27.8 dB, C(s2 , y2 ) = −27.7 dB. Figure 6A compares the residual distortion obtained by the extended gaussianization method with that by gaussianization. The independent components obtained by gaussianization are given in Figure 7. They are shown for comparison with the separation result of the extended gaussianization method. The performance index of the gaussianization method is C(s1 , y2 ) = −11.1 dB, C(s2 , y1 ) = −10.5 dB. We can see that the separation result of the extended gaussianization method is much superior to those obtained by gaussianization. The pdf’s generated by the C-F expansion with the parameters {κ3,i , κ4,i } at convergence are shown in Figure 8 by the solid line, together with the distribution histograms of the linear mixtures ti (which have been normalized to unit variance). These pdf’s fit the actual distributions of ti well. However, due to the limited freedom of the C-F expansion, and more important, due to the particularities of the actual distributions of mixtures ti (e.g., both independent sources are bounded or left bounded), the distributions of the linear mixtures estimated by the C-F expansion might not approach their actual distributions so well. This can also be seen by comparing the cumu-

Extended Gaussianization for Separation of PNL Mixtures

443

A 2

1

1

s

0

−1 −2 0

200

400

600

800

1000

200

400

600

800

1000

6

2

4

s

2 0

B

−2 0

1

t1

0.5 0 −0.5 0

200

400

600

800

1000

200

400

600

800

1000

10

t2

5 0 −5 0

6

6 4

5

2

2

t2

2

4

s

10

x2

C

0

0

0 −2 −2

−5

−2

−1

0

s

1

1

2

−4 −0.5

0

t

1

0.5

1

−10

0

20

x1 40

60

Figure 5: (A) Independent sources used in the first experiment. The column on the right shows their normalized histograms. For comparison, the gaussian distribution with the same mean and variance is given by the dashed curve. (B) Linear mixtures and their histograms. (C) Scatter plots of the sources, linear mixtures, and PNL observations in the first experiment (left to right).

444

K. Zhang and L.-W. Chan

A 5 Channel 1

4 3 2

Channel 2

1 0 −1 −2 −3

Extended Gaussianization Gaussianization

−4 −4

B

−2

0

2

4

6

y1

2

0

−2 400

450

500

550

450

500

550

6

y2

4 2 0 −2 400

C 4

4

2

2 0

y

z

2

2

0

−2 −4

−2

0

z1 2

4

−2

−2

y1

0

2

Figure 6: Separation result by the extended gaussianization. (A) Residual distortion gi ◦ fi in each channel. (For comparison, that obtained by gaussianization is shown by the dash-dot line.) (B) Recovered independent components (the original sources are shown by the dash-dot line). (C) Scatter plots of the recovered linear mixtures and independent components. Performance index: C(s1 , y1 ) = −27.8 dB, C(s2 , y2 ) = −27.7 dB.

Extended Gaussianization for Separation of PNL Mixtures

445

Figure 7: Separation result (independent components) obtained by gaussianization (shown for comparison). The original sources are shown by the dash-dot line. Performance index: C(s1 , y2 ) = −11.1 dB, C(s2 , y1 ) = −10.5 dB.

Figure 8: Distributions constructed by the C-F expansion with parameter values at convergence (solid line) and distribution histograms of the linear mixtures ti (normalized to unit variance). Gaussian distributions are given by the dash-dot line for comparison.

lants of the true mixtures and those of the estimated ones. Skewness and kurtosis of t1 are 0.92 and 1.37, and those of t2 are 0.50 and 0.62, respectively. While z1 has the skewness 0.79 and the kurtosis 0.98, z2 has 0.41 and 0.20. (The parameters at convergence are κ3,1 = 0.77, κ4,1 = 1.20, κ3,2 = 0.43, and κ4,2 = 0.35.) Figure 9 shows the output densities estimated by the gaussian mixture model as well as their normalized distribution histograms. We can see the gaussian mixture model has been adapted to the output densities.

446

K. Zhang and L.-W. Chan

Figure 9: Output distribution histograms and their densities learned by the gaussian mixture model (five gaussians are used).

We also compare our result with that obtained by the method described in Taleb and Jutten (1999b). The latter uses an MLP to model the nonlinearities gi . A two-layer MLP with five hidden neurons is used. Sigmoid activation function is used in the hidden layer, and other neurons are linear. Even with gaussianization as an initialization step, it takes at least 2500 iterations to achieve the same performance index (-27.8 dB for the first source and -27.7 dB for the second source), which is similar to the result Taleb and Jutten reported. Compared to the extended gaussianization method, the MLP method is much more time-consuming. Moreover, the learning is more prone to local optima. In order to demonstrate that our method also performs well in other cases, we repeat the above experiment for 20 runs, and in each run the (1, 1)th entry of the mixing matrix A is randomly chosen between 0 and 0.15. The independent sources are always successfully recovered by the extended gaussianization method. In the worst of the 20 runs, the performance index is −23.6 dB (for the first source) and −23.3 dB (for the second source). And the extended gaussianization method always outperforms the gaussianization method. The average performance index of the extended gaussianization method is −26.1 dB (with standard deviation 1.3) and −27.2 dB (with standard deviation 1.7), while the gaussianization method has an average performance index of −8.9 dB (with standard deviation 4.5) and −8.7 dB (with standard deviation 3.8). In cases when the linear mixture is far from gaussian, the assumptions used in both gaussianization and the extended gaussianization method break down. As a demonstration, we set the (2, 2)th entry of the mixing matrix A as 0 and repeat the above experiment. Now the second linear mixture t2 is far from gaussian, as seen from the distribution of s1 shown in Figure 5A. Both gaussianization and the extended gaussianization method

Extended Gaussianization for Separation of PNL Mixtures

447

behave poorly. The performance index of the gaussianization method is −8.6 dB and −9.9 dB. And the performance index of the extended gaussianization method is −9.1 dB and −9.6 dB. In another experiment, we modify the nonlinear functions fi as  if t1 ≥ 0 (5t√1 )3 , f1 (t1 ) = − −t1 , if t1 < 0  t + 2 tanh(t2 ), if t2 ≥ 0 f2 (t2 ) = 2 t2 , if t2 < 0. The nonlinearities are neither smooth nor symmetrical. Other settings are the same as the first experiment. The separation result by the extended gaussianization method is exactly the same as that in the first experiment. This verifies the robustness of our method to different nonlinearities. We also test our method with natural audio sources. The two sources are a speech signal (with kurtosis of 6.1) and a music signal (with kurtosis 0.5), as shown in Figure 10A. They are mixed by a randomly chosen matrix. The linear mixtures are shown in Figure 10B. The nonlinearities in both channels are the same as in the first experiment. Since the separation result is insensitive to the nonlinearities, the PNL mixtures are not given in the figure. The learning rates in the nonlinear stage and the linear stage are both 0.2. After about 900 iterations, the learning process converges. The parameters at convergence are κ3,1 = −0.04, κ4,1 = 1.06, κ3,2 = −0.02, and κ4,2 = 0.34. The recovered sources are shown in Figure11A, with the performance index C(s1 , y1 ) = −18.1 dB, C(s2 , y2 ) = −25.6 dB. From the performance index, we can see the extended gaussianization exhibits a good separation performance. This can also be confirmed by listening to the recovered signals and the original sources—we perceive very little cross talk between them. Figure 11B shows the recovered signals by gaussianization. And Figure 11C compares the residual distortion obtained using these two methods. Clearly the separation result of the extended gaussianization method is much better. 6 Conclusion and Discussion In this article, we propose the extended gaussianization method to tackle the problem of blind source separation of post-nonlinear mixtures when the linear mixture of independent sources is close to gaussian. The gaussianization technique assumes the linear mixtures are gaussian and estimates the inverse nonlinearities directly. Our method extends gaussianization for the cases when the linear mixtures are not exactly gaussian, just not far from gaussian. Gaussianization can be considered a preprocessing step for our method. After this step, the inverse of remaining nonlinear distortion is modeled by the Cornish-Fisher expansion. Consequently, the inverse nonlinearity can be learned with only a few unknown parameters. The region of these parameters guaranteeing the validity of the Cornish-Fisher expansion

448

K. Zhang and L.-W. Chan

s1

A 0.5 0 −0.5 0

500

1000

1500

2000

2500

0

500

1000

1500

2000

2500

−1 0 0.5

500

1000

1500

2000

2500

500

1000

1500

2000

2500

s2

0.5 0 −0.5

B

t

2

t

1

1 0

0

−0.5 0

Figure 10: Two-channel audio data set. (A) Waveforms of the original sources. (B) Linear mixtures of the original sources.

is also derived. These parameters, as well as the linear demixing matrix, are learned based on the minimization of mutual information between outputs. Although the region indicates the linear mixtures should not be far away from gaussian, this is a reasonable assumption according to the central limit theorem. In the noiseless case, the performance of this method is not affected by different forms of the nonlinearity generating observations. Compared to the gaussianization method, our method extends the region of validity to include distributions in the vicinity of gaussian using parameters adaptation. Compared with other adaptation methods for blind separation of PNL mixtures, our method reduces the problem complexity and simplifies the learning procedure by using the assumption that linear mixtures of independent variables are close to gaussian. The nonlinearity is modeled with the help of the Cornish-Fisher expansion, and this specific parametric model has good flexibility and adaptability under the assumption of our method. Experimental results have been given to support the theoretical claims. In summary, our extended gaussianization method combines two major approaches for blind separation of post-nonlinear mixtures and gives good separation performance with low computational cost.

Extended Gaussianization for Separation of PNL Mixtures

449

A y1

0

y2

5

−5 0 4 2 0 −2 −4 0

B

500

1000

1500

2000

2500

500

1000

1500

2000

2500

500

1000

1500

2000

2500

500

1000

1500

2000

2500

y1

5 0

y

2

−5 0 5 0

−5 0

C 5

Channel 2

Channel 1

0

Extended Gaussianization Gaussianization −5 −1

−0.5

0

0.5

1

Figure 11: Separation result of PNL mixtures of audio signals. (A) Recovered sources using the extended gaussianization method. Performance index: C(s1 , y1 ) = −18.1 dB, C(s2 , y2 ) = −25.6 dB. (B) Recovered sources with gaussianization. Performance index: C(s1 , y1 ) = −10.6 dB, C(s2 , y2 ) = −17.0 dB. (C) Residual distortion gi ◦ fi in each channel.

450

K. Zhang and L.-W. Chan

Acknowledgments The work reported here was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administration Region, China. We are very grateful to the anonymous referees for their valuable comments and suggestions, which led to a strong improvement of this article. References Abramowitz, M., & Stegun, I. A. (1968). Handbook of mathematical functions. New York: Dover. Amari, S.-I. (1998). Natural gradient works efficiently in learning. Neural Comput., 10, 251–276. Amari, S.-I., & Cardoso, J.-F. (1997). Blind source separation—semiparametric statistical approach. IEEE Trans. on Signal Processing, 45, 2692–2700. Amari, S.-I., Cichocki, A., & Yang, H. H. (1996). A new learning algorithm for blind signal separation. In D. Touretzky, M. Mozer, M. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 757–763). Cambridge, MA: The MIT Press. Attias, H. (1999). Independent factor analysis. Neural Computation, 11, 803–851. Bell, A. J. (2003). The co-information lattice. Proc. ICA2003 (pp. 921–926). Nara, Japan. Bell, A. J., & Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7, 1129–1159. Belouchrani, A., Abed-Meraim, K., Cardoso, J., & Moulines, E. (1997). A blind source separation technique using second-order statistics. IEEE Trans. on Signal Processing, 45, 434–444. Breiman, L., & Friedman, J. (1985). Estimating optimal transformations for multiple regression and correlation. Journal of the American Statistical Association, 80, 580–598. Cardoso, J.-F. (1997). Infomax and maximum likelihood for source separation. IEEE Letters on Signal Processing, 4, 112–114. Cardoso, J.-F. (1998). Blind signal separation: Statistical principles. Proceedings of the IEEE, 9, 2009–2025. Cardoso, J.-F., & Laheld, B. (1996). Equivariant adaptive source separation. Signal Processing, 44, 3017–3030. Cardoso, J.-F., & Souloumiac, A. (1993). Blind beamforming for non-gaussian signals. IEE Proceeding-F, 140, 362–370. Chen, S. S., & Gopinath, R. A. (2001). In T.K. Leen, T.G. Dietterich, & V. Tresp (Eds.), Gaussianization. Advances in neural information processing systems 13 (pp. 423–429). Cambridge, MA: MIT Press. Comon, P. (1994). Independent component analysis—a new concept? Signal Processing, 36, 287–314. Cornish, E. A., & Fisher, R. A. (1937). Moments and cumulants in the specification of distributions. Review of the International Statistical Institute, 5, 307–320.

Extended Gaussianization for Separation of PNL Mixtures

451

Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley. Gill, P. E., Murray, W., & Wright, M. H. (1981). Practical optimization. New York: Academic Press. Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2, 359–366. Hyv¨arinen, A. (1999a). Fast and robust fixed-point algorithms for independent component analysis. IEEE Transactions on Neural Networks, 10(3), 626–634. Hyv¨arinen, A. (1999b). Survey on independent component analysis. Neural Computing Surveys, 2, 94–128. Hyv¨arinen, A., Karhunen, J., & Oja, E. (2001). Independent component analysis. New York: Wiley. Hyv¨arinen, A., & Oja, E. (2000). Independent component analysis: Algorithms and applications. Neural Networks, 13, 411–430. Hyv¨arinen, A., & Pajunen, P. (1999). Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12, 429–439. Jaschke, S. R. (2002). The Cornish-Fisher expansion in the context of deltagamma-normal approximations. Journal of Risk, 4, 33–52. Johnson, N. L., & Kotz, S. (1970). Continuous univariate distributions (Vol. 1). New York: Wiley. Jutten, C., & H´erault, J. (1991). Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture. Signal Processing, 24, 1–10. Jutten, C., & Taleb, A. (2000). Source separation: From dusk till dawn. In 2nd International Workshop on Independent Component Analysis and Blind Signal Separation (ICA 2000) (pp. 15–26). Helsinki, Finland. Lee, T.-W., Girolami, M., & Sejnowski, T. (1999). Independent component analysis using an extended infomax algorithm for mixed sub-gaussian and supergaussian sources. Neural Computation, 11, 417–441. Lee, T.-W., Koehler, B., & Orglmeister, R. (1997). Blind source separation of nonlinear mixing models. In N. Morgan, J. Principe, E. Wilson, & L. Gile (Eds.), Neural Networks for Signal Processing, 7 (pp. 406–415). Piscataway, NJ: IEEE Press. Mansour, A., Barros, A., & Ohnishi, N. (2000). Blind separation of sources: Methods, assumptions and applications. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, Special Section on Digital Signal Processing in IEICE EA., E83-A, 1498–1512. Neuts, M. F. (Ed.). (1995). Advanced probability theory (2nd ed.). New York: Marcel Dekker. Oja, E. (1997). The nonlinear PCA learning rule in independent component analysis. Neurocomputing, 17, 25–45. Pham, D.-T. (2000). Blind separation of instantaneous mixture of sources based on order statistics. IEEE Trans. on Signal Processing, 48, 363–375. Pham, D.-T. (2002). Mutual information approach to blind separation of stationary sources. IEEE Trans. on Information Theory, 48, 1935–1946. Pham, D.-T., & Garat, P. (1997). Blind separation of mixture of independent sources through a quasi-maximum likelihood approach. IEEE Trans. on Signal Processing, 45, 1712–1725.

452

K. Zhang and L.-W. Chan

Sol-Casals, J., Babaie-Zadeh, M., Jutten, C., & Pham, D.-T. (2003). Improving algorithm speed in PNL mixture separation and Wiener system inversion. Proc. ICA2003 (pp. 639–644). Nara, Japan. Taleb, A., & Jutten, C. (1997). Nonlinear source separation: The post-nonlinear mixtures. In Proc. ESANN (pp.179–284). Bruge, Belgium. Taleb, A., & Jutten, C. (1999a). Batch algorithm for source separation in postnonlinear mixtures. In Proc. First Int. Workshop on Independent Component Analysis and Signal Separation (ICA’99) (pp. 155–160). Aussois, France. Taleb, A., & Jutten, C. (1999b). Source separation in post-nonlinear mixtures. IEEE Trans. on Signal Processing, 47, 2807–2820. Xu, L., Cheung, C., Yang, H., & Amari, S.-I. (1997). Independent component analysis by the information-theoretic approach with mixture of densities. Proc. of 1997 IEEE Intl. Conf on Neural Networks (IEEE-INNS IJCNN97) (pp. 1821–1826). Houston, TX. Yang, H. H., Amari, S.-I., & Cichocki, A. (1998). Information-theoretic approach to blind separation of sources in nonlinear mixture. Signal Processing, 64, 291–300. Ziehe, A., Kawanabe, M., Harmeling, S., & Mueller, K.-R. (2001). Separation of post-nonlinear mixtures using ACE and temporal decorrelation. In Proc. Int. Workshop on Independent Component Analysis and Blind Signal Separation (ICA2001) (pp. 433–438). Ziehe, A., Kawanabe, M., Harmeling, S., & Mueller, K.-R. (2003). Blind separation of post-nonlinear mixtures using Gaussianizing transformations and temporal decorrelation. InProc. ICA2003 (pp. 269–274). Received November 7, 2003; accepted July 2, 2004.

LETTER

Communicated by Stephen Roberts

Bayesian Analysis of Nonlinear Autoregression Models Based on Neural Networks A. Menchero [email protected]

R. Montes Diez [email protected]

D. R´ıos Insua [email protected] GECD, Rey Juan Carlos University, Madrid 28925, Spain

P. Muller ¨ [email protected] M. D. Anderson Cancer Center. University of Texas, Houston, TX 77030, U.S.A.

We show how Bayesian neural networks can be used for time-series analysis. We consider a block-based model building strategy to model linear and nonlinear features within the time series: a linear combination of a linear autoregression term and a feedforward neural network (FFNN) with an unknown number of hidden nodes. To allow for simpler models, we also consider these terms separately as competing models to select from. Model identifiability problems arise when FFNN sigmoidal activation functions exhibit almost linear behavior or when there are almost duplicate or irrelevant neural network nodes. New reversible-jump moves are proposed to facilitate model selection, mitigating model identifiability problems. We illustrate this methodology analyzing several timeseries data examples.

1 Introduction Many neural network models have been applied to time-series analysis and forecasting. There has been some interest in using recurrent networks such as Elman networks (Elman, 1990), Jordan networks (Jordan, 1986), and realtime recurrent learning networks (Williams & Zipser, 1989). However, the model most frequently used is the multilayer feedforward neural network (FFNN). Many articles compare FFNNs and standard statistical methods for time-series analysis (e.g., Tang, de Almeida, & Fishwick, 1991; Foster, Collopy, & Ungar, 1992; Stern, 1996; Hill, O’Connor, & Remus, 1996; Faraway & Chatfield, 1998). Several articles have found FFNN superior to linear methods such as ARIMA models for several time-series problems and Neural Computation 17, 453–485 (2005)

c 2005 Massachusetts Institute of Technology


454

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

comparable to other nonlinear methods like generalized additive models or projection pursuit regression. We will consider FFNNs to model nonlinear autoregressions. The net output will represent the time-series predicted value, when past values of the series are given as net inputs. Fitting an FFNN model requires many choices about the model structure: activation functions, number of hidden layers, number of hidden nodes, inputs, and so on. A nonlinear optimization algorithm is typically used to estimate weights to optimize some performance criterion, for example, minimization of mean square error, with weight decay. To fix the model structure, rules of thumb traditionally are used. Remus, O’Connor, and Griggs (1998) suggest appropriate rules for FFNN in time-series forecasting. Alternatively, a Bayesian approach to FFNN modeling (Mackay, 1992; Neal, 1996; Muller ¨ & R´ıos Insua, 1998; R´ıos Insua and Muller, ¨ 1998; Holmes & Mallick, 1998; Andrieu, de Freitas, & Doucet, 1999) provides a coherent framework to deal with these issues. FFNN parameters are regarded as random variables whose posterior distribution is inferred in the light of data. Most important, perhaps, we may include the number of hidden nodes as an additional parameter and model its uncertainty. Predictions are obtained by averaging over all possible models and parameter values according to their posterior distributions. In this letter, we introduce a Bayesian FFNN forecasting model for timeseries data. We present an inference scheme based on Markov chain Monte Carlo (MCMC) simulation. To deal with model selection, the standard birth and death reversible-jump moves developed by Muller ¨ and R´ıos Insua (1998) result in a slowly mixing MCMC. Instead, we propose new reversiblejump moves to add or delete special kinds of nodes characterized as linearized, irrelevant, or duplicate. Two examples are used to illustrate the methodology: the lynx time-series data (Priestley, 1988) and the airline passengers data (Box & Jenkins, 1970). 2 Model Definition Consider univariate time-series data {y1 , y2 , . . . , yN }. We would like to model the generating stochastic process in an autoregressive fashion, N 
 p y1 , y2 , . . . , yN = p(y1 , . . . , yq ) p(yt | yt−1 , yt−2 , . . . , yt−q ). t=q+1

We shall assume that each yt is modeled by a nonlinear autoregression function of q past values plus a normal error term: yt = f (yt−1 , yt−2 , . . . , yt−q ) + t , t = q + 1, . . . , N t ∼ N(0, σ 2 ),


so that yt | yt−1 , yt−2 , . . . , yt−q ∼ N f (yt−1 , yt−2 , . . . , yt−q ), σ 2 , t = q + 1, . . . , N.

Bayesian Analysis of Nonlinear Autoregression Models

455

We complete model specification using a block-based strategy to describe f . We propose a mixed model as a linear combination of a linear autoregression term and an FFNN. An FFNN model with q input nodes, one hidden layer with M hidden nodes, one output node, and activation function ϕ is a model relating a response variable  yt and q explanatory variables, in our case, xt = (yt−1 , . . . , yt−q ):  yt (xt ) =

M 

βj ϕ(x t γj + δj ),

j=1

with βj ∈ R, γj ∈ Rq . Biases δj may be assimilated to the rest of the γj vectors if we consider an additional input with constant value one, say, xt = (1, yt−1 , . . . , yt−q ), so that, γj = (γ0j , γ1j, . . . , γqj ) ∈ Rq+1 . To be specific, in the following we will assume a logistic activation function: ϕ(z) = exp(z)/(1 + exp(z)). However, the discussion remains valid for any other sigmoidal function. In the proposed model, the linear term accounts for linear features, whereas the FFNN term could take care of nonlinear ones: f (yt−1 , yt−2 , . . . , yt−q ) = x t λ +

M 

βj ϕ(x t γj ), t = q + 1, . . . , N.

(2.1)

j=1

Initially, the parameters in our mixed model are the linear coefficients λ = (λ0 , λ1 , . . . , λq ) ∈ Rq+1 , the hidden to output weights β = (β1 , β2 , . . . , βM ), the input to hidden weights γ = (γ1 , γ2 . . . , γM ), and the error variance σ 2 . As there will be uncertainty about the number of hidden nodes M to include, we shall model this uncertainty considering M as an unknown parameter. Note that the mixed model, equation 2.1, embeds, as a particular case, the standard linear autoregression model, when M = 0, and the FFNN model, when λ = 0. We shall assume the autoregressive order q to be known in advance. If desired, one could model uncertainty about q as a problem of variable selection in a model selection context. To allow for simpler models, we also consider the linear and nonlinear terms separately as competing models to select from: a simple linear autoregression model,  yt = f (yt−1 , yt−2 , . . . , yt−q ) = x t λ, t = q + 1, . . . , N,

(2.2)

and a nonlinear autoregression feedforward neural net model,  yt = f (yt−1 , yt−2 , . . . , yt−q ) =

M  j=1

for each value of M.

βj ϕ(x t γj ), t = q + 1, . . . , N,

(2.3)

456

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

As in R´ıos Insua and Muller ¨ (1998), we assume a normal/inverse gamma prior βj ∼ N(µβ , σβ2 ), λ ∼ N(µλ , σλ2 I) γj ∼ N(µγ , γ ), σ 2 ∼ InvGamma(aσ , bσ ).

(2.4)

When there is nonnegligible uncertainty about prior hyperparameters, we may extend the prior model with additional hyperpriors. We shall use the following standard conjugate choices in hierarchical models: µβ ∼ N(aµβ , bµβ ), σβ2 ∼ InvGamma(aσβ , bσβ ) µλ ∼ N(aµλ , bµλ ), σλ2 ∼ InvGamma(aσλ , bσλ ) µγ ∼ N(aµγ , bµγ ), γ ∼ InvWishart(aλ , bλ ).

(2.5)

Hyperparameters are a priori independent. Given hyperparameters, parameters are a priori independent. Since the likelihood is invariant with respect to relabelings, we include an order constraint to avoid trivial posterior multimodality due to index permutation. For example, we may use γ1q ≤ γ2q · · · ≤ γMq . 3 MCMC Posterior Inference with a Fixed Model Consider first the mixed model 2.1 with fixed M. The complete likelihood for a given data set D = {y1 , y2 , . . . , yN } is p(D | λ, β, γ , σ 2 ) = p(y1 , . . . , yq | λ, β, γ , σ 2 )p(D | y1 , . . . , yq , λ, β, γ , σ 2 ), where D = {yp+1 , . . . , yN } and p(D | y1 , . . . , yq , λ, β, γ , σ 2 ) =

N 

p(yt | yt−1 , yt−2 , . . . ,t−q , λ, β, γ , σ 2 )

t=q+1

is the conditional likelihood given first q values. From here on, we will make inference conditioning on the first q values, assuming they are known without uncertainty (alternatively, we could include an informative prior over first q values in the model and perform inference with the complete likelihood). Together with the prior assumptions 2.4 and 2.5, the joint posterior distribution is given by p(λ, β, γ , σ 2 , χ | D ) ∝ p(yq+1 , . . . , yN | y1 , . . . , yq , λ, β, γ , σ 2 ) 2

p(λ, β, γ , σ , χ)M!,

(3.1)

Bayesian Analysis of Nonlinear Autoregression Models

457

where   p(λ, β, γ , σ 2 , χ) = p µλ , σλ2 , µβ , σβ2 , µγ , γ p(σ 2 ) p(λ | µλ , σλ2 I)p(β | µβ , σβ2 I)

M 

p(γi | µγ , γ )

i=1

is the joint prior distribution, χ = (µλ , σλ2 , µβ , σβ2 , µγ , γ ) is the set of hyperparameters, and M! appears because of the order constraint on γ . As in Muller ¨ and R´ıos Insua (1998), we propose a hybrid, partially marginalized MCMC posterior sampling scheme to implement inference in the fixed architecture mixed model, equation 2.1. We use a Metropolis step to update the input to hidden weights γj using the marginal likelihood over (β, λ): p(D | y1 , . . . , yq , γ , σ 2 ) to partly avoid the random walk nature of the Metropolis algorithm: 1. Given current values of χ and σ 2 (β and λ are marginalized), for each γj , j = 1, . . . , M, generate a proposal  γj ∼ N(γj , cγ ), calculate the acceptance probability,

p( γ | µγ , γ )p(D | y1 , . . . , yq ,  γ , σ 2) a = min 1, p(γ | µγ , γ )p(D | y1 , . . . , yq , γ , σ 2 )

where γ =  γ1 , . . . , γj−1 , γj , γj+1 , . . . , γM , and  γj , γ = γ1 , . . . , γj−1 ,  γj+1 , . . . , γM . With probability a, replace γ by  γ and rearrange indices if necessary to satisfy order constraint. Otherwise, leave γj unchanged. 2. Generate new values for parameters, drawing from their full conditional posteriors:   ∼ p β | D , γ , λ, σ 2 , χ is a multivariate normal distribution. β    λ ∼ p λ | D , γ , β, σ 2 , χ is a multivariate normal distribution.   2 ∼ p σ 2 | D , γ , β, λ is an inverse gamma distribution. σ 
3. Given current values of γ , β, λ, σ 2 , generate a new value for each hyperparameter by drawing from their complete conditional posterior distributions:   µ β ∼ p µβ | D , β, σβ2 is a normal distribution.   2 ∼ p σ 2 | D , β, µ is an inverse gamma distribution. σ β β β   µ λ ∼ p µλ | D , λ, σλ2 is a multivariate normal distribution.

458

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

  2 ∼ p σ 2 | D , λ, µ is an inverse gamma distribution. σ λ λ λ 
µ γ ∼ p µγ | D , γ , γ is a multivariate normal distribution.
 γ ∼ p γ | D , γ , µγ is an inverse Wishart distribution.  A similar sampling scheme can be used when using a neural net model without linear term 2.3, but likelihood marginalization would be just over β. Posterior inference with the normal linear autoregression model 2.2 is straightforward (see Gamerman, 1997). 4 Modeling Uncertainty About the Architecture We now extend the model to include inference about model uncertainty. In fact, the posterior distribution 3.1 should be written including a reference to the model k considered:
 p(λ, β, γ , σ 2 , χ | D , k) = p θk | D , k ,
 where θk = λ, β, γ , σ 2 , χ represents parameters and hyperparameters in model k. We index models with a pair of indexes mhk , where h = 0(1) indicates absence (presence) of the linear term; k = 0, 1, . . . indicates the number of hidden nodes in the NN term. Therefore, m10 is the linear autoregression model (see equation 2.2), m0k is the FFNN model (see equation 2.3) with k hidden nodes, and m1k is the mixed models (see equation 2.1) with k hidden nodes. Note that models m0k , k ≥ 1 are nested, as well as m1k , k ≥ 1. It would be possible to think of the linear model m10 as a degenerate case of the mixed model m11 when β = 0 and of models m0k , k ≥ 1, as degenerate m1k , k ≥ 1 when λ = 0 and finally consider all models above as nested models. However, given our model exploration strategy outlined below, we prefer to view them as nonnested. We wish to design moves between models to get good coverage of the model space (see Figure 1). When dealing with nested models, it is common to add or delete model components, using add/delete or split/combine move pairs (Green, 1995; Richardson & Green, 1997). Similarly, we could define two reversible-jump pairs: add/delete an arbitrary node selected at random and add/delete the linear term. With such strategy, described in Muller ¨ and R´ıos Insua (1998), it would be possible to reach a model from any other model. However, our experience with time-series data shows that the acceptance rate of such an algorithm is low, that the model space is not adequately covered, and we have identified cases of nonconvergence. That happened, for example, when we tried to model the lynx time series and the airline passengers data sets used in section 5 as numerical examples of our methodology.

Bayesian Analysis of Nonlinear Autoregression Models j2a j3a

m10

j2a j3a j4a

m11 j2d j3d

j2a j3a j4a

j2a j3a j4a

m12 j2d j3d j4d

j1a

j1d

j1d

j2a j3a j4a

...

j2d j3d, j4d j1a

j1d

j2a j3a j4a

m02 j2d j3d j4d

m1k+1

m1k j2d j3d j4d

j1a

m01

459

j1d

j2a j3a j4a

m0k j2d j3d j4d

j1a

m0k+1 j2d j3d j4d

Figure 1: Possible models along with moves available from each model.

To avoid this, let us introduce the notion irrelevant,
of LID (linearized, 
and duplicate) nodes. Consider a node βj , γj with γj = γ0j , γ1j, . . . , γpj in a model m0k or m1k . We call it: Definition 1. (Approximately) linearized if its output βj ϕ(x t γj ) is nearly a hyperplane in the input-output space for the input data range. Definition 2.

(Approximately) irrelevant if βj  0.

Definition 3. We say nodes (βj , γj ) and (βk , γk ) are (approximately) duplicate nodes if γj − γk   0. Note that a model with an (approximately) irrelevant node has (almost) identical likelihood as a model without it. A model with (approximately) du-
plicate nodes will perform similarly to a model with a single node βj +βk , γj instead of them. Finally, the linear behavior of a linearized node may be assimilated into a linear term or another linearized nodes. The appearance of LID nodes therefore may cause problems of model identifiability and multimodality in model space. Recall that the acceptance probability of a proposed model is computed using the marginal likelihood over β and/or λ whenever possible. This marginalization accelerates convergence, but γ is not easily marginalized. Thus, any proposed model structure change is evaluated with the currently imputed value of γ . Adding irrelevant, duplicate, or linearized nodes implies less structural change, so that in our simulation, it was more common to accept add moves that actually add LID nodes. And, vice versa, it is very unlikely to accept a jump to a simpler model than propose an arbitrary node to be deleted. In fact, it is usually easier for a new LID node to be added before old ones are deleted. This motivates an MCMC scheme that includes add/delete of LID nodes as additional moves to a standard reversible-jump scheme. Once moves to delete LID nodes are defined, we shall propose the corresponding add nodes

460

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

moves. Note that whereas delete LID nodes would mitigate our problem, add LID nodes would seem to complicate our scheme, following our above discussion. In principle, we could propose these add LID nodes with low probability. However, besides ensuring balance, add moves have a useful side effect. Note that with add moves, we have control over when and how an LID node is added. Adding LID nodes will have more probability of being accepted than adding an arbitrary node because it usually implies a smaller structural change. We expect that once a model with a new LID node becomes the current model, the next iterations will delete some other LID nodes. In this way, more complex models have a chance of being visited, so it can help to get a wide coverage of model space. In a sense, this is related to some global search optimization techniques, like simulated annealing (Geman & Geman, 1984), in which a worse solution is sometimes proposed to escape from a local minima, with the hope of reaching a better, possibly global, minima from it. As Bayesian inference implicitly embodies Occam’s razor principle and clever delete moves are defined, we expect simpler models to be more common. In the rest of this section, we show how to characterize a linearized node so as to generate it at random or propose deletion. We also introduce modifications to a basic thin/seed move to deal with duplicate nodes, and we put an ad hoc distribution over nodes to increase the probability of an irrelevant node to be proposed for deletion. Note that there is a decision to make about how deterministic a move should be. For example, we could look deterministically for two (approximately) duplicate nodes in the set of nodes and, if they exist, delete them, but in general, it is good to allow for some randomness in move definitions (Green, 1995). At the end of this section, a complete reversible-jump scheme is outlined. A full description of add/delete moves is given in the appendixes. 4.1 (Approximately) Linearized Nodes. To facilitate explanation, consider an FFNN model in a regression problem:  y(x1 , . . . , xq ) = f (x1 , . . . , xq ) =

M 

βj ϕ(ζj )

j=1

ζj (x1 , . . . , xq ) = γj0 + γj1 x1 + · · · + γjq xq . The net output  y is given by a linear combination of hidden node outputs. Each hidden node output ϕ(ζj ) = 1+exp1 −ζ , j = 1, . . . , M, represents a ( j) is approximately linear in its middle range sigmoidal surface in Rq+1 , which
 (see Figure 2). For any data x , i = 1, . . . , N, we may find weights , . . . , x i1 iq
 γj0 , γj1 , . . . , γjp so that −ρ ≤ ζj (x1 , . . . , xq ) ≤ ρ, i = 1, . . . , N,

(4.1)

Bayesian Analysis of Nonlinear Autoregression Models

461

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 6 0.2

4 2

0.1 0 0 −6

−2 −4

−2

0

−4 2

4

6

−6

Figure 2: Sigmoidal surface when input space dimension is q = 2.

where ρ and ϕ(ζj ) resembles a hyperplane in the input data domain. Note also that when ζj > η, say η ≥ 4, the sigmoid is saturated (see Figure 3) and ϕ(ζj )  1. It is possible to adjust weights so that saturation is verified for the whole data set. When fitting an FFNN model, some hidden nodes will ocasionally work in the linear zone, behaving approximately as linear regression terms, or in the saturation zone, behaving as a constant term. For example, in Figures 4 and 5, an FFNN with four nodes fits a cosine function. The joint output of three of the four nodes has a linear trend, but none of them is working in the linear zone. The fourth node helps to compensate this linear trend so that the cosine is fitted. Also, when using mixed models, some nodes may add up their linear behavior to the linear term. Appendix A describes computational aspects of the derivation for generation and deletion of linearized nodes. 4.2 (Approximately) Irrelevant Nodes. We let chance deal with the generation of an irrelevant node. We modify a basic birth-death move when selecting a node for deletion. We choose a node at random with probabilities ψ −1 2 pj = M+1j , j = 1, . . . , M where ψj = exp{ 2 2 , βj }, so that more weight k=1

ψk

is given to those nodes whose βj is almost zero. (Full details are given in section B.3.)

462

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨ 1.0

ϕ(ζj) 0.8

0.6

0.4

0.2

ζj

0.0

-5

-4

-3

-2

-ro -1

0

+ro 1

2

3

4

5

20

Figure 3: When −ρ ≤ z ≤ ρ, sigmoid function could be well approximated by its tangent in z = 0.

-10

0

10

joint output first 3 nodes

-20

output 4th node

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Figure 4: Feedforward neural network with four nodes fitting a cosine function. Three of the four nodes fit the cosine waves with a global up-linear trend. The fourth node adds up its linear behavior to compensate the trend.

4.3 (Approximately)
 Duplicate Nodes. Consider two (approximately) duplicate nodes βj , γj and (βk , γk ) in a model m0k
or m1k . We could propose a simpler model, collapsing them into a new node βj + βk , γj . The proposed add or delete pair is a basic thin or seed move with minor modifications.

463

-1.0

-0.5

0.0

0.5

1.0

Bayesian Analysis of Nonlinear Autoregression Models

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Figure 5: The joint output of the four nodes in the neural net model resembles the target cosine function.


To add a duplicate node, take a node βj , γj at random, and introduce small perturbations
to produce 
two (approximately)
 duplicate new nodes, j ,  j+1 ,  hence substituting βj , γj by β γj and β γj+1 with j = βj (1 − v), β j+1 = βj v β γj = γj , 

γj+1 = γj + δ. 

Perturbations v and δ are generated from v ∼ Beta(2, 2)
 δ ∼ N 0, cγ , where c is a small enough constant (say, c = 0.01). As we introduced an order constraint on γ , the add move proposed will be rejected if  γj and  γj+1 do not satisfy the order constraint.
 Given the on γ , we expect that for a given βj , γj , the
order constraint  next node βj+1 , γj+1 could be an almost duplicate version of it. Then we

 j = βj + βj+1 and propose removing βj+1 , γj+1 and transforming βj , γj as β γj = γj . (See appendix B for full details.)  4.4 Our Reversible-Jump Scheme. We propose the following moves: 1. Add or delete linear term (j1a , j1d ). 2. Add or delete linearized node (j2a , j2d ). 3. Add or delete irrelevant nodes (j3a , j3d ).

464

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

4. Add or delete duplicate nodes (j4a , j4d ). Not all of them are reachable from a given model. For example, we cannot delete a linear term when the current model is the linear model, or delete duplicate nodes if we have just one node (see Figure 1). Specifically: • From the linear autoregression model m10 , valid moves are j2a and j4a . • From a feedforward neural net model with one node m01 , valid moves are j1a , j2a , j3a , and j4a . • From a feedforward neural net model with k ≥ 2 nodes m0k , valid moves are j1a , j2a , j2d , j3a , j3d , j4a , and j4d . • From a mixed model with k ≥ 1 and the nodes m1k , valid moves are j1d , j2a , j2d , j3a , j3d , j4a , and j4d . We will assume that from a given model, all reachable moves are equally likely. Formally, unreachable moves are given zero probability. Note that the definition of each move depends on the current model where to jump from. For example, adding a linearized node when the current model is m10 involves proposing neural network parameters and hyperparameters absent in m10 . However, when the current model is m0k , k ≥ 1, adding a linearized node is a jump between nested models with shared parameters and hyperparameters. Shared parameters and hyperparameters could be proposed with the same values as in the previous model. But unshared parameters and hyperparameters have to be proposed from scratch, for example, from their prior distribution. This usually implies less of a chance for the move to be accepted. It would be useful for convergence purposes to generate unshared parameters using proposals centered in the values they had the last time the model was visited. We shall not deal with this matter here and will use priors as proposals. Similar comments apply to other moves. (See appendix B for a complete description.) The general reversible-jump posterior inference scheme is as follows: 1. Start with an initial model mhk , h = {0, 1}, k = 0, 1, . . . and initial values for its parameters (and hyperparameters if needed) (for example, prior means). Until convergence is achieved, iterate through steps 2 to 4: 2. With probability p1 (say, p1 = 0.5), decide to stay within the current model; otherwise (with probability 1 − p1 ), decide to move to another model. 3. If staying in the current model, perform the MCMC scheme described in section 3. 4. If moving to another model, select at random a move from the list of reachable moves with probabilities assigned as mentioned and propose a new model accordingly. If accepted, the new model is the current model.

465

4000 0

1000

2000

3000

lynx

5000

6000

7000

Bayesian Analysis of Nonlinear Autoregression Models

1820

1840

1860

1880

1900

1920

Time

Figure 6: Annual number of lynx trappings in the Mackenzie River District of Northwest Canada, 1821–1934 (Priestley, 1988).

5 Examples We illustrate the methodology with two well-known time-series data sets and two simulated examples. 5.1 Lynx Data. First, we consider time-series data giving the annual number of lynx trappings in the Mackenzie River District of Northwest Canada for the period 1821 to 1934 (Priestley, 1988; see Figure 6). In neural network applications, the data set is often split into two subsets: one for estimation and the other for validation. We split the lynx data set into a training data set (the first 72 observations) and a test data set (the last 38 observations). Prior distributions for the unknown parameters and hyperparameters in the model are chosen as described in section 2. As for the choice of the hyperhyperparameters, our experience shows that at that level of the hierarchical model, the results do not change much when using different values. Here,

466

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

15 Initial value M = 0 Initial value M = 15

M

10

5

0

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Figure 7: Lynx data. Trace plots of the Markov chains for the number of hidden nodes, from two well-dispersed starting points, M = 0 and M = 15.

we used the following choices for the hyperparameter distributions: µβ ∼ N(0, 3),

σβ2 ∼ InvGamma(9, 1)

µλ ∼ N((0, 0, 0), 3I),

σλ2 ∼ InvGamma(9, 1)

µγ ∼ N((0, 0, 0), 3I),

γ ∼ InvWishart(10, 2.5I)

2

σ ∼ InvGamma(1, 1) Also, the unknown number of nodes M is given a geometric prior distribution with parameter α = 14 , which implies a prior mean E(M) = 4. Two runs were carried out from different starting points: M = 0 and M = 15. A burn–in of 1000 iterations was used, and then 9000 additional iterations were monitored for inference purposes. Figures 7 and 8 show trace plots of the Markov chains for the number of nodes in the FFNN and histogram of the posterior distribution of M, respectively, suggesting M = 2 as the most likely number of nodes for the hidden layer of the FFNN term, although M = 1, 3, 4 received nonnegligible probability. In any case, the analysis clearly indicates the presence of important nonlinearity. Finally, Figure 9 shows the observed values (+) (note that the time series has been log transformated and detrended), the one-step-ahead forecast values (solid line), and 90% probability interval (dash-dot line), showing good performance for our LID-RJMCMC scheme. For comparison, we show the predictions obtained with the unscented particle filter algorithm proposed by van der Merwe, Doucet, de Freitas, & Wan (2000); dotted line). Their algorithm consists of a particle filter (PF) that uses an unscented Kalman filter (UKF) to generate the importance proposal distribution. Our method seems to perform similarly to the PF-UKF algorithm. 5.2 Airline Data. As a second example, we consider the well-known international airline data by Box and Jenkins (1970). This time series is one of the most commonly used seasonal ARIMA models. It was modeled by

Bayesian Analysis of Nonlinear Autoregression Models

467

Histogram of M 0.35

0.30

0.25

0.20

0.15

0.10

0.05

0.00

0

1

2

3

4

5

6

7

M Figure 8: Lynx data. Histogram of the posterior distribution of M, suggesting M = 2 nodes for the hidden layer of the FFNN term.

Box and Jenkins as ARIMA(011)(011)12 and is usually known as airline model due to this time series. Exploratory data analysis suggests seasonal behavior of the time series with similar properties exihibited every 12 observations (every 12 months). We shall therefore model each yt on the basis of the inmediately past values, yt−1 , yt−2 as well as corresponding seasonal values yt−12 , yt−13 . We initialized the MCMC algorithm by two well-dispersed values for the number of hidden nodes M and let the reversible-jump MCMC algorithm described run for a sufficient number of iterations: 2000 burn-in and another 18,000 iterations for making inferences. Similar comments about prior choices in the previous example apply here. Figure 10 shows the histogram of the posterior distribution of M; in this case, M = 3 seems to be the most likely number of nodes, but probabilities for M = 2, 4, 5 are also quite important. It is of interest to look at the reversible-jump algorithm development. Table 1 shows the number of times each move has been proposed and accepted. Note that acceptance rates are considerably larger for irrelevant nodes than for linearized or duplicate nodes, given, perhaps, to the simplicity of the movement.

468

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

TEST 4

3

log−lynx detrended

2

1

0

−1

−2

−3

True y PF−UKF estimate Bayes−FFNN estimate 95% prob. interval

−4

−5 1890

1895

1900

1905

1910

1915

1920

1925

1930

1935

Figure 9: Predicted values for the Lynx time series (log transformed and detrended).

For the airline time series, we also consider fitting both a classical ARIMA model, as presented by Box and Jenkins (1970) and an FFNN with three hidden nodes. Figure 11 presents prediction values (solid line) and 95% intervals (dash-dot line) for our LID-RJMCMC scheme, as well as comparison with the ARIMA(011)(011)12 model (dashed line) and FFNN(3) (dotted line), showing the superiority of neural networks in contrast to classical Table 1: Number of Times Proposed, and Times Accepted for the Different Moves. Move

Times Proposed

Times Accepted

j1a j1d j2a j2d j3a j3d j4a j4d

1 840 483 831 731 823 428 837

1 1 130 36 229 396 68 7

Bayesian Analysis of Nonlinear Autoregression Models

469

Histogram of M 0.35

0.30

Density

0.25

0.20

0.15

0.10

0.50

0.00

0

1

2

3

4

5

6

7

8

9

10

11

12

M Figure 10: Airline data. Histogram of the posterior distribution of M, showing M = 3 as the most likely number of nodes for the hidden layer of the FFNN term.

ARIMA models. Note that our results are very similar to those obtained from the classical FFNN with fixed number of nodes. Yet the main point we are trying to make is that with our algorithm, model selection is performed automatically, without the need to decide on a fixed architecture in advance. Table 2 compares the one-step-ahead mean square errors (MSE) obtained with each method. Again, our method leads to a slightly better performance over standard methods such as classical ARIMA and FFNN. 5.3 Simulated Examples. Finally, we apply our LID-RJMCMC scheme to simple linear and nonlinear simulated processes. Here, we are interested in seeing how our method performs for simple standard processes, deTable 2: One-Step-Ahead Mean Square Errors. Method

LID

ARIMA

FFNN(3)

MSE

0.0399

0.0455

0.0438

470

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

TEST 9

8

True y Bayes−NN estimate 95% prob. interval Arima FFNN(3)

7

air

6

5

4

3

2

120

122

124

126

128

130

132

Figure 11: Predicted values for the airline time series.

tecting them and not producing overly complex models. We generate two simple time series: a linear stationary AR(1) process and a nonlinear process generated by a FFNN with one node in the hidden layer. Table 3 shows the posterior distributions for indexes H (indicator of the linear term) and M (number of hidden nodes), after carrying out 20,000 MCMC iterations. Despite the complicated model, the scheme is able to identify the simpler standard autoregresive linear models (AR(1)) or neural networks (FFNN(1)), according to Occam’s principle of preferring simpler models.

Table 3: Posterior Distribution for Indexes H and M. H

AR(1) FFNN(1)

M

0

1

0

1

2

3

4

0.00 0.62

1.00 0.38

0.87 0

0.12 0.83

0.01 0.11

0.00 0.06

0.00 0.00

Bayesian Analysis of Nonlinear Autoregression Models

471

6 Conclusion We have presented a reversible-jump algorithm for the analysis of timeseries data through FFNNs viewed as nonlinear autoregression models. The advantages of the Bayesian approach outweigh its additional computational effort: as no local optimization algorithms are used, local minima issues are mitigated; model selection is performed as part of Bayesian methodology without the need of deciding explicitly a range of models to select from and having to use ad hoc methods to rank them. Bayesian analysis naturally embodies Occam’s razor—the principle of preferring simpler models to complex models. It is also possible to include a priori information about parameters and number of hidden nodes in accordance with this principle. The final product of a Bayesian analysis is a predictive distribution that averages over all possible models and parameter values, so uncertainty about predictions is estimated and overfitting risk is reduced. Many issues remain to be explored. An obvious extension is the application of the reversible-jump algorithm based on LID nodes to other standard neural network applications like regression, classification, or density estimation. Also, we have confined this analysis to FFNNs, but many other neural network models, analyzed from a Bayesian point of view, might prove useful. Relating to time-series analysis, here we have assumed autoregressive order q to be known in advance. We could also model uncertainty about q as a problem of model selection. Appendix A Linearized Nodes: Technical Details. Comments on section 4.1 emphasize the need to add and remove linearized nodes to get good coverage of the model space. When adding a linearized node, we will have to randomly
 generate weights γj0 , γj1 , . . . , γjq so that the node behaves in the linear zone for the given data set. To show
how to achieve this, consider the case in which q = 2. For a given node γ0 , γ1, γ2 , ζ = γ0 + γ1 x1 + γ2 x2 belongs to 2 a family of straight lines with slope −γ γ1 . This node is linearized if equation 4.1 is verified. Rewrite equation 4.1 as −ρ − γ0 ≤ z (xi1 , xi2 ) ≤ ρ − γ0 ∀ (xi1 , xi2 ) , i = 1, . . . , N,

(A.1)

where z (x1 , x2 ) = γ1 x1 + γ2 x2 = κ (s1 x1 + vs2 x2 ) with si = sgn(γi ), i = 1, 2 κ = |γ1 | v=

|γ2 | . |γ1 |

For some data points, z is maximized or minimized (see Figure 12), so

472

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

x2

zmax=g1*x1+g2*x2

x1 zmin=g1*x1+g2*x2

Figure 12:
To find zmin , we  and
zmaxmax  force −ρ − γ0 ≤ zmin ≤ zmax ≤ ρ + γ0 into a min rectangle xmin , xmax containing all input data points. 1 , x2 1 , x2

the above condition can be summarized as −ρ − γ0 ≤ zmin ≤ zmax ≤ ρ + γ0 , (A.2)
zmin zmin 
zmax zmax  where zmin = z x1 , x2 and zmax = z x1 , x2 . To find zmin and zmax , 



min , xmax , xmax it is easier to force equation A.2 into a rectangle xmin 1 , x2 1 2 containing all input data points: = min (xi1 ) , xmin 1

xmax = max (xi1 ) 1

xmin 2

xmax 2

i

= min (xi2 ) , i

i

= max (xi2 ) . i

We may then write,   1
1
1 + sgn(γi ) xmax 1 − sgn(γi ) xmin + , i = 1, 2 i i 2 2   1
1
= + , i = 1, 2, 1 + sgn(γi ) xmin 1 − sgn(γi ) xmax i i 2 2

xizmax = xizmin and


zmax = γ1 x1zmax + γ2 x2zmax = κ s1 x1zmax + vs2 x2zmax 
zmin = γ1 x1zmin + γ2 x2zmin = κ s1 x1zmin + vs2 x2zmin .

Bayesian Analysis of Nonlinear Autoregression Models

473

Let L = zmax − zmin = δ=

κ δ

1 . min ) + v(xmax − xmin ) (xmax − x 1 1 2 2

Then, rewrite equation A.2 as: L ≤ 2ρ −ρ − γ0 + L ≤ zmax ≤ ρ − γ0 .

(A.3)

We have three degrees of freedom to verify equation A.3. Setting v = γγ21 and signs si , i = 1, 2, fixes the slope of the straight line z, which fixes the sigmoidal surface orientation. Once this is defined, we could take a value for κ so that L ≤ 2ρ, fixing the sigmoidal surface steepness. Finally, we should take γ0 so that −ρ − γ0 + L ≤ zmax ≤ ρ − γ0 , which locates the sigmoidal surface. 1 Rewrite δ = xmaxu−x min for some u1 and 1

1

κ max (x − xmin 1 ) = 2ρb u1 1



− xmin − xmin xmax 1 1 − u1 xmax 1 1 1 1 v = max −1 = u1 x2 − xmin δ(xmax − xmin xmax − xmin 2 1 1 ) 2 2

L=

κ=

2ρbu1 − xmin 1

xmax 1

γ0 = −ρ + L − zmax + u0 (2ρ − L) . These new quantities may be interpreted as follows: • u1 is related to the orientation of the sigmoidal surface, and therefore 1 with the slope of the straight lines z. In fact, 1−u u1 should take values in [0, ∞). • b is related to the width L of the interval (zmin , zmax ) , which is related to the steepness of the sigmoidal surface. It should take values in (0, 1). • u0 is related to the sigmoidal surface location given by how interval (zmin , zmax ) is placed within interval (−ρ − γ0 , ρ − γ0 ). It should take values in (0, 1). Let us rewrite γ0 , γ1 , and γ2 in terms of the above quantities: γ1 = s1

2ρbu1 − xmin 1 )

(xmax 1

474

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

2ρb (1 − u1 ) (xmax − xmin 2 2 )
 γ0 = 2ρu0 1 − b + ρ(2b − 1) − (γ1 x1zmax + γ2 x2zmax ). 
Suppose now that we randomly generate weights γ0 , γ1, γ2 from a multivariate normal distribution, accepting those verifying equation A.3. The sample distribution of these weights has no recognizable shape, but the distributions of the other quantities above are suggested by their histograms: γ2 = s2

p(si = +1) = 0.5; b ∼ Beta(2, 2);

p(si = −1) = 0.5 u1 ∼ U(0, 1) ;

u0 ∼ U(0, 1).

We can then generate random weights corresponding to linearized nodes. The generalization to q > 2 is straightforward: γi =

2ρϕi , i = 1, . . . , q − xmin ) i

(xmax i

γ0 = 2ρu0 (1 − ϕ0 ) + ρ(2ϕ0 − 1) − (γ1 x1zmax + · · · + γq xqzmax ), q with ϕ0 = i=1 |ϕi | and xizmax defined as above. To generate ϕi , i = 1, . . . , q, first generate: q−1 
 ξ1 , . . . , ξq−1 ∼ Dirichlet(1, 1, . . . , 1); ξq = 1 − ξi i=1

b ∼ Beta(q, 2) p(si = +1) = p(si = −1) =

1 , i = 1, . . . , q 2

u0 ∼ U(0, 1), and define ϕi = bsi ξi . To propose a linearized node for deletion, we simply choose a node at random. Instead of finding out deterministically which node, if any, is a linearized one, we allow some randomness in the move definition. Once a node is selected, we recover the quantities defined previously: ξi = |ϕi | = u=

|γi | (xmax − xmin ) i i 2ρ

i = 1, . . . , q

γ0 + (γ1 x1zmax + · · · + γq xqzmax ) − ρ(2ϕ0 − 1) 2ρ(1 − ϕ0 )

b = ϕ0 =

q  i=1

|ϕi | .

Bayesian Analysis of Nonlinear Autoregression Models

475

Should the node selected be approximately linearized, the following conditions would verify: q 

ξi = 1 and ξi < 1

∀i = 1, . . . , q

i=1

u ∈ [0, 1] b ∈ [0, 1]. The density of any of these parameters (see appendix B) would be zero if they do not satisfy that condition. Hence, the probability of deleting a nonlinearized node with this move is zero; the move would be rejected. Appendix B: LID Reversible Jumps This appendix presents the mathematical details for the reversible-jump MCMC scheme for models containing both a linear term and an FFNN term, Linear + NN(m). Throughout this appendix, we use the following notation cur: current model pro: proposed model θ : parameters in current model φ: parameters in proposed model uθ : parameters to be added by the move uθ : parameters to be added by the move uφ : parameters to be deleted by the move According to the general reversible-jump MCMC scheme, proposed models are accepted with probability   p(y | φ) p(φ | pro) p(pro) q(uφ | pro) ppro−→cur · · · · ·J , α = min 1, p(y | θ) p(θ | cur) p(cur) q(uθ | cur) pcur−→pro where p(y|φ) p(y|θ) p(φ|pro) p(θ|cur) p(pro) p(cur) q(uφ |pro) q(uθ |cur)

is the likelihood ratio, is the prior ratio for parameters for given models, is the prior ratio for models, is the proposal ratio,

476

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨ pprocur pcurpro

J

is the transition probabilities ratio, and is the Jacobian of the transformation φ = g(θ, uθ ).

Note the following: • For simplicity, we shall assume all possible models to be equiprobable p(pro) so that p(cur) = 1 for all jumps considered. • Extension to models without the linear term, NN(m) is straightforward. • Likelihoods p(y | φ) and p(y | θ) may be substituted by marginal likelihoods in β. • Rearrangement of γ to satisfy order constraint should be done once the jump between models has been accepted, to save computational time. B.1 Add/Delete Linear Term. We propose this move with probability B1 . B.1a Add Linear Term. NN(m) −→ Linear + NN(m), m ≥ 1 
(θ, uθ ) −→ φ, uφ   θ = β1 , β2 , . . . , βM , γ1 , γ2 , . . . , γM , σ 2 , µβ , σβ2 , µγ , γ uθ = {λ, µλ , λ }   φ = β1 , β2 , . . . , βM , γ1 , γ2 , . . . , γM , σ 2 , µβ , σβ2 , µγ , γ , λ, µλ , λ . The Jacobian of the one-to-one transformation φ = g(θ, uθ ) is J = 1. The proposal is given by µλ ∼ N(mλ , Sλ ) λ ∼ IWishart(wλ , λ ) λ ∼ N(µλ , λ ). Thus, p(φ | pro) = p(λ | µλ , λ )p(µλ | mλ , Sλ )p(λ | wλ , λ ) p(θ | cur)

Bayesian Analysis of Nonlinear Autoregression Models

ppro−→cur B1d = =1 pcur−→pro B1a

assuming B1d = B1a =

477

B1 2

q(uφ | pro) 1 = , q(uθ | cur) N(λ | µλ , λ )N(µλ | mλ , Sλ )IW(λ | wλ , λ )   p(y|φ) so that the acceptance probability is given by α = min 1, p(y|θ ) . B.1b Delete Linear Term . Linear + NN(m) −→ NN(m), m ≥ 1 
(θ, uθ ) −→ φ, uφ   θ = β1 , β2 , . . . , βM , γ1 , γ2 , . . . , γM , σ 2 , µβ , σβ2 , µγ , γ , λ, µλ , λ   φ = β1 , β2 , . . . , βM , γ1 , γ2 , . . . , γM , σ 2 , µβ , σβ2 , µγ , γ uφ = {λ, µλ , λ } . The Jacobian of the one-to-one transformation (φ, uφ ) = g(θ) is J = 1 and 1 p(φ | pro) = p(θ | cur) p(λ | µλ , λ )p(µλ | mλ , Sλ )p(λ | wλ , λ ) ppro→curr B1a = =1 pcur−→pro B1d

assuming B1d = B1a =

B1 2

q(uφ | pro) = N(λ | µλ , λ )N(µλ | mλ , Sλ )IW(λ | wλ , λ ), q(uθ | cur)   p(y|φ) so that the acceptance probability is given by α = min 1, p(y|θ ) . B.2 Add/Delete Linearized Node. We propose this move with probability B2 . B.2a: Add Linearized Node. Linear + NN(m) −→ Linear + NN(m + 1), m ≥ 1 
(θ, uθ ) −→ φ, uφ   θ = β1 , β2 , . . . , βM , γ1 , γ2 , . . . , γM , σ 2 , µβ , σβ2 , µγ , γ

478

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

  uθ = , ϕ1 , ϕ2 , . . . , ϕp , u   2 , . . . , β M , β M+1 ,  γ . 1 , β γ1 ,  γ2 , . . . ,  γM ,  γM+1 ,  σ 2,  µβ ,  σβ2 ,  µγ ,  φ= β The transformation φ = g(θ, uθ ) is defined by j = βj , β M+1 = , β

γj = γj  γM+1,i = 

j = 1, . . . , M 2ρϕi − xmin ) i

i = 1, . . . , p

(xmax i

γM+1,0 = 2ρu(1 − ϕ0 ) + ρ(2ϕ0 − 1) − (  γM+1,1 x1zmax + · · · +  γM+1,p xpzmax )  σ 2 = σ 2, ϕ0 =

p 

 µβ = µβ ,

 σβ2 = σβ2 ,

γ = γ  µγ = µγ , 

|ϕi |

i=1

xizmax =

  1
1
+ , i = 1, . . . , p. 1 + sgn(ϕi ) xmax 1 − sgn(ϕi ) xmin i i 2 2

The Jacobian of the transformation is   2ρ(1 − ϕ0 ) 0 ...   2ρ  − ... (xmax −xmin )  1 1  − 0 ... J =   − 0 ...    − 0 ... 

0

0

0

0

0

0

2ρ max min (xp−1 −xp−1 )

0

(2ρ)p (1 − ϕ0 ) = p . max − xmin ) i i=1 (xi The proposal is given by u ∼ U(0, 1)  ∼ N(µβ , σβ2 ) b ∼ Beta(p, 2) 
ξ1 , . . . , ξp−1 ∼ Dirichlet(1, 1, . . . , 1) p(si = +1) = p(si = −1) = Then define ϕi = si bξi , i = 1, . . . , p.

1 , i = 1, . . . , p. 2

0 2ρ (xpmax −xpmin )

             

Bayesian Analysis of Nonlinear Autoregression Models

479

and it can be shown that   p  1 fϕ (ϕ1 , . . . , ϕp ) = p (p + 2) 1 − ϕi , 2 i=1  |ϕk | f or0 < |ϕi | < 1 −

i = 1, . . . , p.

k=i

Thus, 
p(φ | pro) γ (M + 1)! µβ ,  σβ2 )p γM+1 |  µγ ,  = p(βM+1 |  p(θ | cur) M! ppro→curr B21d = pcur−→pro B21a q(uφ | pro) = q(uθ | cur)

1 M+1

1 2p (p

1

=

1 M+1



+ 2) 1 −

assuming B2d = B2a =

B2 2

1 p

 . 2) ϕ , σ N( | µ i β β i=1

B.2b Delete Linearized Node. Linear + NN(m + 1) −→ Linear + NN(m), m ≥ 1
 (θ, uθ ) −→ φ, uφ   θ = β1 , β2 , . . . , βM , βM+1 , γ1 , γ2 , . . . , γM , γM+1 , σ 2 , µβ , σβ2 , µγ , γ   1 , β 2 , . . . , β M ,  γ φ= β γ1 ,  γ2 , . . . ,  γM ,  σ 2,  µβ ,  σβ2 ,  µγ ,    uφ = , ϕ1 , ϕ2 , . . . , ϕp , u . Select node h = 1, . . . , M + 1 at random, take nodes in θ : 1, . . . , h − 1, h + 1, . . . , M + 1, and relabel
as 1, . . . , M. The transformation φ, uφ = g(θ) is defined by j = βj ,  β γj = γj

j = 1, . . . , M

γ = γ  σ 2 = σ 2,  µβ = µβ ,  σβ2 = σβ2 ,  µγ = µγ ,   = βh ϕi =

γh,i (xmax − xmin ) i i 2ρ

u=

γh,0 + (γh,1 x1zmax + · · · + γh,p xpzmax ) − ρ(2ϕ0 − 1) 2ρ(1 − ϕ0 )

i = 1, . . . , p

480

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

ϕ0 =

p 

|ϕi | ,

i=1

xizmax =

  1
1
+ , i = 1, . . . , p. 1 + sgn(ϕi ) xmax 1 − sgn(ϕi ) xmin i i 2 2

The Jacobian of the transformation is  1  0 ...  2ρ(1−ϕ0 )  max min (x1 −x1 )  ... −  2ρ   − 0 ... J=   − 0 ...     − 0 ...

0

0

0

0

0

0

max min (xp−1 −xp−1 ) 2ρ

0

0

(xpmax −xpmin ) 2ρ

              

p =

max − xmin ) i i=1 (xi (2ρ)p (1 − ϕ0 ).

If γh is not a linearized node, then the acceptance probability is α = 0; otherwise, 1 p(φ | pro) M!
 = p(θ | cur) p(βh | µβ , σβ2 )p γh | µγ , γ (M + 1)! p(prop) =1 p(curr) ppro→curr

assuming Kequiprobable models p(i) =

1 , ∀i K

B21a 1 B21 = M+1 assuming B21d = B21a = 1 pcur−→pro B21d M+1 2   p   p   q(uφ | pro) 1 I[0,1− |ϕk |] (|ϕi |) p (p + 2) 1 − ϕi = k=i q(uθ | cur) 2 i=1 i=1 =

I[0,1] (u)N( | µβ , σβ2 ). B.3 Add/Delete Irrelevant Node. We propose this move with probability B3 . B.3a Add Irrelevant Node . Linear + NN(m) −→ Linear + NN(m + 1), m ≥ 1 
(θ, uθ ) −→ φ, uφ   θ = β1 , β2 , . . . , βM , γ1 , γ2 , . . . , γM , σ 2 , µβ , σβ2 , µγ , γ

Bayesian Analysis of Nonlinear Autoregression Models

481

uθ = {βM+1 , γM+1 }   φ = β1 , β2 , . . . , βM , βM+1 , γ1 , γ2 , . . . , γM , γM+1 , σ 2 , µβ , σβ2 , µγ , γ . The proposal is given by γM+1 ∼ N(µγ , γ ) βM+1 ∼ N(µβ , σβ2 ), rearranging γ to satisfy order constraint γ1,p < . . . <
γM+1,p  . The Jacobian of the one-to-one transformation φ, uφ = g(θ) is J = 1. Also, 
2 p(φ | pro) p(βM+1 | µβ , σβ )p γM+1 | µγ , γ (M + 1)! = p(θ | cur) 1 M! p(prop) =1 p(curr)

assuming Kequiprobable modelsp(i) =

ppro→curr B3d ψM+1 ψM+1 = = M+1 M+1 pcur−→pro B3a k=1 ψk k=1 ψk

1 , ∀i K

assuming B3d = B3a =

B3 2

q(uφ | pro) 1 ,
= q(uθ | cur) p(βM+1 | µβ , σβ2 )p γM+1 | µγ , γ   p(y|φ) M+1 M+1 . so that the acceptance probability is given by α = min 1, p(y|θ ) (M+1)ψ k=1

ψk

B.3b Delete Irrelevant Node. Linear + NN(m + 1) −→ Linear + NN(m), m ≥ 0
 (θ, uθ ) −→ φ, uφ   θ = β1 , β2 , . . . , βM , βM+1 , γ1 , γ2 , . . . , γM , γM+1 , σ 2 , µβ , σβ2 , µγ , γ   φ = β1 , β2 , . . . , βM , γ1 , γ2 , . . . , γM , σ 2 , µβ , σβ2 , µγ , γ uφ = {, δ} . Let ψj =

√1  2π

exp

 −1 2 2

x2



j = 1, . . . , M + 1 for a given . Choose node ψ

h at random with probabilities pj = M+1j h, and relabel as 1, . . . , M.

k=1

ψk

, j = 1, . . . , M + 1. Remove node

482

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨


 The transformation φ, uφ = g(θ) is defined by  = βh δ = γh , with Jacobian J = 1 and p(φ | pro) M! 1 
= 2 p(θ | cur) p(βh | µβ , σβ )p γh | µγ , γ (M + 1)! p(prop) =1 p(curr)

assuming K equiprobable models p(i) = M+1

ppro→curr

B3a 1 = ψh pcur−→pro B3d M+1 k=1

= ψk

k=1

ψk

ψh

1 , ∀i K

assuming B3d = B3a =

B3 2


p( | µβ , σβ2 )p δ | µγ , γ q(uφ | pro) = , q(uθ | cur) 1  M+1  ψk p(y|φ) k=1 so that the acceptance probability is given by α = min 1, p(y|θ ) (M+1)ψ . h B.4 Add/Delete Duplicate Node. We propose this move with probability B4 . B.4a Add Duplicate Node. Linear + NN(m) −→ Linear + NN(m + 1), m ≥ 1 
(θ, uθ ) −→ φ, uφ   θ = β1 , β2 , . . . , βM , γ1 , γ2 , . . . , γM , σ 2 , µβ , σβ2 , µγ , γ uθ = {ν, δ}  h , β h+1 , . . . , βM , φ = β1 , β2 , . . . , β

 γ1 , γ2 , . . . ,  γh ,  γh+1 , . . . , γM , σ 2 , µβ , σβ2 , µγ , γ .


h+1 ,  h ,  γh and (β γh+1 ) where Choose h = 1, . . . , M at random and propose β h = βh (1 − ν) β γh = γh 

Bayesian Analysis of Nonlinear Autoregression Models

483

h+1 = βh ν β γh+1 = γh + δ.  Reject if (γ1 , γ2 , . . . , γh ,  γh+1 , . . . , γM ) is not ordered. The Jacobian of the transformation φ = g(θ, uθ ) is defined by    0 1 0 1   
   ∂ β h+1 ,  γh , β γh+1   (1 − ν) 0 ν 0   h ,   = |βh | . J=  =    −βh  ∂ (γh , βh , ν, δ) 0 βh 0     0 0 0 1  The proposal is given by ν ∼ Beta(2, 2) δ ∼ N(0, cSγ ). Thus,

  h+1 | µβ , σ 2 )p  h | µβ , σ 2 )p  γh | µγ , γ p(β γh+1 | µγ , γ p(φ | pro) p(β β β 
= p(θ | cur) p(βh | µβ , σβ2 )p γh | µγ , γ (M + 1)−1 1 ppro→curr B4d M 1 = = pcur−→pro B4d 1 M

assuming B4d = B4a =

B4 2

q(uφ | pro) 1 = . q(uθ | cur) Beta(ν | 2, 2)N(δ | 0, cSγ ) B.4d Delete Duplicate Node. Linear + NN(m + 1) −→ Linear + NN(m), m ≥ 1 
(θ, uθ ) −→ φ, uφ   θ = β1 , β2 , . . . , βM , βM+1 , γ1 , γ2 , . . . , γM , γM+1 , σ 2 , µβ , σβ2 , µγ , γ   h , . . . , βM , γ1 , γ2 , . . . ,  φ = β1 , β2 , . . . , β γh , . . . , γM , σ 2 , µβ , σβ2 , µγ , γ uφ = {ν, δ} .

 h ,  Select h = 1, . . . , M at random. Remove βh+1 , γh+1 and propose β γh , where h = βh + βh+1 β

484

A. Menchero, R. Montes Diez, D. R´ıos Insua, and P. Muller ¨

γh = γh  ν=

βh+1 βh + βh+1

δ = γh+1 − γh . Then relabel as 1, . . . , M.
 The Jacobian of the transformation φ, uφ = g(θ) is      
    γh , ν, δ ∂ βh ,      =  J=
 ∂ γh , γh+1 , βh , βh+1    

0 0 1

1 0 0

1

0

0 0

−βh+1 (βh +βh+1 )2 βh (βh +βh+1 )2

        1    =    βh + βh+1   0 

−1 1 0

and
 h | µβ , σ 2 )p  γh | µγ , γ (M + 1)−1 p(β p(φ | pro) β  

= p(θ | cur) p(βh | µβ , σβ2 )p γh | µγ , γ p(βh+1 | µβ , σβ2 )p γh+1 | µγ , γ ppro→curr B4a 1 = =M 1 pcur−→pro B4d M

assuming B4d = B4a =

B4 2

q(uφ | pro) = I(0,1) (ν)Beta(ν | 2, 2)N(δ | 0, cSγ ). q(uθ | cur) Acknowledgments This work has been supported by projects from CICYT, CAM, and URJC. R.M.D. gratefully acknowledges receipt of a postdoctoral fellowship from CAM (Comunidad Autonoma ´ de Madrid). We also thank the referees for their illuminating comments. References Andrieu, C., de Freitas, J. F. G., & Doucet, A. (1999). Robust full Bayesian learning for neural networks (Tech. Rep. CUED/FINFENG/TR 343). Cambridge: Cambridge University, Engineering Department. Box, G. E. P., & Jenkins, G. M. (1970). Time series analysis: Forecasting and control. San Francisco: Holden-Day. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14, 179–211. Faraway, J., & Chatfield, C. (1998). Time series forecasting with neural networks: A comparative study using the airline data. Journal of the Royal Statistical Society, Applied Statistics (Series C), 47, 231–250. Foster, B., Collopy, F., & Ungar, L.H. (1992). Neural network forecasting of short noisy time series. Computers and Chem. Engr., 16(4), 293–298.

Bayesian Analysis of Nonlinear Autoregression Models

485

Gamerman, D. (1997). Markov chain Monte Carlo: Stochastic simulation for Bayesian inference. London: Chapman & Hall. Geman, S., & Geman, D. (1984). Stochastic relaxation, Gibbs distribution and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6(6), 721–741. Green, P. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika, 82, 711–732. Hill, T., O’Connor, M., & Remus, W. (1996). Neural network models for time series forecasts, Management Sci., 42, 1082–1092. Holmes, C. C., & Mallick, B. K. (1998). Bayesian radial basis functions of variable dimension. Neural Computation, 10, 1217–1233. Jordan, M. I. (1986). Attractor dynamics and parallelism in a connectionist sequential machine. In Proceedings of the Eighth Conference of the Cognitive Science Society. Mackay, D. J. C. (1992). Bayesian methods for adaptive models. Unpublished doctoral dissertation, California Institute of Technology. Muller, ¨ P., & R´ıos Insua, D. (1998). Issues in Bayesian analysis of neural network models. Neural Computation, 10, 571–592. Neal, R. M. (1996). Bayesian learning for neural networks. New York: SpringerVerlag. Priestley, M. B. (1988). Non-linear and non-stationary time series analysis. London: Academic Press. Remus, W., O’Connor, M., & K. Griggs, (1998). The impact of information of unknown correctness on the judgmental forecasting process. International Journal of Forecasting, 14, 313–322. Richardson, S., & Green, P. J. (1997). On Bayesian analysis of mixtures with an unknown number of components. Journal of the Royal Statistical Society B, 59, 731–792. R´ıos Insua, D, & Muller, ¨ P. (1998). Feedforward neural networks for nonparametric regression. In D. Dey, P. Muller, ¨ & D. Sinha (Eds.), Practical nonparametric and semiparametric bayesian statistics. Berlin: Springer. Stern, H. (1996). Neural networks in applied statistics (with discussion). Technometrics, 38, 205–220 Tang, Z., de Almeida, C., & Fishwick, P. A. (1991). Time series forecasting using neural networks versus Box-Jenkins methodology. Simulation, 57, 303–310 van der Merwe, R., Doucet, A. de Freitas, N., & Wan, E. (2000). The unscented particle filter (Tech. Rep. CUED/F-INFENG/TR 380). Cambridge: Cambridge University Department of Engineering. Williams, R. J., & Zipser, D. (1989). A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1, 270–280. Received January 7, 2004; accepted July 1, 2004.

ˇ ıma Communicated by Jiri S´

LETTER

Loading Deep Networks Is Hard: The Pyramidal Case David Windisch [email protected] Bahnplatz 5, A-2371 Hinterbruehl, Austria

The question of whether it is possible to load deep neural network architectures efficiently is examined by considering the class of pyramidal architectures. This class allows only a low interaction of the nodes. Still, the loading problem is found to be NP-complete. This provides evidence that depth alone is a factor accounting for loading hardness.

1 Introduction The loading problem for neural networks was first considered by Judd (1990). While he proved the loading problem to be NP-complete for arbitrary networks by reduction of the NP-complete 3 satisfiability problem (see Garey & Johnson, 1979), he found a polynomial-time loading algorithm for a restricted class of shallow architectures. Judd also proved NP-hardness of the loading problem for several classes of shallow architectures that do not satisfy his restriction but left the loading problem for deep (i.e., nonshallow) architectures unaddressed . This raises the question whether deep architectures can possibly be loaded in polynomial time. Deep architecˇ ıma (1994), who proved NP-completeness for tures were considered by S´ the class of triangular architectures, and by Hammer (1998a), who proved NP-hardness for certain classes of multilayer perceptrons. Finally, de Souto and de Oliveira (1999) introduced the restricted class of deep pyramidal architectures. These are similar to triangular architectures but allow only a minimal interaction between the nodes. de Souto and de Oliveira also proved NP-completeness of the loading problem for pyramidal architectures, but they overlooked that their proof holds only for a fan-in strictly greater than 2. Their result remains true for fan-in equal to 2 but requires a modified proof, which is given below. Also, NP-completeness is extended to another class of pyramidal architectures, where the number of hidden layers is fixed, and the fan-in varies instead, and to the node function set of linearly separable functions. This result is of interest because it shows that not even for the restricted class of pyramidal architectures, in which only minimal interaction of the nodes is possible, can loading be achieved in polynomial time, provided P
= NP. This leads to the conclusion that depth by itself makes loading hard. Neural Computation 17, 487–502 (2005)

c 2005 Massachusetts Institute of Technology 

488

D. Windisch

q

q

q

q

d q

Figure 1: A pyramidal architecture. The input nodes (empty circles) do not count as a layer of the architecture.

2 Terminology Definition 1. (pyramidal architecture) A pyramidal architecture π(d, q) is a feed forward architecture whose nodes form a q-tree (q ≥ 2) with d + 1 layers, where all the nodes have the same fan-in q. The leaves (in layer 0) are the input nodes feeding the nodes in layer 1. The root of the tree (in layer d) is the output node. Thus, every node receives input from its q children and propagates its output to its parent. See Figure 1 for a schematic illustration and Figure 2 for an illustration of the case d = q = 3. Definition 2. (node function sets used) The functions computed by the nodes can be chosen from either one of the following two sets: 1. AOFns = {AND, OR}, 2. LSFns = {[w, . ≥ c] : w ∈ Rq , c ∈ R}, where ., . denotes the inner product on Rq and
1 if w, y ≥ c [w, y ≥ c] = 0 otherwise By v, we denote a node of an architecture as well as the function computed by it. Also, we do not distinguish between a configured architecture and the function computed by it. This does not cause any confusion throughout this letter.

Loading Deep Networks Is Hard

489

Definition 3. (configuration, item, task, performability) Let A be a class of architectures and F a set of node functions. We call a configuration of A ∈ A a mapping from the set of nodes of A to F . By a task T = {Ii }, we mean a set of correct input-output combinations Ii = (xi , αi ), called items. If the architecture A is equipped with a configuration (i.e., if A has node functions attached to its nodes), then we say that A performs T if, for any item I = (x, α) ∈ T, the configured architecture A, when given the input x specified in I, returns the output α specified in I. Definition 4. (loading problem—decision version) Let A be a class of architectures and F a set of node functions. We define the loading problem as the following decision problem: Given an architecture A ∈ A and a task T, is there a configuration of A such that A performs T? Formally, the loading problem is the problem of recognizing the following language: PerfF = {(A, T) : F allows a configuration of A such that A performs T},

(2.1)

where A denotes a network architecture in A, T a task. A pyramidal architecture (see definition 1, Figure 1) depends on two different parameters: depth d and fan-in q. In order to consider the hardness of the loading problem, we must let one of these parameters vary. We first examine the problem for variable d and fixed q. 3 Variable Depth, Fixed Fan-In We now state the theorem de Souto and de Oliveira (1999) attempted to prove and provide a modified proof that also holds for the case q = 2. Then, we explain where their proof fails. Theorem 1. PerfAOFns is NP-complete for the class of pyramidal architectures with fixed q ≥ 2 and variable d ≥ 3. Proof. The proof is accomplished by reduction of the NP-complete satisfiability problem (SAT). For an explanation of this technique, see Garey and Johnson (1979). By vi , i = 1, 2, . . . , qd−2 , we denote the nodes in layer 2, and by vik , k = 1, 2, . . . , q, the corresponding nodes of layer 1 (i.e., the input nodes of the node vi are the nodes {vik , k = 1, 2, . . . , q}; see Figure 2). As indicated previously, the variables for the nodes also denote the functions computed by them.

490

D. Windisch

v11

v12

v13

v1

v21

v22

v23

v31

v2

v32

v33

v3

v

Figure 2: A pyramidal architecture, where d = 3, q = 3. The empty circles are input nodes.

The SAT (satisfiability) problem can be defined as follows: Given a Boolean expression in conjunctive normal form, is there a truth assignment to its variables, such that the expression is true? Formally, the SAT problem is the problem of recognizing the following language: SAT = {(Z, C) : ∃ such that (C) is true}, where Z = {ζ1 , ζ2 , . . . , ζr } is a set of boolean variables, C a boolean expression in terms of conjunctions of disjunctions of literals, and  : Z → {0, 1} is some truth assignment to the variables in Z. We let (Z, C) be an arbitrary instance of the SAT problem, where C = {c1 , c2 , . . . , ce } is a conjunction of disjunctions over the set Z. For an arbitrarily chosen disjunction cj of C, we use the following notation: cj = {ζ˜i1 , ζ˜i2 , . . . , ζ˜im },

(3.1)

where 1 ≤ i1 < i2 < . . . < im ≤ r, and ζ˜ip is the literal derived from the variable ζip (i.e., ζ˜ip = ζip or ζ˜ip = ¬ζip ). 3.1 Reduction of SAT. We now reduce this instance (Z, C) of SAT to the instance (π, T) of the loading problem. To define the architecture π = π(d, q), we let every node vi in layer 2 represent one of the variables ζi . Hence, we need at least r nodes in layer 2. To meet this criterion, we choose d = 2 + logq r.

(3.2)

Note that the input dimension is then qd . For any disjunction cj in C, we define a corresponding item Ij in the task, T = {I0 , I1 , . . . , Ie , J1 , J2 , . . . , Jqd−2 , K1 , K2 , . . . , Kqd−2 },

Loading Deep Networks Is Hard

491

as follows (and give the justification below): Ij = (([ζ˜1 ]1q

2

−2q

), ([ζ˜2 ]1q

2

−2q

). . .([ζ˜r ]1q

2

−2q

)0q

d

−rq2

, 1), j = 1, 2,. . ., e, (3.3)

where [ζ˜i ] denotes an encoding of the literal ζ˜i , and is defined as follows:  (11)1q−2 (10)1q−2 if ζ˜i = ζi ˜ (3.4) [ζi ] = (10)1q−2 (11)1q−2 if ζ˜i = ¬ζi  (10)1q−2 (10)1q−2 if ζ˜i is not present in cj , where the number of 1’s is chosen such that half of the encoding [ζ˜i ] (i.e., (..)1q−2 ) constitutes the whole input to the node vi1 and the other half the 2 input to vi2 . In every bracket ([ζ˜i ]1q −2q ) in equation 3.3, the number of 2 1’s is chosen such that the bracket ([ζ˜i ]1q −2q ) constitutes the whole input affecting the layer 2 node vi . We define the item I0 (corresponding to the empty disjunction) in the same way but with required output 0: I0 = ([(10)1q−2 (10)1q−2 1q

2

−2q r qd −rq2

]0

, 0).

(3.5)

The following items Ji , i = 1, 2, . . . , qd−2 , force all nodes in layers ≥ 3 to compute OR. This is because only the ith layer 2 node will output 1 from the input of item Ji , so all nodes along the path from vi to the output node (below vi ) must compute OR: Ji = (0(i−1)q 1q 00 . . . 0, 1), 2

2

i = 1, 2, . . . , qd−2 .

(3.6)

And finally, we construct the following items Ki , i = 1, 2, . . . , qd−2 , such that Ki generates the input (100 . . . 0) to layer 2 node vi , and thus forces vi to compute AND: Ki = (0(i−1)q 1q 00 . . . 0, 0), 2

i = 1, 2, . . . , qd−2 .

(3.7)

The construction can be performed in polynomial time with respect to the size of (Z, C). To show this, we note that the length of any item I is qd + 1 = O(r) (by the definition of d, see equation 3.2), the number of items is equal to 1 + e + 2qd−2 = O(e + r), the number of nodes of A is qd+2 −1

1 + q + . . . + qd+1 = q−1 = O(qd ) = O(r), and the number of edges of A is one less than the number of nodes of A. We now give an illustrative example of the above construction for the case q = 3. Let (Z, C) be the following instance of SAT: Z = {ζ1 , ζ2 } C = {c1 , c2 } c1 = {ζ1 } c2 = {¬ζ1 , ¬ζ2 }.

492

D. Windisch

Following the above construction of a corresponding instance of PerfAOFns , we choose d = 2 + log3 2 = 3. This gives the architecture π(3, 3) depicted in Figure 2. The items in the task T are the following: I1 = (111101111

101101111

000000000, 1),

I2 = (101111111

101111111

000000000, 1),

I0 = (101101111

101101111

000000000, 0),

J1 = (111111111

000000000 000000000, 1),

J2 = (000000000 111111111

000000000, 1),

J3 = (000000000 000000000 111111111, 1), K1 = (111000000 000000000 000000000, 0), K2 = (000000000 111000000 000000000, 0), K3 = (000000000 000000000 111000000, 0). Proof. To prove that (Z, C) ∈ SAT ⇔ (π, T) ∈ PerfAOFns , we first assume that (Z, C) ∈ SAT and show that this implies that (π, T) ∈ PerfAOFns . The assumption means that there is a truth assignment  : Z → {0, 1}, such that (C) is true. We make the following choices for the node functions:


AND OR
vi1 , vi2 = vi1 =

if (ζi ) = 1 if (ζi ) = 0,

vik = AND ∀(i, k) : (i ≤ r, k ≥ 3) or i > r vi = AND ∀i (this is layer 2),

(3.8)

and all other nodes compute OR. First, we consider the item I0 (see equation 3.5). For any i, both nodes vi1 and vi2 receive at least one 0 as input. But either vi1 or vi2 computes the AND function, so vi receives one 0 (either from vi1 or from vi2 , at least) as its input. Since vi computes AND, it follows that vi outputs 0. Hence, all the layer 2 nodes output 0, so the final output is 0, as required. Now we consider all other items Ij , j = 1, 2, . . . , e, in the task. Each of these requires output 1. Since all nodes in layers ≥ 3 compute OR, it suffices to check that there is at least one node in layer 2 that outputs 1 in each case. Here, we will use the established correspondence with the disjunctions in C. We choose an arbitrary item Ij = (xj , αj ), say, and we denote the corresponding disjunction again by cj = {ζ˜i1 , ζ˜i2 , . . . , ζ˜im } (see equation 3.1). Since we are assuming the boolean expression C to be satisfied by the assignment , it would suffice to show that vl (xj ) = (ζ˜l ) ∀l ∈ {i1 , i2 , . . . , im }.

Loading Deep Networks Is Hard

493

To show that this is indeed true, we need only to check the four possible cases: (1) If (ζl ) = 1, then the node functions were chosen such that vl1 = AND, and vl2 = OR (see equation 3.8): • If ζ˜l = ζl , then we see from the definition of Ij (see equations 3.3 and 3.4) that the only node of all vlk , k = 1, 2, . . . , q, receiving a 0 as input is node vl2 . But vl2 = OR, so all inputs to node vl are 1’s, and vl (xj ) = 1, as required. • If ζ˜l = ¬ζl , then we see from the definition of Ij (see equations 3.3 and 3.4) that only vl1 receives a 0 as input. Since vl1 = AND, this implies that vl receives a 0 as input, and hence vl (xj ) = 0, as required. (2) If (ζl ) = 0, then the node functions were chosen such that vl1 = OR, and vl2 = AND (see equation 3.8). The same considerations as in case 1 show that vl (xj ) = (ζ˜l ) in this case, too. Hence, π maps all items Ij , j = 0, 1, . . . , e, correctly. If we consider the input of any item Ji , i = 1, 2, . . . , qd−2 (see equation 3.6), we see that it generates input 1q to the layer 2 node vi , so that vi must output 1. Since all nodes below layer 2 compute OR, this implies that the output of the architecture is 1, as required. Finally, the input of an item Ki , i = 1, 2, . . . , qd−2 (see equation 3.7), generates input 0q to all layer 2 nodes, except vi , whose input is 10q−1 . Since vi computes AND, vi outputs 0, like all other layer 2 nodes. Therefore, the output is 0, as required. We therefore conclude that we indeed have (π, T) ∈ PerfAOFns . Now, we assume that (π, T) ∈ PerfAOFns and show that this implies that (Z, C) ∈ SAT. To this end, we define the truth assignment  by
1 if vi1 = AND (ζi ) = 0 otherwise, qd−2

and show that (C) is true. First, we note that the set of items {Ji }i=1 requires all nodes in layers ≥ 3 to compute OR (see equation 3.6), and that qd−2

therefore the items {Ki }i=1 require all layer 2 nodes vi to compute AND (see equation 3.7). We need to choose any disjunction cj (j ∈ {1, 2, . . . , e}) and show that at least one literal in it is assigned the value 1. To do that, we consider the corresponding item Ij , and compare it with I0 . We note that the outputs required by these two items differ. This difference can be accounted for only by some node functions at the nodes where the inputs of these two items differ. Comparing equations 3.3 and 3.4 with equation 3.5, we see that these nodes must be some nodes vi1 or vi2 , say, where i ≤ r. Remembering that the corresponding node vi computes AND, we see that this implies that exactly one of the nodes vi1 , vi2 must compute the AND function, for

494

D. Windisch

in the other two cases, the output of vi would remain the same for inputs from I0 and Ij . Equation 3.4 shows that if vi1 = AND (which implies that (ζi ) = 1), then ζ˜i = ζi , and if vi1
= AND (which implies that (ζi ) = 0), then (vi2 = AND and) ζ˜i = ¬ζi . Hence, we have in both cases (ζ˜i ) = 1, as required, so (C) is true and (Z, C) ∈ SAT. This completes the proof that (Z, C) ∈ SAT ⇔ (π, T) ∈ PerfAOFns . To show that PerfAOFns is in NP, we note that an algorithm checking whether a given configuration of an architecture A performs T has to evaluate all node functions once for every item in T, and hence runs in polynomial time. This completes the proof. It should be noted that the construction of (A, T) in the last proof could be achieved with fewer items Ji , since a Ji for every layer 3 node would suffice. This, however, would complicate the notation unnecessarily. This last proof is very similar to the one used by de Souto and de Oliveira (1999), except for two modifications. The first modification is that instead of the modified satisfiability problem (MSAT), proved to be NP-complete ˇ ıma (1994), we use the classical version SAT. The difference between by S´ these two problems is that MSAT requires positive and negative literals to alternate in every disjunction. In fact, even the proof by de Souto and de Oliveira (1999) can be left unchanged if MSAT is replaced by SAT. The crucial modification, however, is the extension of the task T (of the constructed instance of the loading problem) by the additional items Ji and Ki . These items force all nodes in layers ≥ 3 to compute OR and all nodes in layer 2 to compute AND. This property of a yes-instance of PerfAOFns is needed to infer that exactly one of two layer 1 nodes vi1 , vi2 must compute AND in order to perform the task T. But this last fact is essential for the final argument that given a performing configuration for π, the truth assignment  satisfies the boolean expression C. If the fan-in q is strictly greater than 2 then even without items Ji and Ki , an argument similar to the above allows concluding that exactly one of two layer 1 nodes vi1 , vi2 must compute AND in order to perform the task, because the corresponding node vi must compute AND to distinguish between a pair of items I0 and Ij . However, this last argument breaks down if q = 2. To illustrate this point, we let q = 2, and provide an example of a no instance of SAT (as well as MSAT), which is mapped to a yes instance of PerfAOFns if the items Ji and Ki are omitted. Let Z = {ζ1 , ζ2 }, C = {c1 , c2 , c3 }, c1 = {¬ζ1 , ζ2 }, c2 = {¬ζ2 }, c3 = {ζ1 }. Obviously, C is then not satisfiable. However, using the reduction constructed in the proof, this instance gets mapped to π(3, 2) (i.e., a pyramidal architecture with three layers) with task T = {I0 , I1 , I2 , I3 }, I0 = (10101010, 0), I1 = (10111110, 1), I2 = (10101011, 1), I3 = (11101010, 1). This task is performed by π(3, 2) with node functions (v denotes the output node, otherwise notation as in the above proof) v11 = AND, v12 = OR, v21 = AND, v22 = AND, v1 = AND, v2 = OR, v = OR.

Loading Deep Networks Is Hard

495

q−1

q

Figure 3: The class of architectures described in theorem 2.

It remains an open question whether the same hardness result as the one just proved also holds for the node function set LSFns. A generalization of the method used in the last proof to the LSFns case does not seem obvious. 4 Fixed Depth, Variable Fan-In We now consider the set of pyramidal architectures with fixed d and variable q. We note that these architectures still qualify as deep networks because the support cone configuration space is unbounded. (The support cone configuration space of an output node is the set of all possible configurations of the nodes affecting it. See Judd, 1990.) We consider the node function sets LSFns and AOFns separately. 4.1 LSFns. We first restrict the set under consideration to the set of pyramidal architectures with fixed d = 2. From Blum and Rivest (1992), we know that the loading problem is NP-complete for a similar class of architectures: Theorem 2. PerfLSFns is NP-hard for the class of architectures mapping {0, 1}q−1 to {0, 1}, with q nodes in one hidden layer, each connected to all input nodes and computing LSFns, and one output node computing AND. (See Figure 3.)  . (In fact, the proof We denote the problem in the last theorem by Perf given by Blum & Rivest, 1992, covers even the more general case, where the number of hidden nodes is bounded by a polynomial in q.) We will use theorem 2 to prove the following lemma: Lemma 1. PerfLSFns is NP-hard for the class of pyramidal architectures with fixed d = 2 and variable q ≥ 2.

496

D. Windisch

x1 x2

v^1

v^2

x1 x 2 0 x 1 x 2 0 x 1 x 2 0

v^3

^

v

v1

v2

v3

v

 (left) is reduced Figure 4: Illustration of the proof of lemma 1. An instance of Perf to the instance on the right. The input shows the construction of the first set of ¯ the task T.  (the idea is illustrated in Figure 4). Proof. The proof is by reduction of Perf  , where Perf  is the problem ˆ ˆ We denote by (A, T) an arbitrary instance of Perf proved to be NP-hard in theorem 2. The input dimension of this instance ¯ T) ¯ is q − 1 and the number of hidden nodes q. We construct an instance (A, ¯ Note that this implies that the of our problem PerfLSFns with fan-in q of A. input dimension of A¯ is q2 . Let Tˆ = {(xi , αi )}. We then choose some item of Tˆ with required output 1 and redenote its input by x1 , if necessary. If ˆ T) ˆ is a no such item exists, that is, if all required outputs are 0, then (A,  ), and the reduction can be completed by ˆ T) ˆ ∈ Perf trivial yes instance ((A, constructing a trivial yes instance of PerfLSFns (it suffices to choose the node functions such that the output of A¯ is constant, with a corresponding task). Otherwise, we continue by defining the task T¯ as follows: ˆ T¯ = {((xi 0)q , αi ) : (xi , αi ) ∈ T} ∪{((x1 1)(x1 0)q−1 , 0), ((x1 0)(x1 1)(x1 0)q−2 , 0), . . . , ((x1 0)q−1 (x1 1), 0)}. ˆ where the second set is to make sure that That is, T¯ is an enlargement of T, the output node of A¯ is forced to compute the AND function (without loss of generality; see argument below). To check that this construction can be performed in polynomial time ˆ T)|, ˆ we note that the number of nodes in A¯ is equal to with respect to |(A, ˆ that the number of edges in A¯ is q2 + q = q2 + q + 1 = O(q2 ) = O(|A|), ˆ that |T| ¯ = |T| ˆ + q = O(|(A, ˆ T)|), ˆ O(q2 ) = O(|A|), and that the length of every 2 ¯ ˆ item in T is q + 1 = O(|A|). It remains to be shown that Aˆ can perform Tˆ if and only if A¯ can perform ¯ We first assume that there is a configuration of Aˆ such that Aˆ performs T. ˆ and we show that this implies that there is a configuration of A¯ such T,

Loading Deep Networks Is Hard

497

¯ We denote the hidden nodes (as well as the functions that A¯ performs T. ˆ and v¯ i (for A), ¯ i = 1, . . . , q, and the output computed by them) by vˆ i (for A) ¯ respectively. We denote the performing configuration of Aˆ as node as vˆ or v, ˆ i , y ≥ cˆi ], w ˆ i = (wˆ i1 , wˆ i2 , . . . , wˆ i(q−1) ) (and vˆ = AND). We then vˆ i (y) = [w choose the configuration of A¯ as follows: v¯ i (y) = [(wˆ i1 , wˆ i2 , . . . , wˆ i(q−1) , θ), y ≥ cˆi ] v¯ = vˆ = AND (i.e. r¯(y) = [(11 . . . 1), y ≥ q]), where θ is some low enough value, such that a 1 in the last input bit to any v¯ i generates a threshold value of less than cˆi , and hence the output 0 (we ˆ ˆ i , xj : i = 1, 2, . . . , q, j = 1, 2, . . . , |T|}). may choose any θ < min{ˆci − w The ¯ architecture A¯ configured in this way then performs T. Conversely, we assume that there is some configuration of A¯ such that A¯ ¯ The second set in the definition of T¯ then implies that there is performs T. ¯ z, say, such that v(z) ¯ exactly one possible input to the output node v, = 1, because any change of the input to v¯ must decrease the threshold value of the function computed by v¯ to a value low enough to make v¯ output 0. Thus, the function computed by v¯ is entirely determined by z, the unique vector ¯ in {0, 1}q such that v(z)=1. The node function set LSFns is closed under negation. This is due to the fact that ¬[w, . ≥ c] = [−w, . > −c] = [−w, . ≥ −c + ] for > 0 small enough. Therefore, if we replace the function computed by a node v¯ i ¯ Furthermore, it is by its negation, we still have a valid configuration of A. q easy to see that for any z ∈ {0, 1} , the boolean function from {0, 1}q to {0, 1}, which is equal to 1 only z, is in the set LSFns (in fact, this function can at q be written as [ˆz, . ≥ i=1 zi /2], where zˆ i = zi − 1/2). Since v¯ computes ¯ so such a function, we may therefore adjust the function computed by v, that the mapping computed by the entire architecture remains the same after the previous replacement of v¯ i by its negation. Indeed, it suffices to replace zi by its negation (i.e., to make z˜ = (z1 , z2 , . . . , zi−1 , ¬zi , zi+1 , . . . , zq ) ¯ z) = 1), since after the replacement of v¯ i by its the unique vector such that v(˜ negation, the ith input bit to v¯ is always the negation of what it was before the replacement, whereas the other input bits remain unchanged. Since the mapping is the same after this adjustment, the task is still performed by the architecture. By making the appropriate replacements just described for every i such that zi = 0, we obtain a performing configuration under which ¯ v(y) = 1 only if y = (11 . . . 1), which means that v¯ computes AND. The choice of the first set in the definition of T¯ then implies that Aˆ perˆ Using the above notation for the (adjusted) node functions, we let forms T. ˆ i = (w¯ i1 , w¯ i2 , . . . , w¯ iq ) (we drop the last entry w¯ i(q+1) ), and cˆi = c¯i . w ¯ which completes Hence, Aˆ can perform Tˆ if and only if A¯ can perform T, the proof.

498

D. Windisch

v11/ v11

v12/ v12

v21/ v21 v31 v(d−1)1

v(d−1)2

vd1=v ¯ is 2, the architecture A conFigure 5: In the LS-functions case, if the fan-in of A structed in the proof of theorem 3 looks like this. The circled nodes correspond to ¯ The input nodes shown (empty circles) are the only nodes receiving nonzero A. input by the constructed task T.

Using another reduction, we can generalize this last result to an arbitrary number of fixed layers. Theorem 3. PerfLSFns is NP-complete for the class of pyramidal architectures with fixed d ≥ 2 and variable q ≥ 2. Proof. The proof is accomplished by by reduction of PerfLSFns for pyramidal architectures with fixed d = 2 and variable q ≥ 2, which was proved to ¯ T) ¯ be be NP-hard in lemma 1. We denote this last problem by Perf . Let (A, an arbitrary instance of Perf with fan-in q. We also let T¯ = {(xi , αi )}. Then we construct an instance (A, T) of our problem PerfLSFns with the same fan-in q, and with the following task: ¯ T = {(xi 00 . . . 0, αi ) : (xi , αi ) ∈ T}. From this construction, it follows that the nodes receiving nonconstant input by items in T correspond exactly to A¯ (these are the circled nodes in Figure 5). ¯ T)|, ¯ beThis construction can be performed in polynomial time in |(A, ¯ ¯ cause |T| = |T|, and the length of every item in T is O(|A|), as is the number of nodes and edges in A.

Loading Deep Networks Is Hard

499

It remains to be shown that A¯ performs T¯ if and only if A performs T. We ¯ respectively (see denote the jth node in layer i by vij and v¯ ij , for A and A, ¯ To see that this implies Figure 5), and we first assume that A¯ performs T. ¯ All the other that A performs T, we choose vij = v¯ ij for all nodes v¯ ij of A. node functions of A are chosen arbitrarily, except those along the path from v21 to the output node vd1 = v, which are chosen such that the output of v21 equals the output of v for any input in the task T (the node vk1 just propagates the input received from node v(k−1)1 (k = 3, 4, . . . , d), regardless of all other input bits). With this choice, A performs T. Next, we assume that A performs T. If all outputs occurring in T are ¯ by the definition equal, then this is also true for all outputs occurring in T, ¯ Otherwise, we note that for all of T, and hence A¯ can trivially perform T. items in T, the output of v is entirely determined by the output of v21 . Indeed, all other layer 2 nodes produce the same output for every input present in T. Hence, for inputs in T, the outputs of v and v21 are either always equal or always negations of each other (here, we use the fact that not all outputs occurring in T are equal). Therefore, should the two outputs always be negations of each other, we may replace the functions computed by v21 and v by their negations (thus negating every output in T twice) without influencing the performance of T (here, as in the proof of lemma 1, we use closure under negation of LSFns). With the resulting configuration, A still performs T, and the outputs of v21 and v are equal for every input appearing in T (if v has been replaced by its negation, the output of v21 no longer gets negated). We then choose v¯ ij = vij as the configuration of ¯ which shows that A¯ performs T. ¯ Hence, A¯ performs T¯ if and only if A A, performs T. It remains to be shown that PerfLSFns is in NP. It is known (Muroga, Toda, & Takasu, 1961) that any linearly separable boolean function (i.e., any function in the set LSFns) receiving q inputs can be represented using not more than O(q2 log q) bits. Since an algorithm checking whether a given configuration of A performs T needs to evaluate every node function once for every item in T, this algorithm therefore runs in polynomial time. Hence, PerfLSFns is in NP, and this completes the proof. 4.2 AOFns. Similarly, we now consider AOFns as a node function set. The first result is proved in the same way as theorem 1: Lemma 2. PerfAOFns is NP-complete for the class of pyramidal architectures with fixed d = 3 and variable q ≥ 2. Proof. The proof is similar to that of theorem 1. The reduction of SAT to PerfAOFns remains the same, except that instead of choosing a large enough d, we choose here a large enough q = r, where r is the number of variables in the instance of SAT. The rest of the proof is left unchanged.

500

D. Windisch

The proof of the next theorem is similar to the proof of theorem 3: Theorem 4. PerfAOFns is NP-complete for any class of pyramidal architectures with fixed d ≥ 3 and variable q ≥ 2. Proof. The proof is by reduction of the result of lemma 2, proceeding as in the proof of theorem 3. The following corollary gives a result for LSFns as well as AOFns and is a direct consequence of theorems 3 and 4: Corollary 1. Perf (LS functions or AO functions) is NP-complete for the class of pyramidal architectures with variable q ≥ 2 and variable d ≥ 3 (AO functions)/d ≥ 2 (LS functions), where the parameters d and q are understood to be given as input to the loading algorithm. Proof. The proof is by identity reduction of Perf for the class of pyramidal architectures with fixed d and variable q, which was proved to be NP-hard in theorems 3 and 4. The proof that the problem is in NP is the same as the one in the proof of theorem 1 (AOFns case) or theorem 3 (LSFns case), respectively. 5 Conclusion It remains open whether the above results hold for more widely used classes of node functions, especially in the case of variable depth and fixed fan-in, in which NP-hardness was proved only for the node function set AOFns. An important extension would be to the sigmoidal activation functions used in the backpropagation learning algorithm (Rumelhart, Hinton, & Williams, ˇ ıma 1986). A theorem for sigmoidal activation functions was proved by S´ (1996): NP-hardness holds for the three-node network structure, with a restriction on the node functions, which holds, for example, if the threshold of the function computed by the output node is 0. Due to Hammer (1998b), NP-hardness of loading the three-node structure also holds with a different restriction on weights and classification accuracy. Vu (1998) considered a certain class of two-layered sigmoidal networks and showed that it is NP-hard to decide whether there exist weights minimizing the training error up to a certain constant. Similar hardness results (for approximate minimization of the quadratric error) on two-layered LSFns networks with a sigmoidal ˇ ıma (2002) extended output node are due to Bartlett and Ben-David (2002). S´ NP-hardness of approximate minimization of the squared error to the single sigmoidal node (Hoffgen, ¨ Simon, & VanHorn, 1995, showed a similar singlenode NP-hardness result for the LSFns case). DasGupta and Hammer (2000) considered two-layered sigmoidal (and other multilayered) networks, and

Loading Deep Networks Is Hard

501

the problem, proved to be NP-hard, of whether there are weights maximizing the ratio of correctly classified items in a task up to a certain constant. ˇ ıma (2002) gives a more detailed survey of these and other results. While S´ they suggest that hardness results on pyramidal architectures also hold for sigmoidal activation functions, it is not obvious how a generalization to pyramidal architectures could be achieved. Still, the above contribution provides evidence that depth is a factor that can make loading hard by itself. Acknowledgments This research was supported by an Undergraduate Summer Research Bursary from Keele University, Department of Mathematics, Keele, Staffordshire, UK, ST5 5DT. References Bartlett, P., & Ben-David, S. (2002). Hardness results for neural network approximation problems. Theoretical Computer Science, 284, 53–66. Blum, A., & Rivest, R. L. (1992). Training a three-node neural network is NPcomplete. Neural Networks, 5, 117–127. DasGupta, B., & Hammer, B. (2000). On approximate learning by multilayered feedforward circuits. In H. Arimura, S. Jain, & A. Sharma (Eds.), Algorithmic learning theory ’2000, (pp. 264–278). Berlin: Springer. de Souto, M. C. P., & de Oliveira, W. R. (1999). The loading problem for pyramidal neural networks. Electronic Journal on Mathematics of Computation. Available at: http://gmc.ucpel.tche.br/ejmc/. Garey, M. R., & Johnson, D. S. (1979). Computers and intractability: A guide to the theory of NP-completeness. San Francisco: Freeman. Hammer, B. (1998a). Some complexity results for perceptron networks. In L. Niklasson, M. Boden, & T. Ziemke (Eds.), International Conference on Artificial Neural Networks ’98 (pp. 639–518). Berlin: Springer-Verlag. Hammer, B. (1998b). Training a sigmoidal network is difficult. In M. Verleysen (Ed.), European Symposium on Artificial Neural Networks ’98 (pp. 255–260). Brussels: D-Facto Publications Hoffgen, ¨ K.-U., & Simon, H.-U., & VanHorn, K.S. (1995). Robust trainability of single neurons. Journal of Computer and System Sciences, 50, 114–125 Judd, J.S. (1990). Neural network design and the complexity of learning. Cambridge, MA: MIT Press. Muroga, S., Toda, I., & Takasu, S. (1961). Theory of majority decision elements. Journal of the Franklin Institute, 271, 376–418. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(9), 533–536. ˇ ıma, J. (1994). Loading deep networks is hard. Neural Computation, 6(5), 842– S´ 850. ˇ ıma, J. (1996). Back-propagation is not efficient. Neural Networks, 9(6), 1017– S´ 1023.

502

D. Windisch

ˇ ıma, J. (2002). Training a single sigmoidal neuron is hard. Neural Computation, S´ 14, 2709–2728. Vu, V.H. (1998). On the infeasibility of training neural networks with small squared errors. In M. I. Jordan, M. J. Kearns, & S. A. Solla (Eds.), Advances in neural information processing systems, 10 (pp. 371–377). Cambridge, MA: MIT Press. Received September 4, 2003; accepted July 1, 2004.

NOTE

Communicated by Klaus Obermayer

Maximum Likelihood Topographic Map Formation Marc M. Van Hulle [email protected] K.U.Leuven, Laboratorium voor Neuro- en Psychofysiologie, B-3000 Leuven, Belgium

We introduce a new unsupervised learning algorithm for kernel-based topographic map formation of heteroscedastic gaussian mixtures that allows for a unified account of distortion error (vector quantization), loglikelihood, and Kullback-Leibler divergence. 1 Introduction Several unsupervised learning algorithms have been devised in recent years that develop topographically organized maps of gaussian mixture densities (for references, see Van Hulle, 2002). Unifying accounts of the homoscedastic (equal-variance) case have been introduced by Graepel and coworkers (Graepel, Burger, & Obermayer, 1998) and Heskes (2001). Graepel et al. adopted a statistical physics approach, called deterministic annealing, showing the connection between different classes of kernel-based topographic map formation algorithms, and eventually, as a limiting case, the batch map version of the self-organizing map (SOM) algorithm (Kohonen, 1995). Heskes showed the connection between minimum distortion topographic map formation and maximum likelihood homoscedastic gaussian mixture density modeling. An approach different from distortion-based learning is to optimize an information-theoretic criterion. Linsker (1989) was among the first to do so by applying his principle of maximum information preservation (infomax) to topographic map formation. Another approach is to minimize the Kullback-Leibler divergence (also termed cross-entropy) between the true and estimated input densities, an idea that has been introduced in kernelbased topographic map formation by Benaim and Tomasini (1991), using homoscedastic gaussians, and extended more recently by Yin and Allinson (2001) to heteroscedastic (different-variance) gaussians. In this article, we introduce an algorithm that develops topographically organized maps of heteroscedastic gaussian mixtures. We show the link between distortion error (vector quantization), log-likelihood and Kullback Leibler divergence. We develop an alternative kernel definition that greatly simplifies the link and introduce a likelihood-based regularizer to avoid nonoptimal solutions. Furthermore, we suggest an alternative approach to log-likelihood, called differential log-likelihood, for monitoring, in an unbiased manner, the learning process. Neural Computation 17, 503–513 (2005)

c 2005 Massachusetts Institute of Technology


504

M. Van Hulle

2 Likelihood Maximization Let v = [v1 , . . . , vd ] be a random vector in V ⊆ d generated from the probability density p(v). Assume we have N formal neurons with gaussian activation functions Ki (v, wi , σi ), i = 1, . . . , N, with center wi = [wi1 , . . . , wid ] ∈ d , and radius σi , 
1 −v − wi 2 , (2.1) exp Ki (v, wi , σi ) = d 2σi2 (2πσ 2 ) 2 i

such that we obtain a homogeneous, heteroscedastic gaussian mixture estimate of the unknown input density: ˜ σ) = p(v) ≈ p(v|W,

1  Ki (v, wi , σi ). N i

(2.2)

A standard procedure to estimate the parameters W = (w1 , . . . , wN ) and σ = (σ1 , . . . , σN ) is by maximizing the (average) likelihood or by minimizing the (average) negative log-likelihood for the sample S = {vµ |µ = 1, . . . , M} (Redner & Walker, 1984): F = −log L = −

1  ˜ µ |W, σ ), log p(v M µ

(2.3)

through an expectation-maximization (EM) approach (Dempster, Laird, & Rubin, 1977). Suppose the gaussian mixture is to be developed in such a way that the N neurons form a lattice in which neighboring neurons code for neighboring positions in the input space (topology-preserving map). Define ij as the usual neighborhood function, a symmetric (ij = ji ), monotonically decreasing function of the lattice distance between neurons i and j. Before we define our gaussian activation function for such a lattice, consider the following reasoning. We take the weighted and normalized error term shown next on which we apply a bias-variance decomposition into a term depending on vµ and another independent of vµ ,  ri vµ − wi 2  ri vµ − wr 2 = + wi − wr 2 , 2 2 2 2σ 2σ 2σ r r r r i with

(2.4)

 wi = 1 σ 2i

=

ri wr σ2  rri r σr2 r

 ri r

σr2

.

,

(2.5) (2.6)

Maximum Likelihood Topographic Map Formation

The gaussian activation kernel can then be defined as   1 v − wi 2 exp − , Ki (v, wi , σ i ) = d 2σ 2i (2πσ 2i ) 2 so that we obtain the following mixture density model,  ˜ σ , Q) = qi Ki (v, wi , σi ), p(v|W,

505

(2.7)

(2.8)

i

 ri 2 where Q = (q1 , . . . , qN ), with qi = (exp (− r 2σ 2 wi − wr  ))/Z the kerr nel’s prior probability, which contains the topological information, with Z the usual normalization constant so that i qi = 1. Taking the derivatives of the log-likelihood, equation 2.3, with respect to the wi s and putting them equal to zero leads to the following set of fixed point rules,   µ µ µ r ir pr v wi =   (2.9) µ , ∀i, µ r ir pr µ

where we have substituted for the posterior probabilities P(i|vµ ) ≡ pi = µ qi Ki µ . qK j j j

We then take the derivatives of F with respect to the radii σi and put them equal to zero. This yields the following set of linear equations in (σ 1 )2 , . . . , (σ N )2 , after some straightforward algebraic manipulations:  µ

µ

pj (σ j )2 ij =

1 d

µ

j

µ

pj ij vµ − wi 2 , ∀i.

(2.10)

j

Algorithmically, we adopt an EM approach. In the expectation step, we µ determine the new values for pi . In the maximization step, we determine µ new the new wi and σi , given pi . We determine wnew from equation 2.9, i new 2 then (σ i ) by solving equation 2.10, and then σinew by solving the linear equations in σ12 , equation 2.6. i

3 Link with Vector Quantization Consider the following distortion or quantization error functional: E(W, σ ) =

 µ

+

i

j

 µ

µ µ v

ji pj

j

µ

pj

− wi 2 2σi2


 µ  ji   µ d pi − + p log log . (3.1) i 2 2 1/N i σi µ i

506

M. Van Hulle

The first term on the right-hand side resembles (apart from σi ) the familiar distortion term in topographic map formation (see, e.g., Heskes, 2001), with µ pi the probability that kernel Ki generated input vµ and ij the “confusion probability” that input vµ is instead generated by kernel Kj . The second term contains the effect of having variable kernel radii (note it is a constant when the radii are equal—the homoscedastic case—and is therefore absent in Heskes’s 2001 format, equation 1). The last term is a cross-entropy, intended to minimize the discrepancy between the prior probability assignµ ments P(i) = N1 and the actual ones, pi and thus acts as a regularization term. From the derivative with respect to wi , we obtain equation 2.9. The derivative with respect to σi leads to µ

µ 2    pj d ij ∂E µ v − wi  =− ij pj +  jk 3 3 ∂σi 2σi σi µ µ j j k 2

=−

 µ

j

σk

µ 2  µ ij µ v − wi  ij pj + pj d σ j2 3 , 3 2σi σi µ j

(3.2)

which, when put equal to zero, yields equation 2.10. The derivative with µ respect to pi becomes, after substitution of σ i , qi Ki (vµ , wi , σ i ) µ pi =  . µ j qj Kj (v , wj , σ j )

(3.3) µ

Finally, when filling in the expression for pi in equation 3.1, and after some algebraic manipulations, we obtain that √ E = (−log L + R)M( 2π)d

(3.4)

with      ij 1  R=− log  qi exp − wi − wj 2  , 2 M µ i j 2σj

(3.5)

the part that collects the topological information. We observe that, up to a scale factor, distortion minimization is equivalent to the combined loglikelihood maximization of the mixture model and the minimization of the term R, which specifies the topological relations. Finally, one should note that it is straightforward to extend our format to the heterogeneous case by taking the prior probabilities P(i) = N1 .

Maximum Likelihood Topographic Map Formation

507

4 Alternative Kernel Definition Let us adopt the following kernel function,    ri v − wr 2 1 Ki (v, W, σ ) = , (4.1) exp − Zi 2σr2 r  with Zi such that V Ki (v, W, σ ) dv = 1 (normalization constant), namely,    ij √ Zi = ( 2πσ i )d exp − (4.2) wi − wj 2 , 2 j 2σj and which can be used in the following kernel-based mixture estimate: ˜ σ) = p(v) ≈ p(v|W,

1  Ki (v, W, σ ). N i

(4.3)

Consider the following error function, E(W, σ ) =

 µ

+

i

r

 µ

ri pµ r

i

µ

pi

vµ − wi 2 2σi2


 µ pi + log Zi , log 1/N

(4.4)

which is different from equation 3.1 in the log Zi term. Filling in the expresµ sion for pi , equation 3.3, into the expression for E(W, σ ), and after some algebraic manipulations, we obtain that E = −M log L,

(4.5)

which shows the immediate link between log-likelihood maximization and distortion minimization. Conceptually, the alternative kernel is also simpler since it includes the topological information in its argument. If we take M → ∞ in the previous equation, thenwe can write the expectation of E as ˜ E (E) = − V p(v) log p(v|W, σ )dv ≥ − V p(v) log p(v)dv = E (E)min . Hence, ˜ with the latter the Kullback-Leibler divergence. E (E) − E (E)min = KL(p, p), Or, when put in the finite sample format, E = −M log L = M(KL + entropy). Since the log-likelihood is biased by the differential entropy, we cannot use this as a metric for monitoring the convergence process when the actual density p is not known. We suggest a way around that by replacing the differential entropy by the estimate − V p˜ log p˜ dv: ˜ p(v) log p(v|W, σ )dv LL = − V ˜ ˜ + p(v|W, σ ) log p(v|W, σ )dv, (4.6) V

508

M. Van Hulle

which we call the differential log-likelihood, in analogy with the differential entropy (thus also with respect to a bias). When the density estimate equals the true density, then this metric will equal zero. The advantage is that we can generate as large a data set as desired from p˜ (Monte Carlo simulation of data) for obtaining an accurate kernel density estimate of entropy (Ahmad & Lin, 1976), or a sample spacing estimate in the one-dimensional case (Beirlant, Dudewicz, Gyorfi, ¨ & van der Meulen, 1997). 5 Simulation: Topographic Map Formation Consider the standard case of a 10×10 lattice that is trained according to the alternative kernel formulation on a sample S of size M = 1000 drawn from the uniform distribution in the unit square [−1, 1]2 . The weights are initialized by sampling the same distribution; the radii are initialized by sampling the uniform distribution [0.25, 0.75]. We use a gaussian neighborhood function,  (i, j, σ ) = exp −

ri − rj 2



2σ2

,

(5.1)

with σ the neighborhood radius and ri neuron i’s lattice coordinate. We adopt the following neighborhood cooling scheme, 
t σ (t) = σ0 exp −σ0 , tmax

(5.2)

with t the current epoch and tmax the maximum number of epochs, and σ0 the radius at t = 0. We take tmax = 50, σ0 = 5. The results are shown in Figure 1. We observe that the lattice unfolds extremely rapidly; initially, the weights are at the sample’s centroid, and the kernel radii grow rapidly in size and span a considerable part of the input distribution. Finally, we observe in the evolution of LL (see Figure 2) that from about 30 epochs onward, it hovers around zero, as desired. 6 Correspondence with Other Algorithms We can verify that when taking σi = √1 , ∀i (i.e., homoscedastic gaussian β

mixture case), equation 3.1 becomes equal to equation 1 in Heskes (2001), except for the second term, which is missing in Heskes’s case (since it becomes a constant). Heskes established the link between the error function and the log-likelihood in an approximate √ sense, since the second term was omitted, as well as the scale factor M( 2π)d . Hence, we can say that our first approach is the generalization of Heskes’s to the heteroscedastic case. Our approach with the alternative kernel function is, however, different and

Maximum Likelihood Topographic Map Formation

509

0

2

4

5

10

20

40

50

1.0

|

0.5

|

0.0

|

-0.5

|

∆LL

Figure 1: Evolution of a 10 × 10 lattice as a function of time (epochs). The circles demarcate the radii σi of the corresponding gaussian kernels. The boxes correspond to the range spanned by the uniform input density. The values given below the boxes represent the number of epochs elapsed.

|

-1.0 | 0

| 10

| 20

| 30

| | 40 50 epochs

Figure 2: Evolution of LL as a function of epochs.

becomes equal to Heskes’s only when the neighborhood range vanishes (ij = δij ). The advantage of the heteroscedastic case is that no separate procedure is needed for optimizing the (common) kernel radii. Also, a benefit is expected when the input density consists of components with strongly varying spreads. ˜ as Yin and Allinson (2001) used the Kullback-Leibler divergence KL(p, p) the goal function. Since for our alternative kernel function, we could write ˜ = E (E) − E (E)min , with the latter term a constant, given the that KL(p, p) ˜ and E (E) is equivainput density, performing gradient descent on KL(p, p)

510

M. Van Hulle

lent. The weight and radii update rules should correspond to ours (albeit, they used incremental learning); however, they were developed differently µ since Yin and Allinson considered the posterior pi to be an approximation of the neighborhood function (in input space coordinates) and further simplified the radii update rules by adopting a competitive stage (winner µ = arg maxi pi ). We have separate terms for the neighborhood function (in lattice space coordinates) and the posteriors, and no competitive stage. In the soft topographic vector quantization (STVQ) algorithm for homoscedastic mixture modeling (Graepel, Burger, & Obermayer, 1997), the same fixed-point rule for the weight updates is obtained as ours, but with a common gaussian kernel radius σi = √1 ∀i, with β the inverse temperβ

ature, which is decreased exponentially over time (deterministic annealing) and thus serves a different purpose. Graepel et al. (1998) also proposed the kernel-based soft topographic mapping (STMK) algorithm by introducing a nonlinear transformation that maps the data points to a highdimensional “feature” space and, in addition, admits a kernel function, such as a gaussian with fixed radius, as in (kernel-based) support vector machines (SVMs) (Vapnik, 1995). The topographic map’s parameters (“weights”) are expressed as linear combinations of the transformed inputs, so that the map is in fact developed in feature space rather than in input space directly, as in our case. The idea of Graepel and coworkers was taken up again by Andr´as (2002) but with the purpose of optimizing the map’s classification performance, by individually adjusting the kernel radii using a supervised learning algorithm. This basically sets Andr´as’s approach apart from ours. In the kernel-based maximum entropy learning Rule (kMER) (Van Hulle, 1998), the kernel outputs are thresholded, and, depending on the binary activations, the kernel centers and radii are adapted. In the current approach, both the activation states and the learning rules depend on the continuously graded kernel outputs. µ Finally, when taking pi = δii∗ , with i∗ = arg mini vµ − wi 2 , that is, the “winning” neuron, we obtain the SOM algorithm in batch mode (Kohonen, µ 1995), which only the wi s. When we take pi = δii∗µ , with i∗µ =  adapts 1 µ 2 arg mini j ij v − wj  , and σi = √ , ∀i in equation 3.1, then the error β   function becomes E = β2 µ i ii∗µ vµ −wi 2 , plus a nonrelevant constant. This is the format introduced by Heskes and Kappen (1993) and Luttrell (1991) in order to derive the SOM weight update rule from an error function. 7 Regularization Interestingly, we can derive a likelihood-based regularization term for the likelihood functional, equation 2.3, that will help in finding relevant local  minima (e.g., nonvanishing kernel radii). Since P(i) = V p(v)P(i|v)dv, with  µ 1 ˆ = 1 in our case P(i) = N1 , we should have that P(i) µ pi ≈ N (note that the M

Maximum Likelihood Topographic Map Formation

511

maximum likelihood estimate is asymptotically unbiased for large M; for a textbook account, see Rice, 1995). We suggest the following log-likelihood regularization term, R=−

1  1  µ µ log pi /M = − log pi − log M, NM µ i NM µ i

(7.1)

µ

which is minimized when pi becomes similar to N1 . We can further drop the constant − log M since it is irrelevant to the minimization. The regularized log-likelihood functional then becomes Freg = F + αR, with α a parameter   1 µ (usually α  1). Note that the regularization term − µ i NM log pi is   1 µ minimized, or µ i NM log pi maximized, and that in the error function µ   µ pi is minimized. E(·), equation 3.1, the regularization term µ i pi log 1/N Hence, the two regularization terms are different. 8 Simulations: Regularization Consider a one-dimensional lattice (“chain”) consisting of N = 9 neurons with gaussian kernels, trained on a sample S sized M = 1000, drawn from the standard normal distribution G(0, 1). We take tmax = 50, σ0 = 4.5, σi (0) = 0.2, ∀i. In order to explore the effect of the log-likelihood constraint, we consider α = 0.001 and α = 0, with and without the constraint. We initialize the kernel centers by taking N = 9 data points from either the uniform distributions [−2, 2] or [−1, 1] or the sample set itself; for the initial radii, we take 0.2. We run each simulation 19 times and determine, for an independent test set of size M = 1000, the median value of the mean squared error (MSE) between the density estimate and the true distribution, as well as the first and third quantiles (the mean value is dominated by unsuccessful runs). The results are summarized in Table 1. We repeat the simulations for a skewed distribution consisting of a mixture of two gaussians, G(0, 1) and G(1.5, 1/3), with mixing coefficients 3/4 and 1/4. The results are summarized in Table 2. Both sets of results show a clear advantage of likelihood-based regularization in the case of a uniform random initialization, as is usually done in topographic map formation, and, thus, also that the cross-entropy regularization term in equation 3.1 is not very effective. For the case where the data set is sampled, the advantage is less clear; however, it can still matter for sparse distributed data sets. 9 Conclusion We have introduced a log-likelihood maximization approach to variable kernel-based topographic map formation. We have shown that for a specific kernel definition, a simple relation exists between distortion minimization, log-likelihood maximization, and Kullback-Leibler divergence. We have

512

M. Van Hulle

Table 1: MSE Results (in 10−4 ) in Case of Standard Normal Input Density. Initialization [−2, 2] [−1, 1] S

α = 0.001

α=0

1.46 [0.686, 1.91] 1.33 [0.789, 2.00] 1.20 [0.850, 2.44]

27.1 [3.33, 433] 2.12 [0.984, 2.95] 1.29 [0.724, 2.18]

Notes: Results are listed in terms of medians and first and third quartiles (in brackets). The input density is estimated using 1000 data points and N = 9 gaussian kernels, with (α = 0.001) and without log-likelihood constraint (α = 0), and for three different kernel center initializations: by means of samples taken from the uniform distributions [−2, 2] and [−1, 1] or the sample S.

Table 2: MSE Results (in 10−4 ) in Case of a Skewed Bimodal Input Density. Initialization [−2, 2] [−1, 1] S

α = 0.001

α=0

2.15 [1.32, 3.36] 1.92 [1.06, 2.69] 2.18 [0.961, 2.99]

326 [4.03, 651] 2.08 [0.875, 2.56] 2.28 [0.948, 3.41]

Note: Same conventions as in Table 1.

also introduced a regularizer and suggested a possible metric, the differential log-likelihood, for monitoring the convergence process. There are at least two ways to extend the practical value of this work, as exemplified in Heskes (2001): the extension to the missing values case in an EM sense and the use of the marginal probabilities of the neurons of the trained topographic map for data visualization and exploration purposes. All these are clear advantages over the classic SOM algorithm. Showing the practical value of the differential log-likelihood metric is the subject of a follow-up technical paper. Acknowledgments I was supported by research grants from the Belgian Fund for Scientific Research–Flanders (G.0248.03 and G.0234.04), the Flemish Regional Ministry of Education (Belgium) (GOA 2000/11), and the European Commission (IST-2001-32114 and IST-2002-001917). References Ahmad, I. A., & Lin, P. E. (1976). A nonparametric estimation of the entropy for absolutely continuous distributions. IEEE Trans. Information Theory, 22, 372–375.

Maximum Likelihood Topographic Map Formation

513

Andr´as, P. (2002). Kernel-Kohonen networks. Int. J. Neural Systems 12(2), 117– 135. Beirlant, J., Dudewicz, E. J., Gyorfi, ¨ L., & van der Meulen, E. C. (1997). Nonparametric entropy estimation: An overview. Int. J. Math. and Statistical Sciences 6, 17–39. Benaim, M., & Tomasini, L. (1991). Competitive and self-organizing algorithms based on the minimization of an information criterion. In Proc. ICANN’91 (pp. 391–396). Amsterdam: North-Holland. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood for incomplete data via the EM algorithm. J. Roy. Statist. Soc., B, 39, 1–38. Graepel, T., Burger, M., & Obermayer, K. (1997). Phase transitions in stochastic self-organizing maps. Physical Rev. E, 56(4), 3876–3890. Graepel, T., Burger, M., & Obermayer, K. (1998). Self-organizing maps: Generalizations and new optimization techniques. Neurocomputing, 21, 173–190. Heskes, T. (2001). Self-organizing maps, vector quantization, and mixture modeling. IEEE Trans. Neural Networks, 12(6), 1299–1305. Heskes, T. M., & Kappen, B. (1993). Error potentials for self-organization. In Proc. IEEE Int. Conf. on Neural Networks (pp. 1219–1223). New York: IEEE. Kohonen, T. (1995). Self-organizing maps. Heidelberg: Springer. Linsker, R. (1989). How to generate ordered maps by maximizing the mutual information between input and output signals. Neural Computat., 1, 402–411. Luttrell, S. P. (1991). Code vector density in topographic mappings: Scalar case. IEEE Trans. Neural Networks, 2, 427–436. Redner, R. A., & Walker, H. F. (1984). Mixture densities, maximum likelihood and the EM algorithm. SIAM Review, 26(2), 195–239. Rice, J. A. (1995). Mathematical statistics and data analysis. Belmont, CA: Wadsworth. Van Hulle, M. M. (1998). Kernel-based equiprobabilistic topographic map formation. Neural Computat., 10(7), 1847–1871. Van Hulle, M. M. (2002). Joint entropy maximization in kernel-based topographic maps. Neural Computat., 14(8), 1887–1906. Vapnik, V. N. (1995). The nature of statistical learning theory. New York: SpringerVerlag. Yin, H., & Allinson, N. M. (2001). Self-organizing mixture networks for probability density estimation. IEEE Trans. Neural Networks, 12, 405–411. Received April 28, 2004; accepted July 27, 2004.

NOTE

Communicated by Kiichi Urahama

On Convergence Conditions of an Extended Projection Neural Network Youshen Xia [email protected] Department of Applied Mathematics, Nanjing University of Posts and Telecommunications, China

Gang Feng Department of Manufacturing Engg. and Engg. Management, The City University of Hong Kong, Hong Kong, China

The output trajectory convergence of an extended projection neural network was developed under the positive definiteness condition of the Jacobian matrix of nonlinear mapping. This note offers several new convergence results. The state trajectory convergence and the output trajectory convergence of the extended projection neural network are obtained under the positive semidefiniteness condition of the Jacobian matrix. Comparison and illustrative examples demonstrate applied significance of these new results. 1 Introduction We are concerned with the following extended projection neural network developed in Xia (2004): State equation:   PX (x − (F(x) + ∇ g(x)y − BT z)) − x du  = λ (y + g(x))+ − y dt −Bx + b

(1.1)

Output equation: w(t) = x(t), where λ > 0 is a scaling constant, u = (x, y, z) ∈ Rn ×Rm ×Rr is a state vector, w is an output vector, B ∈ Rr×n , b ∈ Rr , F(x) is a continuously differentiable vector-valued function from Rn into Rn , g(x) = [g1 (x), . . . , gm (x)]T , gi (x)(i = 1, . . . , m) are twice differentiable functions from Rn into R1 , ∇ g(x) = (∇ g1 (x), . . . , ∇ gm (x)), ∇ gi (x) is the gradient of gi (i = 1, . . . , m), X = {x ∈ Rn |l ≤ x ≤ h}, l = [l1 , . . . , ln ]T , h = [h1 , . . . , hn ]T , (y)+ = [(y1 )+ , . . . , (ym )+ ]T , (yi )+ = Neural Computation 17, 515–525 (2005)

c 2005 Massachusetts Institute of Technology 

516

Y. Xia and G. Feng

max{0, yi }, and PX : Rn → X is a projection operator defined by  xi < li li li ≤ xi ≤ hi PX (xi ) = xi  hi xi > hi . As shown in Xia (2004), the extended projection neural network in equation 1.1 has a low computational complexity and a fast convergence rate. Moreover, its equilibrium point is related to the solution of the following constrained variational inequality, (x − x∗ )T F(x∗ ) ≥ 0,

x ∈ ,

(1.2)

where  = {x ∈ Rn | g(x) ≤ 0, Bx = b, l ≤ x ≤ h}. More exactly, if u∗ = (x∗ , y∗ , z∗ ) is an equilibrium point of equation 1.1, then x∗ is a solution of equation 1.2 (Bertsekas & Tsitsiklis, 1989; Harker & Pang, 1990). Thus, the desirable solution to equation 1.2 can be obtained by tracking the continuous trajectory of equation 1.1 in real time (Cichocki & Unbehauen, 1993; Golden, 1996; Urahama, 1996). The output trajectory convergence of the extended projection neural network in equation 1.1 has been proven under the condition that the Jacobian matrix ∇F(x) of F is positive definite on X and constraint functions gi (x) (i = 1, . . . , m) are convex on X (Xia, 2004). As a result, the extended neural network is theoretically guaranteed to solve strictly monotone variational inequality problems with general convex constraints. It is well known that studying the state trajectory convergence of the extended neural network is of research interest and applied significance. On the other side, in much of real-world engineering optimization, the nonlinear mapping F(x) is monotone instead of strictly monotone (De Farias & Van Roy, 2003). Moreover, there exist many cases that F(x) and ∇ gi (x) are not monotone (Oja,1992; Bell & Sejnowski, 1995; Welling & Webber, 2001; Ahn & Oh, 2003). Therefore, studying weak convergence conditions of the extended projection neural network in equation 1.1 is very desirable for wide optimization and engineering applications. This note proposes new conditions of the global convergence of the extended projection neural network in equation 1.1. The proposed new results not only remove the positive definiteness condition of the Jacobian matrix of F on X but also allow F(x) and ∇ gi (x) to be nonmonotone. Therefore, the extended projection neural network is further guaranteed to solve constrained monotone variational inequality problems and a class of constrained nonmonotonic variational inequality problems. 2 Main Results Definition 1. A mapping F is said to be monotone on X if, for each pair of points x, y ∈ X, we have (x − y, F(x) − F(y)) ≥ 0,

On Convergence Conditions of an Extended Projection Neural Network

517

where (· , ·) denotes an inner product. F is said to be strictly monotone on X if the strict inequality holds whenever x = y. Throughout the convergence analysis, we assume that the extended projection neural network in equation 1.1 has at least an equilibrium point u∗ = (x∗ , y∗ , z∗ ). Also, we denote the Hessian matrix of gi (x) by ∇ 2 gi (x). Our main results on the convergence of the extended projection neural network in equation 1.1 are the following. Theorem 1. Assume that F(x) and ∇ gi (x) (i = 1, . . . , m) are monotone on X. If ∇F(x∗ ) is positive definite, then the extended projection neural network in equation 1.1 converges globally to an equilibrium point of equation 1.1. Proof. Let u(t) = (x(t), y(t), z(t)) be the state trajectory of equation 1.1 with the initial point u(t0 ) = (x(t0 ), y(t0 ), z(t0 )). Since by Xia and Wang (2000), we see that u(t) will exponentially approach to set 0 = {u = (x, y, z) ∈ Rn+m+r | x ∈ X, y ∈ Rm / X and y(t0 ) < 0, without + } if x(t0 ) ∈ the loss of generality we assume that x(t0 ) ∈ X and y(t0 ) ≥ 0. Consider the following Lyapunov function, 1 1 V(u) = H(u)T T(u) − T(u)2 + u − u∗ 2 , 2 2

u ∈ 0 ,

(2.1)

where u∗ = (x∗ , y∗ , z∗ ) is an equilibrium point of equation 1.1, and  F(x) + ∇ g(x)y − BT z , H(u) =  −g(x) Bx − b 



 x − PX (x − (F(x) + ∇ g(x)y − BT z)) . T(u) =  y − (y + g(x))+ Bx − b From the result obtained by Xia (2004) we know that V(u) ≥ u(t) ⊂ 0 , and

1 2 u

dV(u) ≤ −λ(H(u∗ ) − H(u))T (u∗ − u) − λT(u)T ∇H(u)T(u), dt where   ∇F(x) + ni=1 yi ∇ 2 gi (x) ∇H(u) =  −∇ g(x)T BT

∇ g(x) O1 O3

 −B O2  , O4

− u∗ 2 ,

(2.2)

518

Y. Xia and G. Feng

and O1 ∈ Rn×m , O2 ∈ Rn×r , O3 ∈ Rr×m , and O4 ∈ Rr×r are zero matrices. n Since2 F(x) and ∇ gi (x) (i = 1, . . . , m) are monotone on X, ∇F(x) + i=1 yi ∇ gi (x) are positive semidefinite on 0 . Thus, ∇H(u) is positive semidefinite, and H(u) is monotone on 0 . It follows that dV(u)/dt ≤ 0, and thus {u(t)} is bounded. By invariant set theorem (Golden, 1996), we see that all solution trajectories of the extended projection neural network in equation 1.1 converge to a largest invariant set χ, where dV(u)/dt = 0. We now prove that dV/dt = 0 if and only if du/dt = 0. Clearly, if du/dt = 0, then ˆ zˆ ) ∈ χ. Then dV/dt = 0 ˆ y, dV(u)/dt = (dV(u)/du)T (du/dt) = 0. Let uˆ = (x, ˆ T (u∗ − u) ˆ = 0. Since implies (H(u∗ ) − H(u)) ˆ T (u∗ − u) ˆ = (H(u∗ ) − H(u))



1

ˆ T (∇F(xs ) (x∗ − x)  2 ∗ ˆ ds, + m i=1 ∇ gi (xs )yi (s))(x − x) 0

ˆ and yi (s) = yˆ i + s(y∗i − yˆ i ) (0 ≤ s ≤ 1), where xs = xˆ + s(x∗ − x) (xˆ − x∗ )T (∇F(xs ) +

m

∇ 2 gi (xs )yi (s))(xˆ − x∗ ) = 0.

i=1

 2 ∗ ∗ Let s = 1. Then xs = x∗ , y∗i (s) = y∗i , and (xˆ − x∗ )T (∇F(x∗ ) + m i=1 ∇ gi (x )yi ) m ∗ ∗ 2 ∗ ∗ ∗ · (xˆ − x ) = 0. Since ∇F(x ) + i=1 ∇ gi (x )yi is positive definite, xˆ = x and thus ˆ u) ˆ = r(u) ˆ T (∇F(x∗ ) + ˆ T ∇H(u)T( T(u)

m

ˆ = 0, ∇ 2 gi (x∗ )y∗i )r(u)

i=1

ˆ = 0. Thus, ˆ = px (xˆ − F(x) ˆ It follows that r(u) ˆ − ∇ g(x) ˆ yˆ + BT zˆ ) − x. where r(u) ˆ = 0 and dz/dt = λ(−Bxˆ + b) = 0. Finally, consider the case dx/dt = −λr(u) ˆ Note that dy/dt = λ((yˆ + g(x)) ˆ = ˆ + − y). ˆ + − y) that dy/dt = λ((yˆ + g(x)) −λα yˆ (0 ≤ α ≤ 1). When α > 0, we have yˆ = 0 since {y(t)} is bounded. Thus, dy/dt = 0 also. So dV/dt = 0 if and only if du/dt = 0. The conclusion of theorem 1 thus holds. As an important corollary of theorem 1, we have the following result: Corollary 1.Assume that F(x) + ∇ g(x)y(t) is monotone on X, where y(t) is the trajectory defined in equation 1.1. If ∇F(x∗ ) is positive definite, then the extended protection neural network in equation 1.1 is globally convergent to an equilibrium point of equation 1.1. Proof. Let u(t) = (x(t), y(t), z(t)) be the state trajectory of equation 1.1 with the initial point (x(t0 ), y(t0 ), z(t0 )), where x(t0 ) ∈ X and y(t0 ) ≥ 0. Consider the Lyapunov function V(u) defined in equation 2.1. Note that

On Convergence Conditions of an Extended Projection Neural Network

519

 2 ∇F(x)+ m i=1 ∇ gi (x)yi (t) is positive semidefinite on X when F(x)+∇ g(x)y(t) is monotone on X. Similar to the analysis in theorem 1, we can obtain that dV/dt ≤ 0 and dV/dt = 0 if and only if du/dt = 0. Therefore, the conclusion of corollary 1 holds. Theorem 2. Assume that F(x) and ∇ gi (x) (i = 1, . . . , m) are monotone on X. If there exists a nonnegative vector p = [p1 , . . . , pm ]T ≤ y∗ such that ∇F(x) + m 2 i=1 ∇ gi (x)pi is positive definite on X, then the output trajectory of the extended protection neural network in equation 1.1 with y(t0 ) ≥ p is convergent to a solution of equation 1.2 at a convergence rate w(t) − x∗ 2 ≤

δ , ∀t > t0 , λ(t − t0 )

(2.3)

where δ is positive constant. Proof. Let u(t) = (x(t), y(t), z(t)) be the state trajectory of equation 1.1 with the initial point (x(t0 ), y(t0 ), z(t0 )) satisfying x(t0 ) ∈ X and y(t0 ) ≥ p. Consider the Lyapunov function V(u) defined in equation 2.1. By the analysis of theorem 1 we have λ

t

(H(u(τ )) − H(u∗ ))T (u(τ ) − u∗ ) dτ

t0

≤ 2V(u(t0 )) − λ

t t0

1

T(u(τ ))T ∇H(u(τ )))T(u(τ )) dτ.

0

Since dy(t)/dt + y = (y + g(x))+ , y(t) = e−(t−t0 ) y(t0 ) + e−t



t

es (y + g(x))+ ds.

t0

Then y(t) ≥ e−(t−t0 ) p for t ≥ t0 . Using r(u) = PX (x−(F(x)+∇ g(x)y−BT z))−x, we have m

T(u)T ∇H(u(t))T(u) = r(u)T (∇F(x(t))+ ∇ 2 gi (x(t))yi (t))r(u) i=1

≥ r(u)T (∇F(x(t))+e−(t−t0 )

m

∇ 2 gi (x(t))yi (t0 ))r(u)

i=1

= e−(t−t0 ) r(u)T (e(t−t0 ) ∇F(x(t))+

m

∇ 2 gi (x(t))pi )r(u)

i=1

≥ e−(t−t0 ) r(u)T (∇F(x(t)) +

m

i=1

∇ 2 gi (x(t))pi )r(u) ≥ 0.

520

Y. Xia and G. Feng

On the other side, note that (H(u) − H(u∗ ))T (u − u∗ ) =



1 0

+

(x − x∗ )T (∇F(xs )

m

i=1 ∇

2 g (x )y (s))(x i s i

− x∗ ) ds,

where xs = x∗ + s(x − x∗ ) and yi (s) = y∗i + s(yi − y∗i ) for 0 ≤ s ≤ 1. Then t t0

1

(x(τ ) − x∗ )T (∇F(xs ) +

m

0

∇ 2 gi (xs )yi (s))(x(τ ) − x∗ ) ds dτ

i

=

t t0



t

=

1

(u(τ ) − u∗ )T ∇H(u∗ + s(u(τ ) − u∗ ))(u(τ ) − u∗ ) ds dτ

0

(H(u(τ ))−H(u∗ ))T (u(τ )−u∗ ) dτ ≤ V(u(t0 ))/λ,

t0

Since y∗ ≥ 0, yi (s) = (1 − s)y∗i + syi ≥ syi (t) ≥ e−(t−t0 ) spi (i = 1, . . . , m). It follows that for 0 < s ≤ 1, (x − x∗ )T (∇F(xs ) +

m

∇ 2 gi (xs )yi (s))(x − x∗ )

i=1

≥ (x − x∗ )T (∇F(xs ) +

m

∇ 2 gi (xs )(se−(t−t0 ) )pi )(x − x∗ )

i=1

= (x − x∗ )T (∇F(xs ) + (se−(t−t0 ) )

m

∇ 2 gi (xs )pi )(x − x∗ )

i=1

≥ s(e(t−t0 ) )(x − x∗ )T (∇F(xs ) +

m

∇ 2 gi (xs )pi )(x − x∗ ) ≥ 0.

i=1

Note that for 1 > s ≥ 0,

1

(x − x∗ )T (∇F(xs ) +

0

m

∇ 2 gi (xs )yi )(x − x∗ ) ds

i

≥

1

(x − x∗ )T (∇F(xs ) + (1 − s)

0

≥ 0

m

∇ 2 gi (xs )y∗i )(x − x∗ ) ds

i 1

(1 − s)(x − x∗ )T (∇F(xs ) +

m

i

∇ 2 gi (xs )y∗i )(x − x∗ ) ds.

On Convergence Conditions of an Extended Projection Neural Network

521

Using y∗ ≥ p, we have t t0

1

(1−s)(x(τ )−x∗ )T (∇F(xs )+

m

0

∇ 2 gi (xs )pi )(x(τ )−x∗ ) ds dτ

i

≤ V(u(t0 ))/λ.  2 Since ∇F(x) + m i=1 ∇ gi (x)pi is positive definite on X, there exists µ > 0 such that  m

∗ T 2 ∗ ∇ gi (xs )yi (x − x∗ ) ≥ µx − x∗ 2 . (x − x ) ∇F(xs ) + i

It follows that t 1 V(u(t0 )) µ x(τ ) − x∗ 2 dτ (1 − s) ds ≤ λ t0 0 and



t

x(τ ) − x∗ 2 dτ ≤

t0

2V(u(t0 )) . λµ

Thus, for any t > t0 ,

t

x(t) − x  < x(tˆ) − x  = ∗ 2

∗ 2

x(s) − x∗ 2 ds

t0

t − t0

≤

δ , λ(t − t0 )

where t0 < tˆ < t and δ = 2V(u(t0 ))/µ. Hence, w(t) − x∗ 2 ≤

δ , ∀t > t0 . λ(t − t0 )

As an important corollary of theorem 2, we have the following result.  2 Corollary 2.Assume that ∇F(x) + m i=1 ∇ gi (x)yi (t) is positive definite on X, where y(t) is the trajectory defined in equation 1.1. Then the output trajectory of the extended neural network in equation 1.1 is globally convergent to a solution of equation 1.2 with a convergence rate in the form 2.3. Proof. Let u(t) = (x(t), y(t), z(t)) be the state trajectory of equation 2.1 with the initial point (x(t0 ), y(t0 ), z(t0 )), where x(t0 ) ∈ X and y(t0 ) ≥ 0. Consider the Lyapunov function V(u) defined in equation 2.1. According to the analysis in theorem 2, we can obtain that dV/dt ≤ 0 and t t0

1

(x(τ ) − x∗ )T (∇F(xs ) +

0

≤ V(u(t0 ))/λ.

m

i

∇ 2 gi (xs )yi (s))(x(τ ) − x∗ ) ds dτ

522

Y. Xia and G. Feng

Therefore, from the rest of the analysis of theorem 2, the conclusion of corollary 2 follows.

3 Comparison and Illustrative Examples In Xia (2004), the existing convergence condition of the extended projection neural network in equation 1.1 is that ∇F(x) is positive definite on X and ∇ gi (x) (i = 1, . . . , m) are monotone. It is easy to see that theorems 1 and 2 have removed the positive definiteness condition of ∇F(x) on X. Moreover, theorem 1 shows the global convergence of the state trajectory of the extended neural network. Furthermore, from corollaries 1 and 2, we see that F(x) and ∇ gi (x) may be nonmonotone on X. Two examples follow. Example 1. Consider the following quadratic program, 1 T x Qx + cT x 2

minimize

f (x) =

subject to

Dx ≤ b,

(3.1)

and its dual, maximize

1 T 1 y Hy + yT d − cT Q−1 c 2 2

subject to

y ≥ 0,

(3.2)

where x ∈2 , y ∈ R4 , H = DQ−1 DT , d = −b−DQ−1 c, b = [35/12, 35/2, 5, 5]T , and  5/12 D= −1

5/2 1

−1 0

T

0 1



2 , Q= 1

   −30 1 . , c= −30 2

Equation 3.1 has a unique optimal solution x∗ = (5, 5), and equation 3.2 has a unique optimal solution y∗ = (0, 6, 0, 0, 9). We use the extended neural network in equation 1.1 to solve both equations 3.1 and 3.2. Let F(x) = Qx + c and  = {x ∈ R2 | Dx − b ≤ 0}. Then equation 3.1 can be rewritten as the problem with the form equation 1.2. When the extended projection neural network in equation 1.1 is applied to this problem, the corresponding extended projection neural network in equation 1.1 can become     d x −c − QT x − DT y = . (y + Dx − b)+ − y dt y

(3.3)

On the other side, by the dual theory of convex program (Bertsekas, 1989), we see that (x∗ , y∗ ) is an optimal solution to equations 3.1 and 3.2, respec-

On Convergence Conditions of an Extended Projection Neural Network

523

tively, if and only if (x∗ , y∗ ) satisfies the Karush-Kuhn-Tucker (KKT) condition, 

c + QT x + DT y = 0, yT (Dx − b) = 0, Dx ≤ b, y ≥ 0.

The KKT condition can be represented by 

c + QT x + DT y = 0, (y + Dx − b)+ = y.

Thus, (x∗ , y∗ ) is an optimal solution to equations 3.1 and 3.2, respectively, if and only if (x∗ , y∗ ) is an equilibrium point of equation 3.3. The existing convergence result (Xia, 2004) can show that the output trajectory w(t) of the extended projection neural network is convergent to the optimal solution of equation 3.1, but it cannot ascertain the convergence of the trajectory y(t) to the optimal solution of equation 3.2. However, theorem 1 guarantees that the state trajectory of the extended projection neural network can converge to optimal solutions of both equations 3.1 and 3.2. All simulation results show that the state trajectory u(t) = (x(t), y(t)) of (8) is always convergent to u∗ = (x∗ , y∗ ). For example,let λ = 10. Figure 1 displays the transient behavior of the norm error u(t)−u∗  based on equation 3.3 with 10 random initial points. Example 2. Consider nonlinearly constrained variational inequality 1.2, where  = {x ∈ R2 | g(x) ≤ 0, x ∈ X}, X = {x ∈ R2 | x ≥ 0},g(x) = [g1 (x), g2 (x)]T , g1 (x) = −x21 − x2 , g2 (x) = −x22 − x1 , and F(x) = [2x1 − 1, 2x2 − 0.5]T . This problem has a solution x∗ = [0.5, 0.25]T . It is easy to see that ∇F(x) is positive definite, but ∇ g1 (x) and ∇ g2 (x) are not monotone on X. Thus, the existing convergence result Xia (2004) cannot ascertain the extended neural network for solving the above problem. However, note that y(t) always converges to zero. Then there exists T > t0 such that when t ≥ T,

∇F(x) +

2

i=1

∇ 2 gi (x)yi (t) =

 2 − 2y1 (t) 0

 0 2 − 2y2 (t)

is positive on X. Thus, corollary 2 guarantees that the extended neural network in equation 1.1 globally converges to x∗ and has a convergence rate in the form of equation 2.3. All simulation results show the output trajectory of equation 1.1 is always convergent to x∗ . For example, let λ = 10. Figure 2 displays the transient behavior of the norm error x(t) − x∗  based on equation 2.1 with 10 random initial points.

524

Y. Xia and G. Feng

14

12

||u(t)−u*||

10

8

6

4

2

0

0

0.1

0.2

0.3

0.4

0.5 Time (sec)

0.6

0.7

0.8

0.9

1

Figure 1: Convergence behavior of u(t) − u∗  with 10 random initial points in example 1. 1.4

1.2

1

*

||x(t)−x ||

0.8

0.6

0.4

0.2

0

0

0.1

0.2

0.3

0.4

0.5 Time (sec)

0.6

0.7

0.8

0.9

1

Figure 2: Convergence behavior of x(t) − x∗  with 10 random initial points in Example 2.

On Convergence Conditions of an Extended Projection Neural Network

525

Conclusion This note proposes new convergence conditions of the extended neurel network. Both theoretical analysis and computational examples show that the extended neural network can solve constrained monotone variational inequality problems and a class of constrained nonmonotonic variational inequality problems. References Ahn, J. H., & Oh, J. H. (2003). A constrained EM algorithm for principal component analysis. Neural Computation, 15, 57–66. Bell, A. J., & Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7, 1129–1159. Bertsekas, D. P., & Tsitsiklis, J. N. (1989). Parallel and distributed computation: Numerical methods. Englewood Cliffs, NJ: Prentice-Hall. Cichocki, A., & Unbehauen, R. (1993). Neural networks for optimization and signal processing. New York: Wiley. De Farias, D.P., & Van Roy, B. (2003). The linear programming approach to approximate dynamic programming. Operations Research, 51, 850–865. Golden, R. M. (1996). Mathematical methods for neural network analysis and design. Cambridge, MA: MIT Press. Harker, P. T., & Pang, J. S. (1990). Finite-dimensional variational inequality and nonlinear complementary problems: A survey of theory, algorithms, and applications. Mathematical Programming, 48, 161–220. Oja, E. (1992). Principal components, minor components and linear neural networks. Neural Networks, 5, 927–935. Urahama, K. (1996). Gradient projection network: Analog solver for linearly constrained nonlinear programming. Neural Computation, 8, 1061–1074. Welling, M., & Webber, M. (2001). A constrained EM algorithm for independent component analysis. Neural Computation, 13, pp. 677–689. Xia, Y. S. (2004). An extended neural network for constrained optimization. Neural Computation, 16, 863–883. Xia, Y. S., & Wang, J. (2000). On the stability of globally projected dynamic systems, Journal of Optimization Theory and Applications, 106, pp. 129–150. Received April 23, 2004; accepted July 28, 2004.

LETTER

Communicated by Jerome Feldman and Peter Thomas

Memorization and Association on a Realistic Neural Model Leslie G. Valiant [email protected] Division of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, U.S.A.

A central open question of computational neuroscience is to identify the data structures and algorithms that are used in mammalian cortex to support successive acts of the basic cognitive tasks of memorization and association. This letter addresses the simultaneous challenges of realizing these two distinct tasks with the same data structure, and doing so while respecting the following four basic quantitative parameters of cortex: the neuron number, the synapse number, the synapse strengths, and the switching times. Previous work has not succeeded in reconciling these opposing constraints, the low values of synapse strengths that are typically observed experimentally having contributed a particular obstacle. In this article, we describe a computational scheme that supports both memory formation and association and is feasible on networks of model neurons that respect the widely observed values of the four quantitative parameters. Our scheme allows for both disjoint and shared representations. The algorithms are simple, and in one version both memorization and association require just one step of vicinal or neighborly influence. The issues of interference among the different circuits that are established, of robustness to noise, and of the stability of the hierarchical memorization process are addressed. A calculus therefore is implied for analyzing the capabilities of particular neural systems and subsystems, in terms of their basic numerical parameters. 1 Introduction We consider four quantitative parameters of a neural system that together constrain its computational capabilities: the number of neurons, the number of neurons with which each neuron synapses, the strength of synaptic connections, and the speed of response of a neuron. The typical values that these parameters are believed to have in mammalian cortex appear to impose extremely severe constraints. We believe that it is for this reason that computationally explicit mechanisms for realizing multiple cognitive tasks simultaneously on models having these typical cortical parameters have not been previously offered. Neural Computation 17, 527–555 (2005)

c 2005 Massachusetts Institute of Technology


528

L. Valiant

Estimates of these four cortical parameters are known for several systems. The number of neurons in mouse cortex has been estimated to be 1.6 × 107 , while the corresponding estimate is in the region of 1010 for humans (Braitenberg & Schuz, 1998). There also exist estimates of the number of neurons in different parts of cortex and in related structures such as the hippocampus and olfactory bulb. The number of neurons with which each neuron synapses, which we shall call the degree, is a little harder to measure. However, it is considered that the effect of multiple synapsing between pairs of neurons is small and therefore that this degree is close to the total number of synapses per neuron, which has been estimated to be 7800 in mouse cortex and in the 24,000 to 80,000 range in humans (Abeles, 1991). The third parameter, the synapse strength, presents a still more complex set of issues. The most basic question here is how many of a neuron’s neighbors need to be sending an action potential in order to create an action potential in the neuron. Equivalently, the contribution of each synapse, the excitatory presynaptic potential, can be measured in millivolts and the fraction that this constitutes of the threshold voltage that needs to be overcome evaluated. While some moderately strong synapses have been recorded (Thomson, Deuchars, & West, 1993; Markram & Tdodyks, 1996a; Ali, Deuchars, Pawelzik, & Thomson, 1998), the average value is believed to be weak. The effective fraction of the threshold that each neighbor contributes, has been estimated (Abeles, 1991) to be in the range 0.003 to 0.2. In other words, it is physiologically quite possible that cognitive computations are characterized by a very small and hard-to-observe fraction of synapses that contribute above this range, perhaps even up to the threshold fraction of 1.0. However, there is no experimental confirmation of this to date, and for that reason, this article addresses the possibility that at least some neural systems work entirely with synapses whose strengths are some orders of magnitude smaller. The time it takes for a neuron to complete a cycle of causing an action potential in response to action potentials in its presynaptic neurons has been estimated as being in the 1 to 10 millisecond range. Since mammals can perform significant cognitive tasks in 100 to 200 milliseconds, algorithms that take a few dozen parallel steps must suffice. The central technical contribution of this article is to show that two basic computational problems, memory formation and association, can be implemented consistently in model neural systems that respect all of the above-mentioned numerical parameters. The first basic function, JOIN, implements memory formation of a new item in terms of two established items: if two items A and B are already represented in the neural system, the task of JOIN is to modify the circuit so that at subsequent times, there is the representation of a new item C that will fire if and only if the representations of both A and B are firing. Further, the representation of C is an “equal citizen” with those of A and B for the purposes of subsequent calls of JOIN. JOIN

Memorization and Association on a Realistic Neural Model

529

is intended as the basic form of memorization of an item and incorporates the idea that such memorization has to be indexed in terms of the internal representations of items already represented. The second function, LINK, implements association. If two “items” A and B are already represented in the neural system in the sense that certain inputs can cause either of these to fire, the task of LINK is to modify the circuit so that at subsequent times, whenever the representation of item A fires, the modified circuit will cause the representation of item B to fire also. Implicit in the definitions of both JOIN and LINK is the additional requirement that there be no deleterious interference or side effects. This means that the circuit modifications do not impair the functioning of previously established circuits; that when the newly created circuit executes, no unintended other items fire; and that the intended action of the new circuit cannot be realized in consequence of some unintended condition. We note that for some neural systems, such as the hippocampus and the olfactory bulb, the question of what items, whether representing location or odors, for example, are being represented has been the subject of some experimental study already. Also for such systems, our four cortical parameters can be measured. We therefore expect that our analysis offers both explanatory and predictive value for understanding such systems. For the parts of cortex that process higher-level functions, the corresponding experimental evidence is more elusive. In order that our results apply to a wide range of neural systems, we describe computational results for systems within a broad range of realistic parameters. We show that for wide ranges of values of the neuron count between 105 and 109 , and of values of the synapse count or degree between 16 and 106 , there is a range of values of the synapse strength between .001 and .125 for which both JOIN and LINK can be implemented. Furthermore, this latter range usually includes synaptic strengths that are at the small end of the range. Tables 1 through 4 summarize these data and show, given the values of the neuron count and degree of a neural system, the maximum synapse strength that is sufficient for JOIN and LINK in both disjoint and shared representations. The implied algorithms for LINK take just one step and for JOIN either two steps (see Tables 1 and 2) or also just one step (see Tables 3 and 4). The simplicity of these basic algorithms leaves room for more complex functions to be built on top of them. We also describe a general relationship among the parameters that holds under some stated assumptions for systems that use the mechanisms described. This relationship (∗) states that kn exceeds rd, but only by at most a fixed small constant factor, where 1/k is the maximum collective strength of the synapses to any one neuron from any one of its presynaptic neurons, n is the number of neurons, r is the number of neurons that represent a single item, and d is the number of neurons from which each neuron receives synapses.

530

L. Valiant

The essential novel contribution of this article is to show that random graphs have some unexpected powers. In particular, for parameters that have been observed in biology, they allow a method of assigning memory to a new item and also allow for paths, and algorithms for establishing the paths, for realizing associations between items. There is a long history of studies of random connectivity for neural network models, notably Beurle (1955), Griffith (1963, 1971), Braitenberg (1978), Feldman (1982), and Abeles (1991). In common with such previous studies, ours assumes random interconnections and does not apply to systems where, for example, the connections are strictly topographic. The other component of our approach that also has some history is the study of local representations in neural networks, including Barlow (1972), Feldman (1982), Feldman and Ballard (1982), Shastri and Ajjanagadde (1993), and Shastri (2001). The question of how multiple cognitive functions can be realized simultaneously using local representations and random connections has been pursued by Valiant (1988, 1994). Our central subject matter is the difficulty of computing flexibly on sparse networks where nodes are further frustrated in having influence on others by the weakness of the synapses. This difficulty has been recognized most explicitly in the work of Griffith (1963) and Abeles (1991). Griffith suggests communication via chains that consist of sets of k nodes chained together so that each member of each set of k nodes is connected to each member of the next set in the chain. If the synaptic strength of each synapse is 1/k, then a signal can be maintained along the chain. Abeles suggests a more general structure, which he calls a synfire chain, in which each set has h ≥ k nodes and each node is connected to k of the h nodes in the next set. He shows that for some small values of k, such chains can be found somewhere in suitably dense such networks. The goals of this article impose multiple constraints for which these previous proposals are not sufficient. For example, for realizing associations, we want that between any pair of items, there is a potential chain of communication. In other words, these chains have to exist from anywhere to anywhere in the network rather than just somewhere in the network. A second constraint is that we want explicit computational mechanisms for enabling the chain to be invoked to perform the association, and the passive existence of the chain in the network is not enough. A third requirement is that for memory formation, we need connectivity of an item to two others. Some readers may choose to view this article as one that solves a communication or wiring problem, and not a computational one. This view is partly justified since once it is established that the networks have sufficiently flexible interaction capabilities, the mechanisms required at the neurons are computationally very simple. For readers who wish to investigate more rigorously what simple means here, we have supplied a section that goes into more details. The model of computation used there is the neuroidal model (Valiant, 1994), which was designed to capture the communication capa-

Memorization and Association on a Realistic Neural Model

531

bilities and limitations of cortex as simply as possible. It assumes only the simplest timing and state change mechanisms for neurons so that there can be no doubt that neurons are capable of doing at least that much. Demonstrating that some previously mysterious task can be implemented even on this simple model therefore has explanatory power for actual neural systems. The neuroidal model was designed to be more generally programmable than its predecessors and hence to offer the challenge of designing explicit computational mechanisms for explicitly defined and possibly multiple cognitive tasks. The contribution of this article may be viewed as that of exhibiting a wide range of new solutions to that model. The previous solutions given for the current tasks were under the direct action hypothesis—the hypothesis that synapses could become so strong that a single presynaptic neuron was enough to cause an action potential in the postsynaptic neuron. Whether this hypothesis holds for neural systems that perform the relevant cognitive tasks is currently unresolved. In contrast, the mechanisms described here are in line with synaptic strength values that have been widely observed and generally accepted. This letter pursues a computer science perspective. In that field, it is generally found, on the positive side, that once one algorithm has been discovered for solving a computational problem within specified resource bounds, many others often follow. On the other hand, on the negative side, it is found that the resource bounds on computation can be very severe. For example, for the NP-complete problem of satisfiability (Cook, 1971; Papadimitriou 1994) of boolean formulas with n occurrences of literals, no n algorithm for solving all instances of it in 2f( ) steps is known for any function f(n) growing more slowly than linear in n. If a device were found that could solve this problem faster, then a considerable mystery would be created: the device would be using some mechanism that is not understood. Neuroscience has mysteries of the same computational nature and needs to resolve them. This letter aims at making one of these mysteries concrete and to resolve it. 2 Graph Theory We consider a random graph G with n vertices (Bollobas, 2001). From each vertex, there is a directed edge to each other vertex with probability p, so that the expected number of nodes to which a node is connected is d = p(n − 1). In this model, a vertex corresponds to a neuron, and a directed edge from one vertex to another models the synapse between the presynaptic neuron and the postsynaptic neuron. Such a model makes sense for neural circuits that are richly interconnected. A variant of the model is that of a bipartite graph where the vertex set can be partitioned into two subsets V1 and V2 , such that every edge is directed from a V1 vertex to a V2 vertex. This would be appropriate for modeling the connections from one area of cortex to a

532

L. Valiant

second, possibly distant, area. The analyses we give for JOIN and LINK apply equally to both variants. We shall use d = pn in the analysis, which is exact for the bipartite case, and a good approximation for the other case for large n. In general, to obtain rigorous results about random graphs, we take the view that for the fixed nodes under consideration, the edges are present or not, each with probability p independent of each other. It is convenient for analysis to view the edges as being generated in that manner afresh rather than as fixed at some previous time. We assume that the maximum synaptic strength is 1/k of the threshold, for some integer k. In the graph theoretic properties, we shall therefore always need to find at least k presynaptic neighbors to model the k presynaptic neurons that need to be active to make the neuron in question active. Finally, we shall model the representation of an item in cortex by a set of about r neurons, where r is the replication factor. In general, such an item will be considered to be recognized if essentially all the constituent neurons are active. In general, different items will be represented by different numbers of neurons, though of the same order of magnitude. We do not try to ensure that they are all represented by exactly r. However, once an item is represented by some r neurons, then it makes sense to assert that if no more than r /2 of its members are firing, then the item has not been recognized. We call a representation disjoint or shared, respectively, depending on whether the sets that represent two distinct items need, or need not, be disjoint. In disjoint representations, clearly, no more than n/r items can be represented, while shared representations allow for many more, in principle. 2.1 Memory Formation for Disjoint Representations. The JOIN property is the following. Given values of n, d, and k, which are the empirical parameters of the neural system, we need to show that the following holds. Given two subsets of nodes A and B of size r, the number of nodes to which there are at least k edges directed from A nodes and also at least k edges directed from B nodes has expected value of r. The vertices that are so connected will represent the new item C. The above-mentioned property ensures that the representation of C will be made to fire by causing the representation of either one of A or B to fire. The required network is illustrated in Figure 1. We want C to be an equal citizen as much as possible with A and B for the purposes of further calls of JOIN. We ensure this by requiring that the expected number of nodes that represent C is the same as the number of those that represent A and B. In general, we denote by B (r, p, k) the probability that in r tosses of a coin that comes up heads with probability p and tails with probability 1−p, there will be k or more heads. This quantity is equal to the sum for j ranging from k to r of the value of T (r, p, j) = (r!/(j!(r − j)!)p j (1 − p)r−j . For constructing our tables, we compute such terms to double precision using an expansion

Memorization and Association on a Realistic Neural Model

r

A

533

k k C k

r

B

k

Figure 1: Graph-theoretic structure needed for the two-step algorithm for disjoint representations for the memorization of the conjunction of items at A and B. For shared representations, the sets A and B may intersect. For the one-step algorithm, there is a bound km on the total number of edges coming from A and B rather than bounds on A and B separately.

for the logarithm of the factorial or gamma function (Abramowitz & Stegun, 1964). We now consider the JOIN property italicized above. For each vertex u in the network, the probability that it has at least k edges directed toward it from the r nodes of A is B (r, p, k), since each vertex of A can be regarded as a coin toss with probability p of heads (i.e., of being connected to u) and we want at least k successes. The same holds for the nodes of B. Hence the probability of a node being connected in this way to both A and B is p = (B (r, p, k))2 , and hence the expected number of vertices so connected is n times this quantity. The stated requirement on the JOIN property therefore is that the following be satisfied: n(B (r, p, k))2 = r.

(2.1)

This raises the important issue of stability. Even if the numbers of nodes assigned to A and B are both exactly r, this process will assign r nodes to C only in expectation. How stable is this process if such memorization operations are performed in sequence, with previously memorized items forming the A and B items of the next memorization operation? Fortunately, it is easy to see that this process gets more and more stable as r increases. The argument for relationship 2.1 showed that the number of nodes assigned to C is a random variable with a binomial distribution defined as the number of successes in n trials where the probability of success in each trial is p .

534

L. Valiant

This distribution therefore has mean np , as already observed, and variance np (1 − p ). The point is that this variance is close to the mean r = np if p is small relative to 1, and then the standard deviation, which is the square root √ of the variance, is approximately r. Hence, the standard deviation as a √ fraction of the mean decreases as 1/ r as the mean np = r increases. For the ranges of values that occur typically in this article, such as r equal to 103 , 104 , √ or even much larger, this r standard deviation will be a small fraction of r, and hence one can expect the memorization process to be stable for many stages. Thus, for stability, the large k large r cases considered in this article are much more favorable than the k = 1 case considered in Valiant (1994) with r = 50. For the latter situation, some analysis and suggested ways of coping with the more limited stability were offered in Valiant (1994) and Gerbessiotis (2003). If fewer than a half of the representatives of an item are firing, we regard that item as not being recognized. As a side-effect condition, we therefore want that if no more than a half of one of A or B is active, then the probability that more than a half of C is active is negligible. Since we cannot control the size of C exactly, we ensure the condition that at most half of C be active by insisting that at most a much smaller fraction of r, such as r/10, be active. The intention is that r/10 will be smaller than a half of C even allowing for the variations in the size of C after several stages of memory allocation. This gives

B (n, B (r/2, p, k)B (r, p, k), r/10) ∼ 0.

(2.2)

The second side-effect condition we impose is related to the notion of capacity, or the number of items that can be stored. To guarantee large capacity, we need an assurance that the A ∧ B nodes allocated will not be caused to fire if a different conjunction is activated. The bad case is if the second conjunction involves one of A or B, say A, and another item D different from B. If the node sets allocated to A ∧ B and not to A ∧ D is of size at least 2r/3, then we will consider there to be no interference since if A ∧ B is of size at most 4r/3, then the firing of A ∧ D will cause fewer than half of the nodes of A ∧ B to fire. The probability that a node receives k inputs from B and k from A, but fewer than k from D, is p = (1 − B (r, p, k))(B (r, p, k))2 , and we want that the number of nodes that are so allocated to A ∧ B but not to A ∧ D to be at least 2r/3. Hence, we want

B (n, p , 2r/3) ∼ 1.

(2.3)

2.2 Association for Disjoint Representations. We now turn to the LINK property, which ensures that B can be caused to fire by A via an intermediate set of “relay” neurons: Given two sets A and B of r nodes, for each B vertex u with high probability, the following occurs: the number of (relay) vertices from which

Memorization and Association on a Realistic Neural Model

k r

A

535

k B

r

Figure 2: Graph-theoretic structure needed for the algorithm for establishing an association of the item at A to the item at B.

there is a directed edge to u and to which there are at least k edges directed from A nodes is at least k. We shall call this probability Y. This property ensures that each neuron u that represents B will be caused to fire with high probability if the A representation fires. This property is illustrated in Figure 2. For the LINK property, we note that the probability of a vertex having at least k connections from A and also having a connection to B vertex u is pB (r, p, k). We need the number of such nodes to be at least k with high probability, or in other words that Y = B (n, pB (r, p, k), k) ∼ 1.

(2.4)

As a side-effect condition, we need that if at most half of A fire, then with high probability, fewer than half of B should fire. We approximate this quantity by assuming independence for the various u:

B (r, B (n, pB (r/2, p, k), k), r/2) ∼ 0.

(2.5)

As a second side-effect condition, we consider the probability that a third item C for which no association with B has been set up by a LINK operation will cause B to fire because some relay nodes are shared with A. Further, we make this more challenging by allowing there to have been t, rather than just one, association with B previously set up, say from A1 , . . . , At . Now, the probability that a node will act as a relay node from A1 to a fixed node u in B is pB (r, p, k). For t such previous associations, the probability that a node acts as a relay for at least one of the Ai is p = p(1−(1− B (r, p, k))t ). If we require that these nodes be valid relay nodes from item C also, then this probability gets multiplied by another factor of B (r, p, k) since C is disjoint from A1 , . . . , At .

536

L. Valiant

Then the side-effect requirement becomes that p = B (n, p B (r, p, k), k), the probability of there being at least k relay nodes for u is so small that it is unlikely that a large fraction, say at least r/2, of the B nodes are caused to fire by C. We approximate this quantity also by making the assumption of independence for the various u:

B (r, p , r/2) ∼ 0.

(2.6)

2.3 Memory Formation for Shared Representations. By shared representation, we mean a representation where each neuron can represent more than one item. There is no longer a distinction between nodes that have and those that have not been allocated. The items each node will be assigned by JOIN are already determined by the network connections without any training process being necessary. The actual meaning of the items that will be memorized at a node will, of course, depend on the meanings assigned by a different process, such as the hard wiring of certain sensory functions to some input nodes. The model here is that an item is represented by r neurons, randomly chosen. The expected intersection of two such sets then is of size r2 /n. We can recompute the relations corresponding to equations 2.1 to 2.6 under this assumption. For simplicity, we shall consider only the case in which for JOIN, the neurons of A and B are in one area, and those of C in another, and for LINK the neurons of A, B and the relay nodes are in three different areas, respectively. Then, if for simplicity we make the assumption that the intersection is of size exactly r , the closest integer to r2 /n, then relation 2.1 for JOIN becomes: n{B (r , p, k) + (i=0:k−1) [T (r , p, i)(B (r − r , p, k − i))2 ]} = c1 r,

(2.1’)

where i in the summation indexes the number of connections from the intersection of A and B. We need to show that c1 is close to one. Equation 2.1 we adapt from equation 2.2 to be the following:

B (n, p , r/10) ∼ 0,

(2.2’)

where p =(B (r , p, k)+(i=0,...,k−1) [T (r , p, i)B (r−r , p, k−i)B (r/2−r , p, k− i)]), where i in the summation indexes the number of connections from a neuron that is in both A and B, and r = r2 /n. We adapt equation 2.3 from equation 2.3 by considering the case that the intersections A ∩ B, A ∩ D, and B ∩ D all have their expected sizes r = r2 /n, and that A ∩ B ∩ D has its expected size r = r3 /n2 . For a fixed node in C, we shall denote the numbers of connections to these four intersections by i + m, j + m, l + m, and m, respectively, in the summation below. Then the probability of a node being allocated to A ∧ B and not to A ∧ D is lower

Memorization and Association on a Realistic Neural Model

537

bounded by the following quantity p , where r∧ = r − r : (i=0:k) T (r∧ , p, i)(j=0:k−i) T (r∧ , p, j)(l=0:k−i−j) T (r∧ , p, l) (m=0:k−i−j−l) T (r , p, m) [B (r − 2r + r , p, k − i − j − m)B (r − 2r + r , p, k − j − l − m) (1 − B (r − 2r + r , p, k − i − l − m))]. (Note that to allow for terms in which the intersection of A and B have more than k connections to the C node, we need to interpret the first term T (r∧ , p, i) in the above expression for the particular value i = k to mean B (r∧ , p, k).) Then the relationship we need is

B (n, p , 2r/3) ∼ 1.

(2.3’)

2.4 Association for Shared Representations. Equations 2.4 and 2.5 in the shared coding are identical to equations 2.4 and 2.5 since we are assuming here, for simplicity, that in an implementation of LINK between A and B, the neurons representing A, B, and the relay nodes are from three disjoint sets. Equation 2.6 will correspond to equation 2.6 in the special case of t = 1. For a fixed node in B, the probability of a node u being a relay node to it and having k edges coming from both A and C is p = p(B (r , p, k) + (i=0,...,k−1) [T (r , p, i)(B (r − r , p, k − i))2 ]). The probability that there are at least k of these is p = B (n, p , k). We want the probability that at least half of the members of B have such sets of at least k relay nodes to be small. We approximate this quantity by assuming independence for the various u:

B (r, p , r/2) ∼ 0.

(2.6’)

2.5 One-step Memory Formation for Shared Representations. The algorithm for memorization implied above requires A and B to be active at different time instants and therefore requires the memorization process to take two steps. Here we shall discuss a process by which memorization can be achieved in one step. For brevity, we shall consider one-step only for the shared representation case. The results are presented in Tables 3 and 4. The main advantage of these one-step algorithms is that the algorithms become even simpler and assume even less about the timing mechanism of the model of computation. But one small complication arises: the number of inputs needed to fire a node in the memorization algorithm is now different from that needed in the association algorithm. Instead of having a single parameter k, we shall have two parameters, km and ka , respectively. It turns

538

L. Valiant

out that km = 2ka works. This means that we can fix the parameter k of the neurons to be km , and then use a weight in the memorization algorithm that is only half as strong as the maximum value allowed. We now consider the JOIN property. For each vertex u that is a potential representative of C, the probability that it has at least km edges directed toward it from the nodes of A ∪ B is now p = B (2r − r , p, km ), provided the intersection of A and B is of size exactly r , the integer closest to the expectation r2 /n. This is because each vertex of A ∪ B may be regarded as a coin toss with probability p of coming up heads (i.e., of being connected to u), and we want at least km successes. Hence, the expected number of vertices so connected is n times this quantity. The stated requirement on the JOIN property therefore is that the following be satisfied: nB (2r − r , p, km ) − r.

(2.1”)

Alternatively one could impose some specific probability distribution on the choice of A and B and compute an analog of equation 2.1 that is precise for that distribution and does not need the assumption that the intersections are of size exactly r . If fewer than half of the representatives of an item are firing, we regard that item as not being recognized. As a side-effect condition, we therefore want that if no more than half of one of A or B is active, then the probability that more than half of C is active is negligible. In exact analogy with equations 2.2 and 2.2 , we have that

B (n, B (3r/2, p, km ), r/10) ∼ 0.

(2.2”)

Here, 3r/2 upper bounds the number of firing nodes in A ∪ B if at most r/2 are firing in A and all are firing in B, say. As a second side-effect condition, we again need an assurance that the A ∧ B nodes allocated will not be caused to fire if a different conjunction is activated. Again, a bad case is if the second conjunction is A ∧ D where D is different from B. If the node set allocated to A ∧ B and not to A ∧ D is at least of size 2r/3, we will consider there to be no interference since if A ∧ B is of size less than 4r/3 then the firing of A ∧ D will cause fewer than half of the nodes of A ∧ B to fire. If A, B, and D were disjoint sets of r nodes, then the probability that a node receives km inputs from A ∪ B but fewer than k = km from A ∪ D would be p = (s=0:k−1) T (r, p, s)(B (r, p, km − s))(1 − B (r, p, km − s)), where s denotes the number of nodes in A that are connected to that node. We want the number of nodes that are so allocated to A ∧ B but not to A ∧ D to be at least 2r/3. Hence we would want that

B (n, p , 2r/3) ∼ 1.

(2.3”)

Memorization and Association on a Realistic Neural Model

539

In the event that A, B, and D are not disjoint but randomly chosen sets of r elements, we need equation 2.3 but with a value of p computed as follows. We assume that the intersections A∩B, A∩D, and B∩D all have their expected sizes r = r2 /n, and that A ∩ B ∩ D has its expected size r = r3 /n2 . For a fixed node in C, we shall denote the number of connections to these four intersections by i + m, j + m, l + m, and m, respectively, in the summation below. Then the probability of a node being allocated to A ∧ B and not to A ∧ D is lower-bounded by the following quantity p , where r∧ = r −r and r# = r − −2r + r : p = (s=0:k−1) T (r# , p, s)(i=0:k−s−1) T (r∧ , p, i)(j=0:k−i−s−1) T (r∧ , p, j) (m=0:k−i−j−s−1) T (r , p, m)(l=0:k−i−j−m−s−1) T (r∧ , p, l) [(B (r# , p, km − s − i − j − l − m)) (1 − B (r# , p, km − s − i − j − l − m))]. Here s indexes the number of connections from nodes that are in A but not in B or D. 2.6 One-Step Association with Shared Representations. For the LINK property we simply have relations 2.4, 2.5, and 2.6 with k replaced by ka : Y = B (n, pB (r, p, ka ), ka ) ∼ 1,

(2.4”)

B (r, B (n, pB (r/2, p, ka ), ka ), r/2) ∼ 0,

(2.5”)

B (r, p , r/2) ∼ 0,

(2.6”)

and

where p = B (n, p , k), and p = p(B (r , p, ka ) + (i=0,...,k−1) [T (r , p, i)(B (r − r , p, ka − i))2 ]). Since equation 2.6 guarantees only that one previous association to item C will not lead to false associations with C, we also use equation 2.6 to compute an estimate of the maximum number of previous associations that still allow resistance to such false associations. 3 Graph-Theoretic Results In Tables 1 and 2 we summarize the solutions we have found. For each combination of n, d, and k, the table entry gives a value of r that satisfies all six

540

L. Valiant

conditions 2.1 to 2.6 to high precision, as well as their equivalents, conditions 2.1 , 2.2 , 2.3 , and 2.6 , for shared representations. (There are essentially only eight equations since equations 2.2 and 2.6 , subsume equations 2.2 and 2.6, respectively.) For example consider a neural system with n = 1,000,000 neurons where each one is connected to 8192 others on the average and the maximum synaptic strengths are 1/64 of the threshold amount. The entry 8491 found in Table 1 gives the value of r that solves the 10 constraints. It means that if each item is represented by about 8491 neurons, then the graph has the capability of realizing JOIN and LINK using the algorithms to be outlined in later sections. The central positive result of this article is the existence of entries in the tables for combinations of neuron numbers, synapse strengths, and synapse numbers that are widely observed in neural systems. We expect that the analysis that underlies the tables offers a basis for a calculus for understanding the algorithms and data structures used in specific systems, such as the hippocampus or the olfactory bulb. In interpreting the tables the following comments are in order. The solutions were found by solving equation 2.1 using binary search, and discarding any solutions that failed to solve the remaining relations. As a further comment, we note that equation 2.1 and some of the others are defined only for integer values of r, and in our search we therefore imposed the constraint of allowing only such integer values. The values of r are therefore the integer values for which the value of the left-hand side of equation 2.1 is as close as possible to r. We detail the exact integer values as a reminder of this. For all the entries shown, the difference between the two sides of equation 2.1 is at most 1%, except for those labeled ∗, where a difference of 10% is allowed. The following are some further observations: The case of k = 1 in equation 2.4 is known (Valiant, 1994) to give an asymptotic value of Y ∼ 1−1/e = .63 . . .. In general, values k = 1, 2, and 4 violate at least one of either equation 2.2 or 2.5. Most of the entries support equation 2.6 only up to t = 1, except for some entries with k = 8 or 16 and with some of the higher values of d. Finally, corresponding entries for different values of the neuron count n are in the ratio of the values of n. We note that for the k = 1 case, the analysis in Valiant (1994) relates to the analysis here in the following way. It is observed there that in general for any r and n, the graph density that supports JOIN is too sparse to support LINK with a Y value close to 1. The suggested solution there is to use a graph that is dense enough to support LINK and to have it ignore a random fraction of the connections when implementing JOIN so as to effectively use a sparser graph regime for that purpose. On a separate issue, the earlier analysis did not have any equivalents of the relations 2.2 and 2.5 above. For disjoint representations, where the intention is that in each situation, either all or none of the representatives of an item fires, these conditions might be argued to be too onerous. Tables 3 and 4 summarize the solutions we have found that support the one-step algorithm. We note that in general, corresponding values of r are a

Memorization and Association on a Realistic Neural Model

541

Table 1: Value of the Replication Factor r That Is the Closest Integer Solution to Equation 2.1 for the Given Values of the Neuron Count n, the Degree d, and the Inverse Synaptic Strength k. k=8 n = 100,000 neurons d = 128 d = 256 1981 d = 512 899 d = 1024 412 d = 2048 d = 4096 d = 8192 d = 16,384 d = 32,768 d = 65,536 n = 1,000,000 neurons d = 128 d = 256 19,803 d = 512 8979 d = 1024 4105 d = 2048 1888 ∧ 873 d = 4096 ∧ 406 d = 8192 ∧ 190 d = 16,384 d = 32,768 d = 65,536 d = 131,072 d = 262,144 d = 524,288 n = 10,000,000 neurons d = 128 d = 256 198,025 d = 512 89,777 d = 1024 41,033 d = 2048 18,868 ∧ 8717 d = 4096 ∧ 4043 d = 8192 ∧ 1882 d = 16,384 ∧ 879 d = 32,768 d = 65,536 d = 131,072 d = 262,144 d = 524,288 d = 1,048,576

k = 16

5025 2338 1098 519 *247 *119

50,235 23,365 10,957 5168 2449 1165 ∧ 557 ∧ 268 ∧ *130

k = 32

5420 2582 1238 597 *290 *143

54,181 25,796 12,353 5940 2866 1388 ∧ 675 ∧ 330 ∧ *163

k = 64 k = 128 k = 256 k = 512 k = 1024

2749 1337 654 *322 ∧ *162

2865 1407 *695 ∧ *347

27,460 13,330 6491 3169 1552 ∧ *763 ∧ *378 ∧ *190

28,607 14,015 6883 ∧ 3388 ∧ *1672 ∧ *830 ∧ *415

1458

∧ *727

14,497 7161 ∧ 3545 ∧ 1761

*1496

∧ *753

14,836 ∧ 7360 ∧ 3660 ∧ *1827

502,339 233,636 541,791 109,547 257,940 51,660 123,498 274,571 24,467 59,368 133,265 286,021 11,628 28,628 64,866 140,098 ∧ 5542 13,839 31,643 68,761 144,886 ∧ 2649 6704 15,464 33,802 71,516 148,248 ∧ 1269 ∧ 3255 ∧ 7570 ∧ 16,640 ∧ 35,342 ∧ 73,463 ∧ 610 ∧ 1584 ∧ 3712 ∧ 8202 ∧ 17,484 ∧ 36,437 ∧ 295 ∧ 773 ∧ 1824 ∧ 4049 ∧ 8660 ∧ 18,089

∧ 74,842

Notes: Equation 2.1 is accurate to ratio 10−2 (but only 10−1 if marked by ∗) and Equation 2.4 to 10−6 . Equations 2.2, 2.3, 2.5, 2.6, 2.2 , 2.3 , and 2.6 are accurate to 10−6 . For unmarked entries these accuracies are achieved even if the noise rates are 10−4 for equations 2.2, 2.3, 2.5, and 2.2 ; 10−5 for 2.6 and 2.3 ; and 10−6 for 2.6 . For entries marked with a ∧ , this accuracy is achieved with the lower noise rate 10−6 for all seven equations. Equation 2.1 is satisfied with constant 1 < c1 < 1.1.

k = 16

5,023,377 2,336,342 1,095,455 516,578 244,648 116,254 ∧ 55,394 ∧ 26,455 ∧ 12,660 ∧ 6069 ∧ 2915

k=8

n = 100,000,000 neurons d = 128 d = 256 1,980,239 d = 512 897,763 d = 1024 410,318 d = 2048 188,664 ∧ 87,153 d = 4096 ∧ 40,412 d = 8192 ∧ 18,797 d = 16,384 ∧ 8766 d = 32,768 ∧ 4098 d = 65,536 d = 131,072 d = 262,144 d = 524,288 d = 1,048,576 5,417,894 2,579,374 1,234,952 593,653 286,244 138,351 67,001 ∧ 32,501 ∧ 15,789 ∧ 7681

k = 32

2,745,675 1,332,609 648,612 316,375 154,582 ∧ 75,636 ∧ 37,053 ∧ 18,172

k = 64

2,860,166 1,400,921 687,547 337,949 ∧ 166,314 ∧ 81,930 ∧ 40,397

k = 128

1,448,778 715,063 ∧ 353,310 ∧ 174,721 ∧ 86,468

k = 256

1,482,369 ∧ 734,499 ∧ 364,217 ∧ 180,718

k = 512

∧ 371,950

∧ 748,226

k = 1024

Table 2: The Value of the Replication Factor r That Is the Closest Integer Solution to Equation 2.1 for the Given Values of the Neuron Count n, the Degree d, and the Inverse Synaptic Strength k.

542 L. Valiant

50,233,759 23,363,401 10,954,534 5,165,758 2,446,454 1,162,518 553,914 ∧ 264,523 ∧ 126,564 ∧ 60,656 ∧ 29,112

n = 1,000,000,000 neurons d = 128 d = 256 19,802,377 d = 512 8,977,613 d = 1024 4,103,166 d = 2048 1,886,626 ∧ 871,517 d = 4096 ∧ 404,099 d = 8192 ∧ 187,946 d = 16,384 ∧ 87,638 d = 32,768 ∧ 40,955 d = 65,536 d = 131,072 d = 262,144 d = 524,288 d = 1,048,576 54,178,923 25,793,723 12,349,496 5,936,498 2,862,401 1,383,469 669,965 ∧ 324,966 ∧ 154,842 ∧ 76,758

k = 32

27,456,714 13,326,048 6,486,077 3,163,694 1,545,764 ∧ 756,296 ∧ 370,461 ∧ 181,644

k = 64

28,601,613 14,009,157 6,875,400 3,379,417 ∧ 1,633,055 ∧ 819,214 ∧ 403,874

k = 128

14,487,695 7,150,540 ∧ 3,532,995 ∧ 1,747,094 ∧ 864,553

k = 256

∧ 1,807,014

∧ 3,642,013

14,823,572

∧ 7,344,851

k = 512

∧ 3,719,279

∧ 7,482,072

k = 1024

Notes: Equation 2.1 is accurate to ratio 10−2 and equation 2.4 to 10−6 . Equations 2.2, 2.3, 2.5, 2.6, 2.2 , 2.3 , and 2.6 are accurate to 10−6 . For unmarked entries, these accuracies are achieved even if the noise rates are 10−4 for 2.2, 2.3, 2.5, and 2.2 ; 10−5 for 2.6 and 2.3 ; and 10−6 for 2.6 . For entries marked with a ∧ , this accuracy is achieved with the lower noise rate 10−6 for all seven equations. Equation 2.1 is satisfied with constant 1 < c1 < 1.1.

k = 16

k=8

Table 2: Continued.

Memorization and Association on a Realistic Neural Model 543

544

L. Valiant

Table 3: The Value of the Replication Factor r That Is the Closest Integer Solution to Equation 2.1 for the Given Values of the Neuron Count n, the Degree d, and the Inverse Synaptic Strength k, where km = 2k and ka = k. k = 8 k = 16

k = 32

k = 64 k = 128 k = 256 k = 512 k = 1024

n = 100,000 neurons d = 128 d = 256 d = 512 2134 5170 d = 1024 1000 2436 d = 2048 473* 1164 2653 d = 4096 562* 1284* d = 8192 628* d = 16,384 d = 32,768 d = 65,536 n = 1,000,000 neurons d = 128 d = 256 d = 512 d = 1024 3571 9974 24327 d = 2048 4707 11,603 26,487 d = 4096 2233 5576 12,792 d = 8192 1065 2692 6219 d = 16,384 1305 3037 d = 32,768 636* 1488* d = 65,536 733 d = 131,072 364* d = 262,144 d = 524,288 n = 10,000,000 neurons d = 128 d = 256 d = 512 d = 1024 35,693 99,248 d = 2048 16,451 46,807 115,310 d = 4096 22,307 55,487 126,970 d = 8192 10,619 26,877 61,909 d = 16,384 5069 13,005 30,302 d = 32,768 6307 14,815 d = 65,536 7258 d = 131,072 3562 d = 262,144 d = 524,288 d = 1,048,576

2809 1372* 678* 339*

2922 1438* 716*

1488*

28,027 13,650 6688 3290 1625* 807* 406*

29,122 14,265 7028 3477 1728* 865

29,899 14,706 7274 3614 1806

7454 3717*

66,577 32,814 16,155 7967 3936

69,939 34,635 17,131 8487

72,347 35,946 17,838 8867

36,887 18,350

Notes: Equation 2.1 is accurate to ratio 10−2 (but only 10−1 if marked by *) and equation 2.4 to 10−6 . Equations 2.2 , 2.3 , 2.5 , and 2.6 are accurate to 10−6 . These accuracies are achieved even if the noise rates for equations 2.2 , 2.3 , 2.5 , and 2.6 are 10−4 , 10−4 , 10−4 , and 10−5 , respectively.

n = 100,000,000 neurons d = 128 d = 256 d = 512 d = 1024 356,186 d = 2048 164,349 d = 4096 d = 8192 d = 16,384 d = 32,768 d = 65,536 d = 131,072 d = 262,144 d = 524,288 d = 1,048,576

k=8

991,770 469,174 222,765 106,086 50,640

k = 16

1,152,860 555,466 268,344 129,914 63,004

k = 32

1,270,184 619,272 302,491 147,976 72,484 35,546

k = 64

665,623 327,502 161,315 79,539 39,248

k = 128

698,966 345,606 171,018 84,678

k = 256

722,820 358,615 178,029 88,413

k = 512

367,903 183,035

k = 1024

Table 4: The Value of the Replication Factor r That Is the Closest Integer Solution to Equation 2.1 for the Given Values of the Neuron Count n, the Degree d, and the Inverse Synaptic Strength k, Where km = 2k and ka = k.

Memorization and Association on a Realistic Neural Model 545

k=8

9,917,674 4,691,653 2,227,722 1,060,910 506,456

k = 16

11,528,462 5,554,651 2,683,448 1,299,108 630,010

k = 32

12,701,863 6,192,598 3,024,779 1,479,667 724,718 355,323

k = 64

6,656,114 3,274,935 1,613,041 795,184 392,293

k = 128

6,989,595 3,455,971 1,710,073 846,699

k = 256

7,228,103 3,585,972 1,780,004 883,950

k = 512

3,678,864 1,830,101

k = 1024

Notes: Equation 2.1 is accurate to ratio 10−2 , and equation 2.4 to 10−6 . Equations 2.2 , 2.3 , 2.5 , and 2.6 are accurate to 10−6 . These accuracies are achieved even if the noise rates for equations 2.2 , 2.3 , 2.5 , and 2.6 are 10−4 , 10−4 , 10−4 , and 10−5 , respectively.

n = 1,000,000,000 neurons d = 128 d = 256 d = 512 d = 1024 3,561,948 d = 2048 1,643,398 d = 4096 d = 8192 d = 16,384 d = 32,768 d = 65,536 d = 131,072 d = 262,144 d = 524,288 d = 1,048,576

Table 4: Continued.

546 L. Valiant

Memorization and Association on a Realistic Neural Model

547

little smaller in these tables than in Tables 1 and 2. With regard to equation 2.6, our findings, which are not detailed here, are as follows. While the parameters of Tables 1 and 2 support maximum values of t = 1 usually, the smaller values of r in Tables 3 and 4 (where equation 2.6 is only an approximation) lead to rather larger values of t, scattered in the range 1 to 7. Further, it turns out that instead of using km = 2ka , as we do in these tables, we can find similar results for slightly smaller coefficients, such as km = 1.95ka , or km = 1.9ka , and these give even smaller values of r and larger values of t. 4 The Computational Model and Algorithms As explained earlier, our goal is not only to show that the connectivity of the networks we consider are sufficient to provide the minimum communication bandwidth needed for realizing memorization and association, but also to show that algorithms are possible that modify the network so as to be able to execute instances of these tasks. In particular, each of these two tasks and for each representation, one needs two algorithms—one for creating the circuit, say for associating A to B in the first place, and one for subsequently executing the task, namely, causing B to fire when A fires. For describing such algorithms, we need a model of computation. We employ the neuroidal model because it is programmable and well suited to describing algorithms (Valiant, 1994). As mentioned earlier, the neuroidal model is designed to be so simple that there is no debate that real neurons have at least as much power. It is not designed to capture all the features of real neurons. Our algorithms are described for a variant of the neuroidal model that allows synapses to have memory in addition to weights. This has some biological support (Markram & Tsodyks, 1996b) and allows for somewhat more natural programming, even though temporary values of synaptic weights may be used instead, in principle, to simulate such states (Valiant, 1994). A brief summary of the model is as follows. A neuroidal net consists of a weighted directed graph G with a model neuron or neuroid at each node. A neuroid is a threshold element with some additional internal memory, which can be in one of a set of modes. The mode si of node i at an instant will specify the threshold Ti , and may also have further components such as a member qi of a finite set of states Q. In particular, a mode is either firing or nonfiring and fi has value 1 or 0 accordingly. The weight of an edge from node j to node i is wji and models the strength of a synapse for which j is presynaptic and i is postsynaptic. Each synapse can also have a state qji , which with wji forms a component of the mode sji . The only way a neuroid i can be influenced by other neuroids is through the quantity wi which equals the sum of the weights wji over all nodes j presynaptic to i that are in firing modes. Each neuroid executes an algorithm that is local to itself and can be formally defined in terms of mode update functions δ and λ, for the neuroid

548

L. Valiant

itself and each synapse, respectively: δ(si , wi ) = si , and λ(si , wi , sji , fj ) = sji . These relations express the values of the modes of the neuroid and synapses at one step in terms of the values at the previous step of the variables on which they are permitted to depend. Thus, the mode of a neuroid can depend only on its mode at the previous step and on the sum wi of weights of synapses incoming from firing nodes. The mode of one of its synapses can depend only on the mode of the same synapse, on the mode of the neuroid, on the firing status of the presynaptic neuroid, and on the sum wi of weights of synapses incoming from firing presynaptic neuroids. The model assumes a timing mechanism that has two components. Each transition has a period. We assume here that all transitions have a period of 1, except for threshold transitions, those that are triggered by wi ≥ Ti , which work on a faster timescale. There is a global synchronization mechanism such that, for example, if some external input is to cause the representations of two items A and B to fire simultaneously, then the nodes partaking in these representations will be caused to fire synchronously enough that the algorithms that will be caused to execute can keep in lockstep for the duration of these local algorithms. These durations will be typically no more than 10, and for the purposes of this article, just two steps. By a disjoint representation, we mean a representation where each neuroid can represent at most one item, though one item may be represented by many neuroids. The specific disjoint representation that our algorithms support has been called a positive representation (Valiant, 1994). The generalization that allows a node to represent more than one item, as needed for the shared representations of the next section, we call a positive shared representation. The neuroidal model allows for negative weights, which may be needed, for example, for inductive learning. However, the algorithms we describe here for JOIN and LINK do not use negative weights. We shall start with the algorithms needed for the disjoint two-step scheme implied by relations 2.1 to 2.6. The algorithms for implementing JOIN and LINK are very similar to those that were described for the same tasks for unit weights (Valiant, 1994, algorithms 7.2 and 8.1). It is clear, however, that once graph-theoretic properties such as equations 2.1 to 2.6 are guaranteed, then a rich variety of variants of these algorithms also suffices. We shall describe these algorithms informally here. The following algorithm for creating JOIN needs the nodes of A and B to be caused to fire at distinct time steps. The nodes that are candidates for C = A∧B (1) are initially in “unallocated” state q1, (2) have a fixed threshold T, (3) have each synapse in initial state qq1, and (4) have all the presynaptic weights initially at the value T/k.

Memorization and Association on a Realistic Neural Model

549

The algorithm acting locally on each candidate C node will act over two steps. The first step is prompted by the firing of A and the second by the firing of B one time unit later. Following these two prompts, each candidate C node initially in state q1 that has at least k connections from A and also at least k connections from B will be in state q2, indicating that it has become a C node and assigned to store something. Incoming weights from nodes other than A or B will be made zero. An incoming weight from A will equal T/x if there are x ≥ k of them, and those from B will equal T/y if there are y ≥ k of them. A candidate node that does not receive two successive prompts will return to the initial unassigned condition. The algorithmic mechanism that realizes this outcome is the following. First, note that a node that does become a C node needs to make the incoming synapses have one of the three weight values depending on whether it comes from A or B or neither. The trick is that after the A prompt, the synapses from A will memorize the value T/x for an x ≥ k as its weight, and memorize the fact that it is in this transitory condition by having the synapse state have the temporary value qq2. Also, the node will memorize the fact that it is in a transitory state in which k connections from A have been found by going to state q3. At the B prompt, if at a candidate node in state q3 some y ≥ k synapses come from firing nodes, then these synapses can be updated to have value T/y. At the same time, the A synapses, in state qq2, can go on to take on the values T/x, and the remaining synapses the value 0. However, if no such k connections from B nodes are found at this second step, then the whole neuroid returns to its initial condition. The reader can verify that the circuit constructed as described can execute the created conjunction using a very similar two-step process if at any later time A and B are presented at successive time instants. However, many variations are possible. For example, if the weights are set to T/2x and T/2y instead of T/x and T/y, then simultaneous presentation of A and B will work for recognition. Thus, one-step execution is possible even with twostep creation. We observe here that in general, there is a chance that the set of neuroids identified to represent a conjunction are mostly previously taken, and the new ones that can be assigned form only a small fraction of r. Condition 2.2 ensures that this effect is initially limited. The situation here is akin to that of a hashing scheme in which as the memory fills up, fewer and fewer new places are available (Valiant, 1994). We now go on to discuss an algorithm for creating LINK. We consider A to be represented in one area from which there are directed edges to an area of relay neuroids, from which in turn there are directed edges to a third area containing B. Initially, the relay neuroids have threshold T, and all weights on incoming edges weight T/k, and on outgoing edges weight 0. The relay neuroids never change, except for firing. The neuroids in B are in state q1 initially.

550

L. Valiant

The algorithm for creating LINK has one step in which the representations of A and B are caused to fire at the first prompt. For the nodes in B that are in state q1 and are caused to fire, each incoming edge from a firing node is given weight T/k. The algorithm for executing LINK is even simpler. It requires simple threshold firing at both the relay level and in the B nodes. We now go on to discuss the shared representation expressed by relations 2.1 to 2.6 . The algorithm given for creating and executing LINK in the disjoint case described above applies unchanged to the shared case also. For JOIN, the creation algorithm given has to be modified so that no distinction is made any more between allocated and unallocated nodes. Now no creation process is necessary. Execution can be realized by the following modification of the creation algorithm for the disjoint case: (1) the final state is made the same as the initial state q1, rather than a new state q2, and (2) no synaptic weights are changed at all. The evaluation algorithm is unchanged. The descriptions of the two algorithms above assume bipartite graphs, in which A and B will be in different areas for the case of LINK and C in a different area from A and B in the case of JOIN. To adapt the algorithms to general graphs, small modifications are needed to allow for the node sets having nonzero intersections. For the shared representation one-step algorithm, there is again no creation process. Evaluation requires again only threshold firing. For LINK, there is no difference between the one-step and two-step cases. 5 Capacity and Interference The intention of our representations is that when all or most of the nodes representing an item fire, then the item is considered recognized. For example, the activation of sufficiently many neurons in a representation in a motor area of cortex would cause a certain muscle movement. This style of representation gives rise to a pair of related concerns: How many items can be represented in the system? What exactly does it mean for an item to be represented if unforeseen interference from other activities in the circuit can occur? First, we note that these concerns occur in a fundamentally novel way in our approach as compared with some previous theories. In a traditional associative memory, for example, there is just one kind of execution, retrieval, and we can assume that nothing else is going on simultaneously with an instance of it. Hence, the notion of capacity, the number of items that can be stored, is analyzable in a clean manner (Graham & Willshaw, 1997). In this letter, we have two kinds of tasks, memorization and association. We also have diverse environments arising from different histories of past circuit creations. Further, we may want some robustness to other simultaneous activities in the circuit, and our longer-term aim may be to support further tasks. These factors make possible a large number of potential sources of interference, which we define to be the effect on the execution

Memorization and Association on a Realistic Neural Model

551

of an algorithm of network conditions that arise from sources not specific to that execution. Our guarantee that the circuit acts correctly is only with respect to some specified set of noninterference conditions such as relations 2.2, 2.3, 2.5, or 2.6 and a robustness condition that, as described in the section to follow, upper-bounds the total number of nodes that are active in the whole circuit. No guarantees are implied for situations that are outside these constraints. For example, if the circuit has a “seizure” so that half of the nodes extraneous to the task at hand fire, then this is a pathological condition for which no guarantees are offered. A second observation is that our guarantees are only probabilistic and, at least in this article, only as computed by some numerical calculations of limited precision. In particular, in order to bound the probability of error in the various relations, we have computed the tails of the Bernouilli distribution B (n, p, k) by adding the individual nonnegligible terms T (n, p, i), using double-precision calculations. Since the number of terms grows with n, our accuracy was limited to about 10−6 for the largest values of n that we considered. In principle, the calculations may be performed to arbitrary accuracy in the following sense. One could compute for what minimum integer x is the probability of error less than 10−x for each relation. Doing such detailed calculations is beyond the scope of this article. It will suffice here to observe that certainly in some cases within the tables of parameters that we consider, the errors are much smaller than the claimed 10−6 , in fact, less than 10−1000 in one extremity. As an example, we give a simple analytic upper bound on the error for relation 2.2 :

B (n, B (3r/2, p, km ), r/10) ∼ 0 under the assumption 2.1 , nB (2r − r , p, km ) ∼ r where, further, km = 2k. Now, for generic variables n, k, p, and b, if k = (1 + b)np and 0 ≤ b ≤ 1, then

B (n, p, k) ≤ exp(−b2 np/3) can be derived from Chernoff’s bound (Angluin & Valiant, 1979), where “exp” denotes exponentiation to the base e = 2.71 . . .. From this bound, it follows that

B (3r/2, p, 2k) ≤ exp(−(2k/(3rp/2) − 1)2 (3rp/2)/3). Now in all cases in the tables, rp < k (a fact that also follows from equation 2.1 if we assume r to be negligible and r/n to be small enough to ensure that km = 2k exceeds the mean). It then follows by substitution that B (3r/2, p, 2k) ≤ exp(−rp/18). So if we choose values from the tables with n = 109 , rp ≥ 180, and r ≥ 107 , say, then equation 2.2 becomes

552

L. Valiant

B (109 , exp(−10), 106 ) ∼ 0. Applying the general bound on B given above a second time gives that the error in equation 2.2 is at most B (109 , 10−4 , 106 ) ≤ B (109 , 10−4 , 2 ∗ 105 ) ≤ exp(−105 /3) ≤ 10−1000 . Thus, at one extremity of our range of parameters, the errors for the one relation 2.2 are indeed extremely small. Our point is that while for conceptually simpler models, the notion of capacity, a single value for the number of items that can be represented, makes sense, for more complex models a more appropriate way of expressing the same notion is that of upper-bounding the probability of various kind of interference in any execution of the task. For example, relation 2.2 does not give an absolute guarantee of the relevant interference’s not happening. It says that if A and B have total size 3r/2 and the graph is regarded as randomly generated with respect to those sets, then the probability of the unwanted interference is very small (e.g., 10−6 or 10−1000 ). This we interpret to say, roughly, in the fixed network, that if A is fixed and of size r and B is a random set of r/2 nodes, then the interference effect will occur with such small probability. 6 Robustness to Noise The discussion on capacity referred to specific interactions in the network. A more generic source of interference that can be analyzed is that due to some fixed fraction of neurons being active in the network that are extraneous to the task being executed. The main question is the fraction of extraneous neurons that can be active without interfering with the intended effects of the task at hand. We shall call that fraction the noise rate σ . In general, we can refine each of our noninterference constraints to allow for the expected number s = σ n of extraneous nodes being additionally active. We have restricted the entries in Tables 1 and 2 to those where noninterference relations 2.2, 2.3, 2.5, 2.6, 2.2 , 2.3 , and 2.6 , held even with perturbations corresponding to noise rates σ in the range from 10−4 to 10−6 . In all seven relations, we replaced the relevant quantities r or r/2, when they referred to the input neurons of the task, by r+σ n or r/2+σ n, as appropriate. In particular, for the respective relations, the replacements were done for the following neuron sets: equations 2.2 and 2.2 : A, B; equations 2.3 and 2.3 : A, B, D; equation 2.5: A; and equations 2.6 and 2.6 : A1 , B. In Tables 3 and 4, the entries are restricted in an exactly analogous way. We note that with the exception of equations 2.3, 2.3 , and 2.3 , it is clear that with a lower noise rate, it is easier to satisfy these equations. Estimates of the noise rates that can be tolerated can be made in a number of other senses also. For example, we could assume that all situations, including both circuit creations and executions, are subject to some noise rate, and solve equation 2.1 under that assumption. We also note that we have sought noise rates that can be supported by a very wide range of the parameters. Higher rates can be tolerated for individual parameter combinations.

Memorization and Association on a Realistic Neural Model

553

7 Predictions The entries in our tables are all solutions to equations 2.1, 2.1 , or 2.1 . Remarkably, there is the following simple interdependence among r, d, k, and n: rd < kn < c2 rd,

(∗)

where for each entry in Tables 1 to 4, c2 is a modest constant. In fact, for all entries in the tables with k ≥ 64, it is the case that c2 is smaller than 1.35. For entries with k = 32, k = 16, and k = 8, it is smaller than 1.6, 2.1, and 3.0, respectively. This relationship can be explained as follows. Equation 2.1 is of the form f(B(r, p, k)) = r/n, where f is a fixed function, here the squaring function. For const k, the expectation rp of the associated binomial distribution will stay a constant as r goes up by factor of 10 and p goes down by factor of 10, or, equivalently, as n goes up by a factor of 10 for constant d. (Note that this explains why the corresponding entries in the tables for the various values of n are approximately in the ratio of the magnitudes of n.) Since, in general, r/n is small, solutions of equation 2.1 will correspond to having combinations of r, p, k that correspond to a point somewhat above the expectation. In other words, k will be somewhat above rp = rd/n. Since the binomial distribution falls off exponentially above, the mean k will not be much larger than rn/d, from which we deduce that kd is a little larger than rn. It is easy to see that the same argument also holds for relation 2.1 if km = 2k. This simple relation can be taken as a prediction for systems that allocate memory in the style of our memorization mechanism, provided the number of representatives for a concept at the lower level, that is, A and B, is the same as at the next level, C. This is an attractive assumption for a memory system that treats all memorized concepts as “equal citizens.” It may not be true for all systems. For example, in various levels of a vision or other sensory system, there may be amplification or reduction in the number of neurons that represent an item between the various levels, and in that case appropriate modifications of equations 2.1 or 2.1 need to be solved instead. Finally, we note that a node in our formalism may be simulating a unit that consists of more than one biological neuron. For example, the suggestion that local connections between different layers in cortex may have the effect of increasing the effective degree of a node in the long-range connection network is analyzed in detail in Valiant (1994). 8 Discussion We have shown that networks of model neurons having the four parameters of neuron count, synapse count, strength of synapses, and switching time all within ranges widely observed in biology, can realize the two basic tasks of memory formation and association. We have given tables of values for

554

L. Valiant

these parameters that are consistent with the quantitative constraints that we have identified as being sufficient for the realization of these two tasks. Our positive result is that entries exist for realistic combinations of these numerical parameters. Further, the algorithms needed for creating and executing the circuits for these tasks are of the simplest kind, requiring as little as one step of vicinal or neighborly interaction. The two basic tasks that we have considered here have been the basis for implementing a broader variety of cognitive tasks, including memorization of conjunctions and disjunctions, handling relations, and inductive learning, under the direct-action hypothesis of strong synapses (Valiant, 1994). For the less restrictive setting of the current article, any such broader implications that may follow have yet to be worked out. In particular, if items have a shared representation and these are to be the targets of inductive learning, then special challenges arise. If multiple concepts are being learned and the examples for them are intermingled in time, then having a single synapse take part in the learning of more than one concept would appear to be problematic. This work offers apparently the first explanation of how the basic cognitive tasks that we consider here can be performed at all by neural systems that have synaptic strengths that are as weak as those that are typically observed experimentally. It is probable that different neural systems exploit different combinations of the numerical parameters, and do so in different ways. It is possible, and even probable, for example, that higher-order cognitive tasks require disjoint representations and stronger synapses. Our methodology offers a calculus for investigating such phenomena. Acknowledgments This work was supported in part by grants from the National Science Foundation: NSF-CCR-98-77049, NSF-CCR-03-10882, NSF-CCF-0432037, and NSF-CCF-04-0427129. I am also grateful to two anonymous referees for their helpful comments. References Abeles, M. (1991). Corticonics: Neural circuits of the cerebral cortex. Cambridge: Cambridge University Press. Abramowitz, M., & Stegun, I. (1964). Handbook of mathematical functions. Cambridge, MA: National Bureau of Standards. Ali, A. B., Deuchars, J., Pawelzik, H., & Thomson, A. M. (1998). CA1 pyramidal to basket and bistratified cell EPSPs: Dual intracellular recordings in rat hippocampal slices. J. Physiol., 507, 201–217. Angluin, D., & Valiant, L. G. (1979). Fast probabalistic algorithms for Hamiltonian circuits and matchings. J. Comput. and Syst. Sciences, 8, 155–193. Barlow, H. B. (1972). Single units and sensation: A neuron doctrine for perceptual psychology. Perception, 1, 371–394.

Memorization and Association on a Realistic Neural Model

555

Beurle, R. L. (1955). Properties of a mass of cells capable of regenerating impulses. Philos. Trans. R. Soc. Lond. [Biol.], 240, 55–87. Bollobas, B. (2001). Random graphs. Cambridge: Cambridge University Press. Braitenberg, V. (1978). Cell assemblies in the cerebral cortex. In R. Heim & G. Palm (Eds.), Theoretical approaches to complex systems (pp. 171–188). Braitenberg, V., & Schuz, A. (1998). Cortex: Statistics and geometry of neuronal connectivity. Berlin: Springer-Verlag. Cook, S. A. (1971). The complexity of theorem proving procedures. In Proc. 3rd ACM Symp. on Theory of Computing (pp. 151–158). New York: ACM Press. Feldman, J. A. (1982). Dynamic connections in neural networks. Biol. Cybern., 46, 27–39. Feldman, J. A., & Ballard, D. H. (1982). Connectionist models and their properties. Cog. Sci., 6, 205–254. Gerbessiotis, A. V. (2003) Random graphs in a neural computation model. International Journal of Computer Mathematics, 80, 689–707. Graham, B., & Willshaw, D. (1997). Capacity and information efficiency of the associative net. Network: Comput. Neural Syst., 8, 35–54. Griffith, J. S. (1963). On the stability of brain-like structures. Biophys. J., 3, 299–308. Griffith, J. S. (1971). Mathematical neurobiology: An introduction to the mathematics of the nervous system. New York: Academic Press. Markram, H., & Tsodyks, M. (1996a). Redistribution of synaptic efficiency: A mechanism to generate infinite synaptic input diversity from a homogeneous population of neurons without changing the absolute synaptic efficacies. J. Physiol. (Paris), 90, 229–232. Markram, H., & Tsodyks, M. (1996b). Redistribution of synaptic efficacy between pyramidal neurons. Nature, 382, 807–810. Papadimitriou, C. H. (1994). Computational complexity. Reading, MA: AddisonWesley. Shastri, L., & Ajjanagadde, A. (1993). From simple associations to systematic reasoning: A connectionist representation of rules, variables and dynamic bindings using temporal synchrony. Behavioral and Brain Sciences, 16(3), 417– 494. Shastri, L. (2001). A computational model of episodic memory formation in the hippocampal system. Neurocomputing, 38–40: 889–897. Thomson, A. M., Deuchars, J., & West, D. C. (1993). Large, deep layer pyramidpyramid single axon EPSPs in slices of rat motor cortex displayed paired pulse and frequency-dependent depression, mediated presynaptically and self-facilitation, mediated postsynaptically. J. Neurophysiol., 70, 2354–2369. Valiant, L. G. (1988). Functionality in neural nets. In Proc. AAAI-88 (Vol. 2, pp. 629–634). San Mateo, CA: Morgan Kaufmann. Valiant, L. G. (1994) Circuits of the mind. New York: Oxford University Press. Received March 9, 2004; accepted July 6, 2004.

LETTER

Communicated by Bard Ermentrout

Effects of Noisy Drive on Rhythms in Networks of Excitatory and Inhibitory Neurons Christoph Borgers ¨ [email protected] Department of Mathematics, Tufts University, Medford, MA 02155, U.S.A.

Nancy Kopell [email protected] Department of Mathematics and Center for BioDynamics, Boston University, Boston, MA 02215, U.S.A.

Synchronous rhythmic spiking in neuronal networks can be brought about by the interaction between E-cells and Icells (excitatory and inhibitory cells). The I-cells gate and synchronize the E-cells, and the Ecells drive and synchronize the I-cells. We refer to rhythms generated in this way as PING (pyramidal-interneuronal gamma) rhythms. The PING mechanism requires that the drive II to the I-cells be sufficiently low; the rhythm is lost when II gets too large. This can happen in at least two ways. In the first mechanism, the I-cells spike in synchrony, but get ahead of the E-cells, spiking without being prompted by the E-cells. We call this phase walkthrough of the I-cells. In the second mechanism, the I-cells fail to synchronize, and their activity leads to complete suppression of the Ecells. Noisy spiking in the E-cells, generated by noisy external drive, adds excitatory drive to the I-cells and may lead to phase walkthrough. Noisy spiking in the I-cells adds inhibition to the E-cells and may lead to suppression of the E-cells. An analysis of the conditions under which noise leads to phase walkthrough of the I-cells or suppression of the E-cells shows that PING rhythms at frequencies far below the gamma range are robust to noise only if network parameter values are tuned very carefully. Together with an argument explaining why the PING mechanism does not work far above the gamma range in the presence of heterogeneity, this justifies the “G” in “PING.” 1 Introduction The gamma rhythm, a 30 to 80 Hz rhythm in the nervous system, has been associated with early sensory processing (Singer & Gray, 1995; Fries, Neuenschwander, Engel, Goebel, & Singer, 2001; Fries, Roelfsema, Engel, Konig, ¨ & Singer, 1997), attention (Tiitinen et al., 1993; Pulvermuller, ¨ Birbaumer, Lutzenberger, & Mohr, 1997), and memory (Tallon-Baudry, Bertrand, PerNeural Computation 17, 557–608 (2005)

c 2005 Massachusetts Institute of Technology


558

C. Borgers ¨ and N. Kopell

onnet, & Pernier, 1998; Slotnick, Moo, Kraut, Lesser, & Hart, 2002). It has also been associated with the formation of cell assemblies, that is, temporarily synchronous sets of cells that work together in some aspect (Engel, Konig, ¨ Kreiter, Schillen, & Singer, 1992; Singer & Gray, 1995; Singer, 1999; Olufsen, Whittington, Camperi, & Kopell, 2003). However, it is still not completely understood which mechanisms underlie gamma rhythms, which biophysical parameters are important in these mechanisms, and how gamma rhythms participate in the formation of cell assemblies. Several different kinds of gamma rhythms have been observed in vivo, in vitro, and in computational studies. Synchronization may result from chemical synapses between inhibitory cells (Whittington, Traub, & Jefferys, 1995; Traub, Whittington, Colling, Buzs´aki, & Jefferys, 1996; Wang & Buzs´aki, 1996; Whittington, Traub, Kopell, Ermentrout, & Buhl, 2000; Tiesinga, Fellous, Jos´e, & Sejnowski, 2001; Hansel & Mato, 2003). Gamma rhythms generated in this way have been called interneuronal gamma (ING) rhythms (Whittington, et al., 2000) or γ -I rhythms (Tiesinga et al., 2001). The mechanism has also been called the mutual inhibition mechanism by Hansel and Mato (2003). In this letter, we study gamma rhythms resulting from chemical synapses between excitatory (pyramidal) cells and inhibitory cells. The I-cells (inhibitory cells) gate and synchronize the E-cells (excitatory cells), and the E-cells drive and synchronize the I-cells (Whittington et al., 2000; Tiesinga et al., 2001; Hansel & Mato, 2003). Gamma rhythms of this kind have been called pyramidal-interneuronal gamma (PING) rhythms (Whittington et al., 2000) or γ -II rhythms (Tiesinga et al., 2001). The mechanism has also been called the cross-talk mechanism by Hansel and Mato (2003)1 In vitro, PING is produced by tetanic stimulation (Whittington et al., 2000). PING is believed to be associated with the creation of cell assemblies (Whittington et al., 2000; Olufsen et al., 2003). Other mechanisms, not modeled here, are believed to play a role in the generation of at least some gamma rhythms. Electrical coupling between dendrites of interneurons can contribute to enhancing the coherence of rhythms (Tam´as, Buhl, Lorincz, ¨ & Somogyi, 2000; Traub et al., 2001). Electrical coupling between axons of pyramidal cells is believed to play a central role in driving persistent gamma oscillations in the CA3 region of the hippocampus (Traub et al., 2000). Chattering cells, which intrinsically generate bursts of action potentials at gamma frequency, play a role in generating some gamma rhythms (Gray & McCormick, 1996; Traub, Buhl, Golveli, & Whittington, 2003). Unlike these forms of gamma, PING can be captured by models that abstract from much of the biophysical detail. In general, pyramidal cells are capable of producing many ionic currents. However, some currents that are not involved in producing action potentials, such as Ih and IT , are negligible 1

PING is one of two “cross-talk mechanisms” discussed by Hansel and Mato (2003).

Rhythms in the Presence of Noise

559

during PING activity, since the voltage does not become sufficiently low during any part of the cycle. Other currents, such as, slow outward potassium currents, are weakened by the large activation of metabotropic glutamate receptors needed to produce PING (Storm, 1989; Charpak, G¨ahwiler, Do, & Knopfel, ¨ 1990; Whittington et al., 2000). Consequently, during PING, the electrophysiological behavior is well described by standard HodgkinHuxley equations, which can often be well approximated by reduced equations such as the quadratic integrate-and-fire model (Latham, Richmond, Nelson, & Nirenberg, 2000) and the theta model (Ermentrout & Kopell, 1986; Hoppensteadt & Izhikevich, 1997; Gutkin & Ermentrout, 1998). These reductions and relevant parameter regimes are described in section 2. It is easy to describe and understand the PING mechanism in a network consisting of just one E-cell and one I-cell. The E-cell fires and excites the I-cell enough to fire. The I-cell fires and temporarily inhibits the E-cell. This simple mechanism, discussed in more detail in section 3, requires that external excitatory drive to the I-cell be sufficiently weak that the I-cell fires only in response to the E-cell, not on its own. When this condition is violated, phase walkthrough of the I-cell occurs, that is, the I-cell does not wait for input from the E-cell to fire, destroying the regular rhythm.2 The region in parameter space in which phase walkthrough occurs is bounded by a hypersurface that we call the phase walkthrough boundary. It is discussed in detail in section 4. Clearly, phase walkthrough of the I-cell occurs more easily when the E-cell is driven less strongly. The dynamical properties of the rhythm become more subtle when there are large populations of E- and I-cells. The key to the PING rhythm in large networks is the mechanism by which the I-cells synchronize the E-cells. Section 5 gives an explanation of this mechanism. If approximate synchronization is to occur in the presence of heterogeneity, the ratio r=

strength of inhibitory synaptic currents into E-cells strength of excitatory input drive to E-cells

(1.1)

must be sufficiently large (see section 5). (A refined definition of r is given later.) In a two-cell network, phase walkthrough of the I-cells is the only mechanism by which the PING rhythm can be lost as external drives are varied. In larger networks, there is a second important mechanism: suppression of the E-cells by asynchronous activity of the I-cells. (The I-cells are most effective at suppressing the E-cell when they are asynchronous. See appendix A.) In a simulation in which the I-cells are asynchronous initially and receive strong external drive, suppression of the E-cells may occur immediately, preventing the PING mechanism from synchronizing the network. The region in 2 Throughout this letter, we take the word rhythm to denote a regular rhythm. For instance, the firing pattern in the right panel of Figure 4 is not considered a “rhythm.”

560

C. Borgers ¨ and N. Kopell

parameter space in which asynchronous activity of the I-cells is capable of suppressing the E-cells is bounded by a hypersurface that we call the suppression boundary. It is discussed in detail in section 6. It turns out that the quantity r plays the central role here: suppression occurs more easily for larger r. Bistability between suppression and synchrony is analyzed in section 7. Our study of the phase walkthrough and suppression boundaries casts light on the behavior of PING rhythms in the presence of noisy external drive. The rhythm may not be disrupted by occasional out-of-order spiking of E-cells, caused by noisy external drive. However, too much noisy spiking of the E-cells generates too much tonic excitatory drive to the I-cell, resulting in phase walkthrough. This is analyzed in section 8. Since phase walkthrough of the I-cells occurs more easily for weaker external drive to the E-cells, PING rhythms are more easily abolished by noisy activity in the E-cells when external drive to the E-cells is weaker; the analysis in section 8 makes this quantitative. Similarly, the rhythm may not be disrupted by occasional out-of-order spiking of I-cells. However, too much noisy spiking of the I-cells generates too much tonic inhibitory drive to the E-cells, resulting in their suppression. This is analyzed in section 9. Since suppression of the E-cells occurs easily for large values of r, PING rhythms are easily abolished by noisy activity in the I-cells when r is large. The analysis and most of the simulations presented in this article assume homogeneity in network parameter values and all-to-all connectivity. However, we believe that moderate amounts of heterogeneity and significant sparseness in connectivity do not affect the conclusions. Some simulations, including heterogeneity and sparseness, are presented in section 10. To lower the frequency of a PING rhythm, one must decrease the external drive IE to the E-cells or increase r (or both). A decrease in IE makes phase walkthrough of the I-cells caused by noisy spiking of the E-cells more likely; an increase in r makes the rhythm more susceptible to noise in the I-cells. Our results thus imply that slower PING rhythms are less robust to noise. A PING rhythm will be disrupted by very low levels of noise if its period is many times larger than the decay time constant τI of inhibition, that is, if its frequency is far below the gamma range. If one attempts to raise the frequency above the gamma range (i.e., if one tries to decrease its period below τI ) by raising IE , leaving other parameter values unchanged, then r drops, causing the synchronization mechanism to break down, particularly in the presence of heterogeneity. To restore synchronization, one must raise the strength of the I→E synapses. This brings the frequency back into the gamma range. These arguments are presented in section 11. In summary, our results justify the “G” in “PING”: to be robust in the presence of both heterogeneity and noise, a PING rhythm must have a period that is greater than τI , but not many times greater.

Rhythms in the Presence of Noise

561

2 Networks of Theta Neurons 2.1 The Theta Model. In the theta model, a neuron is represented by a point P = (cos θ, sin θ) moving on the unit circle S1 . This is analogous to the Hodgkin-Huxley model, which represents a periodically spiking spaceclamped neuron by a point moving on a limit cycle in a four-dimensional phase space. In the absence of synaptic coupling, the differential equation describing the motion on S1 is dθ 1 − cos θ = + I (1 + cos θ) . dt τ

(2.1)

Here, I should be thought of as an input “current,” measured in radians per unit time. The time constant τ > 0 is needed to make equation 2.1 dimensionally correct. Its meaning will be clarified shortly. For negative I, equation 2.1 has two fixed points, one stable and the other unstable. As I increases, the fixed points approach each other. When I = 0, a saddle node bifurcation occurs: the fixed points collide, and cease to exist for positive I. For a theta neuron, to “spike” means to reach θ = π (modulo 2π), by definition. The transition from I < 0 to I > 0 is the analog of the transition from excitability to spiking in a neuron. If I > 0 and −π ≤ θ1 ≤ θ2 ≤ π , the time it takes for θ to rise from θ1 to θ2 equals


θ2 θ1

dθ = (1 − cos θ)/τ + I(1 + cos θ)



τ I

  tan(θ/2) θ2 . arctan √ τI θ1

(2.2)

Setting θ1 = −π and θ2 = π in this formula, we find that the period is  T=π

τ . I

(2.3)

We denote the time it takes for θ to rise from π/2 to 3π/2 by δ. This should roughly be thought of as the spike duration. Applying formula 2.2 with (θ1 , θ2 ) = (π/2, π) and (θ1 , θ2 ) = (−π, −π/2) and adding the results, we find   τ 1 δ = π − 2 arctan √ . (2.4) I τI For physiological realism, we wish to ensure δ/T  1. By equations 2.3 and 2.4, δ 2 1 = 1 − arctan √ . T π τI

562

C. Borgers ¨ and N. Kopell

Therefore, δ/T  1 means the same as τ I  1. Since arctan(1/ ) = π/2− + O( 2 ) as → 0, equation 2.4 implies δ ≈ 2τ when τ I  1. This reveals the meaning of τ : in the parameter regime of interest to us, τ is approximately half the spike duration. Motivated by this discussion and by the fact that spike durations in real neurons are on the order of milliseconds, we set τ =1 for the remainder of this article, think of time as measured in milliseconds, and always consider input currents I  1. The frequency of a theta neuron is defined to be ν=

1000 . T

(2.5)

Since we think of t as time measured in milliseconds, ν should be thought of as frequency measured in Hz. Neuronal models are called of type I if the transition from excitability to spiking involves a saddle node bifurcation on an invariant circle, and of type II if it involves a subcritical Hopf bifurcation (Hodgkin, 1948; Ermentrout, 1996; Gutkin & Ermentrout, 1998; Rinzel & Ermentrout, 1998; Izhikevich, 2000). Thus, the theta model is a type I neuronal model. It is canonical, in the sense that other type I models can be reduced to it by coordinate transformations (Ermentrout & Kopell, 1986; Hoppensteadt & Izhikevich, 1997). In this letter, the E-cells are always modeled as theta neurons. The neuronal model used for the I-cells is irrelevant for many of our arguments; where it matters, we model the I-cells at theta neurons at well. The theta model can also be derived from the quadratic integrate-and-fire model (Latham et al., 2000)3 . The general equation of this model is dV = a(V − V0 )(V − V1 ) + Q, dt

(2.6)

with a > 0 and V0 < V1 . After scaling and shifting V and scaling t and Q, equation 2.6 becomes dV = 2V(V − 1) + Q. dt

(2.7)

For Q < 1/2, equation 2.7 has two fixed points, one stable and the other unstable. As Q increases, the fixed points approach each other. When Q = 1/2, a saddle node bifurcation occurs: the fixed points collide and cease 3 This derivation was shown to us by Rob Clewley, who learned it from Bard Ermentrout.

A

V

Rhythms in the Presence of Noise

563

1 0 10

20

30

40

50

40

50

B

V

t

1 0 10

20

30 t

Figure 1: Voltage traces for quadratic integrate-and-fire neuron with (A) threshold at 1 and reset at 0 and (B) threshold at ∞ and reset at −∞.

to exist for positive Q > 1/2. We supplement equation 2.7 with the reset condition V(t + 0) = 0 if V(t − 0) = 1 .

(2.8)

Figure 1A shows V as a function of t for Q = 0.51. The change of variables V=

θ 1 1 + tan 2 2 2

(2.9)

transforms equation 2.7 into equation 2.1 with I=

Q − 1/2 . 1/2

(2.10)

Note that I is the relative deviation of Q from the threshold value 1/2. The only difference between the quadratic integrate-and-fire model and the theta model lies in the reset condition. The two models are precisely equivalent if equation 2.8 is replaced by V(t + 0) = −∞ if V(t − 0) = ∞.

(2.11)

564

C. Borgers ¨ and N. Kopell

(Note that V rises from −∞ to ∞ in finite time when Q > 1/2.) Figure 1B illustrates the effect of replacing equation 2.8 by 2.11. 2.2 Synapses. The derivation of the theta neuron from the quadratic integrate-and-fire neuron offers a way of modeling conductance-based synapses in the framework of the theta model.4 One begins by adding a synaptic current to the right-hand side of equation 2.7, dV = −2V(1 − V) + Q + gs(Vrev − V) , dt

(2.12)

where g denotes the maximum conductance, s = s(t) ∈ [0, 1] is a synaptic gating variable (the quantity that the synapse directly acts on), and Vrev is the reversal potential of the synapse. For excitatory synapses, Vrev should be substantially above the threshold voltage. For inhibitory synapses, it should be somewhat below the reset voltage. We use Vrev = 6.5 for excitatory synapses and Vrev = −0.25 for inhibitory synapses throughout this article. Numerical experiments indicate that changes in these values have no qualitative and little quantitative effect on our conclusions. With the change of variables (see equation 2.9), equation 2.12 becomes   dθ = 1 − cos θ + I + (2Vrev − 1)gs(t) (1 + cos θ ) − gs(t) sin θ . dt Thus, our equation for a theta neuron receiving an excitatory synapse (Vrev = 6.5) is   dθ = 1 − cos θ + I + 12gs(t) (1 + cos θ) − gs(t) sin θ , dt and our equation for a theta neuron receiving an inhibitory synapse (Vrev = −0.25) is   dθ 3 = 1 − cos θ + I − gs(t) (1 + cos θ) − gs(t) sin θ . dt 2

(2.13)

For theoretical arguments in this article, we let s(t) = H(t − t0 )e−(t−t0 )/τD ,

(2.14)

where t0 denotes the time of the spike of the presynaptic neuron, H is the Heaviside function, 1 if t > 0 , (2.15) H(t) = 0 if t ≤ 0 , 4

This too was shown to us by Rob Clewley, with attribution to Bard Ermentrout.

Rhythms in the Presence of Noise

565

and τD > 0 is the synaptic decay time constant. In numerical simulations of networks of theta neurons, we use a smooth approximation to equation 2.14, defining s to be a solution of the differential equation, ds 1−s s + e−η(1+cos θ) , =− dt τD τR where θ denotes the dependent variable associated with the presynaptic neuron, η = 5, and τR = 0.1. Thus, s rises rapidly toward 1 when θ ≈ π modulo 2π and decays exponentially with time constant τ otherwise. 2.3 Networks of Theta Neurons. The numerical simulations presented in this article are for networks of excitatory and inhibitory theta neurons. Unless stated otherwise, the connectivity is all-to-all, and there is no heterogeneity in network properties. The following notation will be used throughout. NE NI τE τI IE II gIE

= = = = = = =

gII =

gEI =

gEE =

TE TI TP νE

= = = =

νI = νP = r =

number of E-cells in the network number of I-cells in the network synaptic decay time constant for excitatory synapses synaptic decay time constant for inhibitory synapses external drive to E-cells external drive to I-cells sum of all conductances associated with inhibitory synapses acting on a given E-cell (so an individual I → E synapse has strength gIE /NI ) sum of all conductances associated with inhibitory synapses acting on a given I-cell (so an individual I → I synapse has strength gII /NI ) sum of all conductances associated with excitatory synapses acting on a given I-cell (so an individual E → I synapse has strength gEI /NE ) sum of all conductances associated with excitatory synapses acting on a given E-cell (so an individual E → E synapse has strength gEE /NE ) √ intrinsic period of E-cells = π/ IE (see equation 2.3) intrinsic period of I-cells (see section 2.4) period of population rhythm intrinsic frequency of E-cells = 1000/TE (see equation 2.5 and the comment following it) intrinsic frequency of I-cells = 1000/TI frequency of population rhythm = 1000/TP (3/2)gIE /IE

566

C. Borgers ¨ and N. Kopell

The definition of r given here will, for the remainder of the article, replace the informal definition in equation 1.1. The reason for including the factor 3/2 in the definition of r will become apparent in section 3. 2.4 Parameter Choices. Unless otherwise stated, we use τE = 2 and τI = 10 in this letter. This is motivated by the decay time constants of excitatory synapses involving AMPA receptors, approximately 2 ms, and inhibitory synapses involving GABAA receptors, approximately 10 ms. (Recall that we think of t as time in milliseconds.) As discussed in section 2.1, the external drive I to a theta neuron should be  1 for the theta model to be biologically reasonable. We therefore assume that the external drives√IE and II are  1. An external drive I corresponds √ to the period T = π/ I, and therefore to the frequency ν = 1000 I/π . For example, I = 0.1 corresponds to ν ≈ 100, and I = 0.4 corresponds to ν ≈ 200. For the PING rhythms studied in this article, it is important that gIE be at least comparable in size to IE (see section 5). We will often choose gII = gIE , but we will also discuss the effects of choosing much smaller values of gII , or even gII = 0. We choose gEI in such a way that a population spike of the E-cells promptly triggers a population spike of the I-cells but does not trigger multiple spikes. If the I-cells are modeled as quadratic integrate-and-fire neurons, the following argument gives a rough indication of how large gEI should be. Consider a network consisting of a single E-cell and a single Icell. (This is equivalent to a network in which each group of cells, E and I, is fully synchronized.) Recall that Vrev = 6.5 for excitatory synapses. Since V is near 0 except during spikes, we approximate the term gs(Vrev − V) in equation 2.12 by gsVrev . We use the idealized form of s given by equation 2.14. Thus, a spike of the E-cell at time t0 gives rise to injection of the current gEI H(t − t0 )e−t/τE Vrev into the I-cell. Since τE = 2 is small in comparison with the periods of the rhythms of interest to us, we introduce the additional approximation that the charge injection resulting from an excitatory synapse is instantaneous, causing a rise in the membrane potential of the I-cell by V = gEI τE Vrev . The excitatory synapse is sure to cause a nearly instantaneous spike of the I-cell if V = 1, that is, if gEI = 1/(τE Vrev ). Since we use τE = 2 and Vrev = 6.5 throughout, this means gEI = 1/13 ≈ 0.077. However, even V = 0.2 typically leads to a spike in the I-cell after a brief delay, so even gEI = 0.2/(τE Vrev ) = 0.2/13 ≈ 0.015 will suffice for PING rhythms. In the CA1 region of the hippocampus, E→E connections are known to have low density (Knowles & Schwartzkroin, 1981). Also, some types

Rhythms in the Presence of Noise

567

of gamma oscillations are induced by acetylcholine (Fisahn, Pike, Buhl, & Paulsen, 1998), which in turn suppresses recurrent excitatory connections (Hasselmo, 1999). Motivated by these experimental findings and following many previous modeling studies of gamma rhythms (e.g., Whittington, Stanford, Colling, Jefferys, & Traub, 1997), we assume gEE = 0 throughout this article. It would be interesting to study the effects of E→E synapses in the presence of noise, particularly in the presence of noisy spiking of the E-cells; however, this will be deferred to future work. 2.5 The Intrinsic Frequency of the I-Cells. The intrinsic frequency νI of the I-cells is the frequency at which the I-cells would spike if the E→I synapses were removed. In the presence of I→I synapses, νI differs from the frequency of an isolated I-cell and depends on whether the I-cells synchronize. In many of our arguments, we will make no specific assumption about the neuronal model used for the I-cells, and simply consider νI a network parameter. However, we will also apply our general results to the case when the I-cells are modeled as theta neurons. For this, we will need to compute νI . The intrinsic frequency νI is related to the intrinsic period TI by νI = 1000/TI . We will discuss how to compute TI . 2.5.1 Intrinsic Period in the Absence of I→I Synapses. If gII = 0, then π TI = √ II

(2.16)

by equation 2.4. 2.5.2 Intrinsic Period in the Presence of I→I Synapses, Assuming Synchronous Network Activity. Assume now that gII > 0. We model synapses in the idealized way described by equations 2.13 and 2.14. If the I-cells are in perfect synchrony, the period TI is the time that it takes for θ to reach π if θ is governed by   3 dθ −t/τI = 1 − cos θ + II − gII e (1 + cos θ ) − gII e−t/τI sin θ , dt 2 θ (0) = −π . (2.17) This cannot be computed analytically, but it is easy to compute numerically. 2.5.3 Intrinsic Period in the Presence of I→I Synapses, Assuming Asynchronous Network Activity. If the I-cells are in complete asynchrony, the function

568

C. Borgers ¨ and N. Kopell

e−t/τI in equation 2.17 is replaced by its time average. (We neglect fluctuations resulting from the finiteness of the network.) Thus, each I-cell is governed by   dθ 3 = 1 − cos θ + II − gII s (1 + cos θ) − gII s sin θ dt 2

(2.18)

with s=

1 TI




TI

e−t/τI dt =

0

1 − e−TI /τI . TI /τI

(2.19)

If θ obeys equation 2.18, and θ(0) = −π, then θ reaches π at time


π

dθ   −π 1 − cos θ + II − (3/2)gII s (1 + cos θ) − gII s sin θ  π   tan(θ/2)−gII s/2 arctan √  II −(3/2)gII s−(gII s)2 /4   =   I − (3/2)g s − (g s)2 /4  I II II −π

π = . II − (3/2)gII s − (gII s)2 /4

(2.20)

This formula holds if II − (3/2)gII s − (gII s)2 /4 > 0. For II − (3/2)gII s − (gII s)2 /4 ≤ 0, θ = π is never reached. The period TI is therefore determined by TI =

π II − (3/2)gII s − (gII s)2 /4

.

(2.21)

Note that the right-hand side depends on TI , since s does. We cannot solve this equation analytically. However, it has a unique solution because the right-hand side is a strictly decreasing function of TI . This solution can easily be found numerically, for instance, using the bisection method. 2.5.4 An Observation on the Effects of I→I Synapses for Weakly Driven I-Cells. A major focus of this letter is the behavior of PING rhythms for weak external drives. We will show later that PING rhythms tend to become noise sensitive when the external drives become weak, but that the noise sensitivity can be counteracted, to some extent, by introducing I→I synapses. This motivates our interest in the effects of I→I synapses in the limit as II → 0. The main result of the following discussion is equation 2.23, which will play a role in section 7. Under the assumption of perfect synchrony of the I-cells, I→I synapses become irrelevant as II → 0. In this limit, TI → ∞, so the decay time τI

Rhythms in the Presence of Noise

569

of the inhibitory synapses becomes negligible in comparison with TI , and therefore π TI ∼ √ II as II → 0. (The symbol ∼ expresses that the ratio of the two quantities converges to 1.) Under the assumption of complete asynchrony, the I→I synapses remain relevant in the limit as II → 0. As II → 0, TI → ∞ and therefore s∼

τI TI

(2.22)

by equation 2.19. Using equation 2.22 in equation 2.21, we find that as II → 0, the fixed point equation for TI becomes TI =

π I − (3/2)gII τI /TI − (gII τI /TI )2 /4

.

This is equivalent to II TI2 =

3 gII τI TI + C 2

with C = π2 +

(gII τI )2 , 4

so TI ∼

3 gII τI C 3 gII τI + ∼ . 2 II II TI 2 II

(2.23)

Comparing equation 2.23 with 2.16, we see that the presence of I→I synapses makes an important difference in the limit as II → 0. 2.6 The Asynchronous State. In our simulations, we frequently initialize theta neurons in a state of complete asynchrony. We define here what we mean by “complete asynchrony.” Consider a theta neuron with an external drive I > 0. Suppose that θ(0) = θ0 ∈ (−π, π ). The time to the next spike is then
π
π dt 1 dθ = dθ R= θ0 dθ θ0 1 − cos θ + I(1 + cos θ )   1 tan(θ/2) π = √ arctan √ I I θ0   1 π tan(θ0 /2) = √ − arctan . (2.24) √ I 2 I

570

C. Borgers ¨ and N. Kopell

To initialize a population of uncoupled neurons in complete asynchrony, √ we choose R = Uπ/ I with U ∈ (0, 1) random, uniformly distributed. We then compute θ0 from equation 2.24, √  I tan (Vπ/2) , (2.25) θ0 = 2 arctan with V = 1 − 2U uniformly distributed in (−1, 1). 3 PING in a Simple Two-Cell Network In this section, we consider a network of a single E-cell and a single I-cell. The E-cell is modeled as a theta neuron; it is assumed to receive external drive IE > 0. We make no explicit assumption about the neuronal model used for the I-cell, but denote by νI its intrinsic frequency. (If the I-cell is driven below threshold, νI = 0.) We make the following idealizing assumptions: 1. A spike of the E-cell instantaneously triggers a spike of the I-cell but has no effect lasting beyond the spike time. 2. The spike of the I-cell gives inhibitory input to the E-cell in the idealized form described by equation 2.14. 3. The I-cell does not spike again until prompted by the next spike of the E-cell. Assumptions 1 and 3 imply that the I-cells spike exactly once per oscillation cycle. By strengthening the E→I synapses, one can generate PING-like rhythms in which each I-cell fires multiple times on each oscillation cycle. These rhythms differ from those considered in this letter only in some details of minor interest; for instance, the oscillation frequency is reduced if the I-cell population fires several population spikes on each oscillation cycles. We note that spike doublets of the I-cells play a much subtler and more important role when conduction delays are substantial (Ermentrout & Kopell, 1998). However, in this article, we neglect conduction delays. If the two cells spike at t = 0, the E-cell is governed by the initial value problem   dθ 3 −t/τI = 1−cos θ + IE − gIE e (1+cos θ)−gIE e−t/τI sin θ for t dt 2 > 0, (3.1) θ (0) = −π ,

(3.2)

until it spikes again. We denote by TP the time at which θ, governed by equations 3.1 and 3.2, reaches π , and write νP = 1000/TP . Assumption 3 then becomes νI ≤ νP .

(3.3)

Rhythms in the Presence of Noise

571

νP

100 0 0

0 IE 0.1 0.5

gIE

Figure 2: The PING frequency νP as a function of gIE and IE , for τI = 10.

We will now discuss the dependence of the frequency νP on IE , gIE , and τI . Our main conclusion will be that νP depends weakly on IE and gIE but strongly on τI . For τI = 10, Figure 2 shows the graph of νP as a function of IE and gIE . The graph is quite flat in the region in which neither gIE /IE nor IE is small. We will argue in later sections that in those regions in which gIE /IE or IE are small, PING rhythms are not robust. Specifically, one needs gIE /IE > 1 for rapid and robust synchronization in large networks (see section 5), and PING rhythms with small IE are highly susceptible to noise in the E-cells (see section 8). Thus, Figure 2 shows that in the most relevant parameter regime, the dependence of νP on IE and gIE is fairly weak. We will next derive an approximate formula for TP , valid for sufficiently large gIE /IE . This formula will confirm that the dependence of TP on IE and gIE is weak and also show the importance of τI . Defining J = IE −

3 gIE e−t/τI , 2

equations 3.1 and 3.2 can be written as follows: 2 dθ = 1 − cos θ + J(1 + cos θ) − (IE − J) sin θ dt 3 IE − J dJ = for t > 0 , dt τI θ (0) = −π ,

for t > 0 ,

(3.4) (3.5) (3.6)

572

C. Borgers ¨ and N. Kopell

*

J=J

A

J

0 −0.2 −0.4 −π

0 θ

π

q

1

B

0 0.4

20

IE 0 5

τI

Figure 3: (A) Phase portrait for equations 3.4 and 3.5 with IE = 0.1 and τI = 10, with the stable river indicated in bold. (B) q = (IE − J∗ )/IE as a function of IE and τI .

J(0) = IE −

3 gIE . 2

(3.7)

We will study the phase portrait for the two-dimensional dynamical system, equations 3.4 and 3.5. This is very similar to a discussion for inhibitory current pulses (not inhibitory synaptic inputs) that we gave earlier (Borgers ¨ & Kopell, 2003). The phase portrait for equations 3.4 and 3.5 is shown in Figure 3A for IE = 0.1 and τI = 10. The figure should be extended periodically in θ with period 2π. The flow is upward, in the direction of increasing J. The most striking feature of the phase portrait is the existence of strongly attracting and strongly repelling trajectories. Trajectories of this kind are found in many systems of ordinary differential equations and are called rivers

Rhythms in the Presence of Noise

573

(Diener, 1985a, 1985b). The figure reveals a stable river, that is, a trajectory, (θs , Js )

with Js (t) = IE −

3 gIE e−t/τI , 2

(3.8)

that is attracting in forward time. The stable river is indicated as a bold line in Figure 3A. As t → −∞, Js → −∞, and θs → θs0 , where θs0 is the unique solution in (−π, π ) of 1 + cos θ +

2 sin θ = 0 . 3

(It is easy to see that θs0 = −2 arctan(3/2) ≈ −1.966.) We denote by T∗ the time when θs (T∗ ) = π, and define J∗ = Js (T∗ ) ∈ (0, IE ) .

(3.9)

We define r=

(3/2)gIE , IE

(3.10)

and assume r>1,

(3.11)

so J(0) < 0. If J(0) is sufficiently negative, the trajectory (θ (t), J(t)) is rapidly attracted to (θs (t), Js (t)). At the time when θ = π, we therefore have J ≈ J∗ , or t ≈ T∗ . Thus, TP is approximately T∗ . Equations 3.8 and 3.9 imply that T∗ = ln



(3/2)gIE IE − J ∗

 τI .

(3.12)

We write q = q(τI , IE ) =

IE − J ∗ ∈ (0, 1) . IE

(3.13)

Figure 3B shows the graph of q. From equations 3.10, 3.12, and 3.13, T∗ = ln

  r τI . q

(3.14)

Of course, this is not an explicit formula for T∗ , since q depends on IE and τI and is not given by an explicit formula. Equation 3.14 does, however,

0

I−cell

I−cell

E−cell

C. Borgers ¨ and N. Kopell

E−cell

574

100 200 300

0

100 200 300

Figure 4: From PING (left panel) to phase walkthrough (right panel) as a result of raising drive to the I-cell.

explicitly describe the dependence of T∗ on gIE , since the only quantity on the right-hand side of equation 3.14 depending on gIE is r = (3/2)gIE /IE . We have concluded   r TP ≈ ln τI , q

(3.15)

if r is sufficiently large. Numerical experiments show that r = 3 is sufficient for this formula to be quite accurate. Figure 3B shows that the graph of q is fairly flat unless IE is small. In a parameter regime in which q is approximately constant, formula 3.15 shows that TP depends strongly (namely, approximately linearly) on τI , but only weakly (namely, logarithmically) on gIE and IE . 4 The Phase Walkthrough Boundary A necessary condition for the PING mechanism to work is that the I-cells spike only when prompted by the E-cells, not independently. This is trivially true if the drive to the I-cells is subthreshold, that is, νI = 0. If νI > 0, then the PING mechanism works only if the frequency of the rhythm is greater than the intrinsic frequency of the I-cells. Figure 4 illustrates, using a network of a single E-cell and a single I-cell, what happens when this condition is not met. (The figure indicates spike times.) We refer to the phenomenon shown in Figure 4 as phase walkthrough of the I-cells. Figure 5 shows, in various different ways, that phase walkthrough occurs more easily when IE is smaller or gIE is larger, in other words, when the PING frequency is lower. The figure also shows that I→I synapses protect against phase walkthrough. We will now discuss the details of Figure 5.

Rhythms in the Presence of Noise

575

A

B

100

II

νI

0.1

0 0

0

ν

0 0 IE

gIE

E

0 gIE

100 0.5

0.1 0.5

C

D

0.06 II

II

0.1

0 0

0 0

IE

0.2

0 IE

g

IE

0.1 0.5

Figure 5: Phase walkthrough boundary for τI = 10 (A) in (gIE , νE , νI )-space, (B) in (gIE , IE , II )-space for gII = 0, (C) in the (IE , II )-plane, for gII = 0 and gIE = 0.2, (D) in (gIE , IE , II )-space, for gII = gIE .

4.1 The Phase Walkthrough Condition. In the simple two-cell network of section 3, a necessary and sufficient condition for phase walkthrough of the I-cells to be avoided is νI ≤ νP .

(4.1)

The equation νI = νP

(4.2)

(or equivalently TI = TP ) defines a hypersurface in parameter space that we call the phase walkthrough boundary. As discussed in section 3, νP is √ a function of IE , gIE , and τI . Since νE = (1000/π ) IE , we can also think of νP as a function of νE , gIE , and τI . For given νE , gIE , and τI , it is easy to

576

C. Borgers ¨ and N. Kopell

calculate νP numerically with great (ten-digit) accuracy. We first determine the time Tp at which θ, governed by equations 3.1 and 3.2, reaches π , and then compute νP = 1000/Tp . For τI = 10, Figure 5A shows the phase walkthrough boundary in (gIE , νE , νI )-space. If (gIE , νE , νI ) lies below the surface in Figure 5A, a PING rhythm is possible. Above the surface, there is no PING rhythm because of phase walkthrough. Note that slowing the rhythm without reducing drive to the I-cell eventually results in νI > νP , that is, in phase walkthrough. 4.2 I→I Synapses Protect Against Phase Walkthrough. Since νI is a decreasing function of gII , I→I synapses make phase walkthrough of the I-cells less likely to occur. (Recall that by the notational conventions of section 2, the definition of νI takes I→I synapses into account, but νE denotes the frequency of the E-cells in the absence of any synapses.) To make this quantitative, we must assume a specific model for the I-cell. For the remainder of this section, we therefore assume that the I-cell is a theta neuron. We still make assumptions 1 and 2 of section 3, and we also assume that I→I synapses take the idealized form given by equation 2.14. For fixed values of τI and gII , the phase walkthrough boundary can then be thought of as a surface in (gIE , IE , II )-space. For τI = 10 and gII = 0, this surface is plotted in Figure 5B. The surface in Figure 5B is thus identical to that of Figure 5A, except for the choice of coordinates. The parameter νI in Figure 5A is replaced by II in Figure 5B. Equation 2.16 yields the relation between νI and II . (Note that νI = 1000/TI .) Figure 5B shows that in the parameter regimes of interest to us (r > 1, that is, gIE > (2/3)IE )), II must be much smaller than IE for PING to be possible. In Figure 5C, we show a section through the surface of Figure 5B. In addition to fixing τI = 10 and gII = 0 as in Figure 5B, gIE = 0.2 is fixed as well. The curve in Figure 5C is the graph of a function of IE . For later reference, we denote this function by F = F(IE ), suppressing in the notation the dependence on τI , gII , and gIE . Thus, the curve shown in Figure 5C is given by II = F(IE ) .

(4.3)

Phase walkthrough occurs above the curve, but not below it. The surface in Figure 5B changes dramatically when gII is taken to be equal to gIE . It is easy to see that in this case, the walkthrough boundary is given by II = IE

(4.4)

(see Figure 5D). Thus, I→I synapses have a stabilizing effect, greatly enlarging the parameter regime in which PING is possible.

Rhythms in the Presence of Noise

577

5 PING in Networks of More Than Two Cells The key to PING rhythms in networks of more than two cells lies in the fact that a population of uncoupled neurons receiving a single common strong inhibitory input pulse synchronizes. In this section, we give an explanation of this synchronization mechanism and illustrate it with numerical examples. The main conclusion of this section is that rapid and robust synchronization in large networks requires r = (3/2)gIE /IE > 1. In earlier work (Borgers ¨ & Kopell, 2003), we discussed the synchronization of a population of theta neurons by an inhibitory current pulse. We present here a very similar discussion referring to an inhibitory synaptic pulse. A single theta neuron receiving an inhibitory synaptic input at t = 0 is described by equations 3.4 and 3.5, with initial conditions θ (0) = θ0 , and equation 3.7. Synchronization of a population of uncoupled theta neurons by a single inhibitory synaptic pulse can be understood from the phase portrait in Figure 3A. Recall that the trajectory (θs , Js ) indicated as a bold line in Figure 3A, the “stable river,” attracts nearby trajectories. Also recall that T∗ is defined by θs (T∗ ) = π and J∗ by J∗ = Js (T∗ ). Assume, as in section 3, r > 1, so J(0) < 0. If J(0) is sufficiently negative and θ0 is sufficiently far from π, (θ(t), J(t)) is rapidly attracted to the stable river. At the time when θ = π, we therefore have J ≈ J∗ , or t ≈ T∗ . Thus, the first spike after time zero occurs approximately at time T∗ . If θ0 is close to π, then θ(t) quickly passes through π and is then rapidly attracted to (θs (t) + 2π, Js (t)). (Recall that Figure 3A should be thought of as extended periodically in θ with period 2π.) When θ (t) reaches 3π , then J ≈ J∗ and therefore t ≈ T∗ . Thus, in that case, a spike occurs soon after time zero, followed by a spike approximately at time T∗ . Only for values of θ0 in a narrow transition band is (θ (t), J(t)) attracted to neither (θs (t), Js (t)) nor (θs (t) + 2π, Js (t)). A population of E-cells is approximately synchronized by an inhibitory pulse with sufficiently large r because T∗ is independent of θ0 . Figure 6A shows a simulation for a network as described in section 2, gEI = 0.05, gIE = 0.20, gII = 0, IE = 0.1 (νE ≈ 101), II = 0.002. (Other parameter values are as specified in section 2.) The common inhibitory input received by all E-cells leads to their rapid synchronization as a result of the mechanism described earlier. This in turn leads to synchronization of the I-cells, which are driven by the E-cells. Note that here, r=

(3/2)gIE 0.3 =3. = IE 0.1

PING rhythms are possible even for r < 1. For instance, Figure 6B shows results similar to those of Figure 6A, but with gIE = 0.05, corresponding to r = 0.75. A rhythm appears, but only after a number of periods, not

C. Borgers ¨ and N. Kopell

0 40

B

160

I−cells

C

0 40

0 0

100 200 300

160

D

100 200 300

0 40

0 0

100 200 300

E−cells

E−cells

0 0

160

I−cells

I−cells

A

E−cells

160

I−cells

E−cells

578

0 40

0 0

100 200 300

Figure 6: (A) PING with strong inhibitory synapses (gEI = 0.05, gIE = 0.2, gII = 0, IE = 0.1, II = 0.002). (B) PING with weak inhibitory synapses (gIE = 0.05; all other parameter values as in A). (C) As in A, with 5% heterogeneity in IE . (D) As in B, with 5% heterogeneity in IE .

immediately. Thus, when r ≤ 1, PING may still be possible, but it is less attracting and more difficult to analyze. Numerical experiments suggest that PING with r ≤ 1 is also highly sensitive to heterogeneity. As an example, we introduce 5% heterogeneity in IE . That is, we take IE to be a normally distributed random number with mean 0.1 and standard deviation 0.005. Figure 6A turns into Figure 6C—the rhythm is barely affected. However, Figure 6B turns into Figure 6D—the rhythm is destroyed. The effect of heterogeneity in IE is to spread out the spiking of the E-cells. If r is sufficiently large, each population spike of the I-cells approximately erases the memory of the past, bringing the E-cells back together and preventing the effects of heterogeneity from accumulating over time. In our numerical experience, at least r ≈ 3 is required if the rhythm is to survive heterogeneity in network parameter values on the order of 20%.

Rhythms in the Presence of Noise

579

6 The Suppression Boundary As seen in section 4, in a two-cell network (or, equivalently, in a larger network in which the E-cells and the I-cells are perfectly synchronized), PING is possible only if the intrinsic frequency νI of the I-cells is sufficiently small. If the I-cells intrinsically spike too rapidly, phase walkthrough occurs. In a network with many cells, there is another way in which rapid spiking of the I-cells can destroy PING: the I-cells can suppress the E-cells altogether. The I-cells are most effective at suppressing the E-cell when they are asynchronous. For linear integrate-and-fire neurons, this is proved in appendix A. Although we have not proved it for theta neurons, numerical evidence suggests that it is true in that case as well. We therefore assume in this section that the I-cells spike in complete asynchrony and ask under which circumstances the I-cells suppress the E-cells. Figure 7 shows, in various different ways, that suppression of the E-cells occurs more easily when IE is smaller or gIE is larger, in other words, when the PING frequency is lower. In this section, we will discuss the four panels of Figure 7 in detail. We will show that the quantity determining how easily suppression of the E-cells occurs is the ratio gIE /IE , which, up to the constant factor 3/2, is r. 6.1 The Suppression Condition. Consider a large population of I-cells, spiking in asynchrony, acting on a single E-cell driven above threshold. We ask under which conditions spiking in the E-cell will be prevented altogether by the inhibition. We model the synaptic gating variable in the idealized way described by equation 2.14. Because the I-cells are assumed to spike in asynchrony, the term s(t) in equation 2.13 is replaced by its time average. (We neglect fluctuations resulting from the finiteness of the network.) Thus, the equation governing the target neuron is   dθ 3 (6.1) = 1 − cos θ + IE − gIE s (1 + cos θ ) − gIE s sin θ, dt 2 with s defined in equation 2.19. The E-cell escapes suppression if and only if 2 3 1 (6.2) gIE s + gIE s < IE . 2 4 To see this, replace gII by gIE and II by IE in equation 2.20. Since we assume IE  1, this is approximately equivalent to 3 gIE s < IE . 2 Using the definition of r, equation 3.10, this condition becomes s<

1 . r

(6.3)

580

C. Borgers ¨ and N. Kopell

A

B

100

II

νI

0.1

0 0

0

νE

0 0

g

0 I

g

E

IE

IE

100 0.5

0.1 0.5

C

D

0.06 II

I

I

0.1

0 0

0 0

I

E

0.2

0 IE

gIE 0.1 0.5

Figure 7: Suppression boundary for τI = 10, (A) in (gIE , νE , νI )-space, (B) in (gIE , IE , II )-space, for gII = 0, (C) in the (IE , II )-plane for gII = 0 and gIE = 0.2 (dashes), together with the phase walkthrough boundary (solid) and the region of bistability (shaded), (D) in (gIE , IE , II )-space, for gII = gIE .

The equation s=

1 r

(6.4)

defines a hypersurface in parameter space that we call the suppression boundary. Since s depends on only TI /τI = 1000/(νI τI ) (see equation 2.19), the suppression boundary could be drawn as a surface in (gIE , νE , νI τI )space. However, for easier comparison with Figure 5A, we set τI = 10 and plot, in Figure 7A, the suppression boundary as a surface in (gIE , νE , νI )space. If (gIE , νE , νI ) lies below the surface in Figure 7A, suppression of the E-cells by asynchronous activity of the I-cells is impossible. Above the surface, suppression occurs if the I-cells spike asynchronously.

Rhythms in the Presence of Noise

581

Asynchronous activity of the I-cells can simply be the result of initial conditions, in the absence of any mechanism synchronizing the I-cells. (The E-cells cannot synchronize the I-cells if they are suppressed.) It can also be the result of stochastic external inputs to the I-cells. In that case, each I-cell spikes in an irregular fashion. However, if TI is taken to be the average interspike interval, then equation 6.3 is still the condition under which the E-cells escape suppression. This will be discussed further in section 9. Figure 7A shows that in a large portion of parameter space (the portion where the surface is flat), PING rhythms are highly susceptible to suppression of the E-cells as a result of asynchronous activity of the I-cells. (Recall that suppression occurs above the surface shown in Figure 7A.) In this portion of parameter space, PING rhythms are easily abolished by noisy external drive to the I-cells (see section 9). Equation 6.3 shows that r is the quantity that matters here: the larger r, the more severe is condition 6.3. 6.2 Slow PING Rhythms Are Typically Susceptible to Suppression of the E-Cells. If a PING rhythm is slowed by lowering νE , that is, lowering IE , with all other parameter values fixed, then r grows, and therefore the rhythm becomes more susceptible to suppression of the E-cells by the I-cells. Similarly, if a PING rhythm is slowed by raising gIE , with all other parameter values fixed, then r grows, and the rhythm again becomes increasingly susceptible to suppression of the E-cells. 6.3 Suppression of the E-cells Can Be Avoided Even at Low Frequencies by Careful Parameter Tuning. Slow PING rhythms are not impossible. If IE → 0 and gIE → 0 in such a way that r remains fixed, then νP → 0. This can be seen from Figure 2 or from equation 3.15 in conjunction with Figure 3B (which shows that q decreases as IE decreases and q → 0 as IE → 0). However, the right-hand side of equation 6.3 remains unchanged in this limit. Thus, a PING rhythm with small IE and proportionally small gIE is slow, but not susceptible to suppression of the E-cells by the I-cells. 6.4 I→I Synapses Protect Against Suppression. I→I synapses make it harder for the I-cells to suppress the E-cells for two reasons. First, they often destabilize asynchrony of the I-cells. However, even if the I-cells were to remain completely asynchronous, I→I synapses would make it harder for the I-cells to suppress the E-cells, by lowering νI . To illustrate the latter point, we now assume that the I-cells are theta neurons. As before, we assume that I→I synapses take the idealized form given by equation 2.14. For fixed τI and gII , we draw the suppression boundary as a surface in (gIE , IE , II )-space. For τI = 10 and gII = 0, this is shown in Figure 7B. The dashed curve in Figure 7C shows a section through the surface of Figure 7B. In addition to fixing τI = 10, gII = 0, gIE = 0.2 is fixed as well. This curve is the graph of a function of IE . For later reference, we denote this function by G = G(IE ), suppressing in the notation the dependence on

582

C. Borgers ¨ and N. Kopell

τI , gIE , and gII . Thus, the dashed curve in Figure 7C is given by II = G(IE ) .

(6.5)

The phase walkthrough boundary for the same parameter values was plotted in Figure 5C; for comparison, it is reproduced in Figure 7C (solid curve). The two curves in Figure 7C intersect each other, enclosing a region, shaded in Figure 7C, that lies above the suppression boundary (so suppression of the E-cells by asynchronous activity of the I-cells is possible), but below the phase walkthrough boundary (so PING is possible as well). This is a region of bistability; it will be discussed further in section 7. Figure 7B changes dramatically when gII is taken to be equal to gIE (see Figure 7D). The region in phase space in which suppression of the E-cells by the I-cells is impossible (the region below the surface) is greatly enlarged by the I→I synapses. As will be shown in the next section, there is no region of bistability in this case. 7 A Region of Bistability We observed in section 6 that Figure 7C shows a region of bistability in parameter space, a region in which both asynchronous activity of the I-cells with suppression of the E-cells and PING are possible. Which of these two states occurs depends on initial conditions. This observation is not centrally important here, but it is an interesting example demonstrating that PING rhythms can be locally but not globally attracting network states. In this section, we give analytic arguments demonstrating the existence of the region of bistability (or, to be more precise, demonstrating that the two curves in Figure 7C must intersect in a point other than the origin). We also give a numerical example illustrating the bistability. The solid curve in Figure 7C, the phase walkthrough boundary, is the graph of a function. Recall from section 4 that we denote this function by F, so the curve in Figure 7C is given by II = F(IE ). We shall first show F (0) = 1

(7.1)

and lim F(IE ) = ∞ .

IE →∞

(7.2)

To prove equation 7.1, note that TP → ∞ as IE → 0. Thus, the (fixed) decay time τI becomes negligible in comparison with TP . This √ implies that the√inhibitory synapses become negligible, so TP ∼ TE = π/ IE , and TI ∼ π/ II . The phase walkthrough boundary √ is generally √ given by TP = TI . In the limit as IE → 0, this means π/ IE ∼ π/ II , or IE ∼ II , that is, equation 7.1.

Rhythms in the Presence of Noise

583

Figures 5B and 5C appear to suggest that F (0) is positive, but much smaller than 1, in contradiction to the result just derived. The explanation is that IE has to be extremely close to 0 for F (IE ) to come close to 1. A numerically computed blow-up of Figure 5C near the origin indeed confirms that F (0) = 1. To prove equation 7.2, we first note that F increases as gII increases: the greater gII , the more drive II to the I-cells is needed for phase walkthrough to occur. Therefore, equation 7.2 follows for gII > 0 if it can be shown for gII = 0. To prove equation 7.2 for gII = 0, we note that in the limit IE → ∞, the I→E synapses become negligible as well, but for a different reason: the terms, including the factor IE in the equation of the E-cell, dominate all others, in particular, the terms modeling the synapse. As a result, TP ∼ TE as IE → ∞. The phase walkthrough boundary is generally given by TP = √ TI . In the √limit as IE → ∞, this means TE ∼ TI . For gII = 0, this means π/ IE ∼ π/ II , or IE ∼ II . This implies equation 7.2. The dashed curve in Figure 7C, the suppression boundary, is the graph of a function as well. Recall from section 6 that we denote this function by G, so the dashed curve in Figure 7C is given by II = G(IE ). We next show G (0) =

gII gIE

(7.3)

and lim

IE →(3/2)gIE

G(IE ) = ∞ .

(7.4)

To show equation 7.3, note that as IE → 0, II = G(IE ) → 0; therefore, TI → ∞, and τI s∼ TI by equation 2.19. Thus, the suppression boundary becomes TI ∼r τI

(7.5)

by equation 6.4. Assuming gII > 0, using equation 2.23, this becomes (3/2)gIE (3/2)gII ∼ , II IE or II ∼

gII IE , gIE

that is, equation 7.3. If gII = 0, we use equation 2.16 to turn equation 7.5 into π (3/2)gIE √ ∼ IE II

584

C. Borgers ¨ and N. Kopell

or II ∼ π 2

IE2 (9/4)g2IE

.

Thus, equation 7.3 holds for gII = 0 as well. As IE → (3/2)gIE , r → 1. The suppression boundary is given by 1 1 − e−TI /τI = TI /τI r

(7.6)

(see equations 6.4 and 2.19). The left-hand side of equation 7.6 can easily be shown to be a strictly decreasing function of TI /τI that tends to 1 as TI /τI → 0. Thus, as r → 1, TI /τI → 0, so II → ∞. This proves equation 7.4. Equations 7.1 to 7.4, taken together, imply that the suppression and phase walkthrough boundaries in the (IE , II )-plane must intersect (in a point other than the origin) as long as gII < gIE . We present a numerical example illustrating the bistability that our theoretical arguments predict. We use gIE = 0.2, gEI = 0.05, gII = 0 and (II , IE ) = (0.0025, 0.03) , a point inside the region of bistability. We initialize the I-cells asynchronously (see section 2.6). If we initialize all E-cells at θ = −π/2, complete suppression of the E-cells results, as shown in Figure 8A. On the other hand, if we initialize the E-cells at θ = π/2, they start out so close to spiking that they are able to spike before enough inhibition has built up to suppress them. This moves the I-cells away from asynchrony, removing their ability to suppress the E-cells. The result is the PING rhythm shown in Figure 8B. The two simulations producing the two panels of Figure 8 are identical, except for initial conditions. 8 Phase Walkthrough of the I-Cells Resulting from Stochastic Spiking of the E-Cells In this section, we consider the effects of noisy spiking of the E-cells, for instance, as a result of stochastic external input. We show that noisy spiking of the E-cells disrupts PING rhythms by causing phase walkthrough of the I-cells. We give an approximate analysis of the conditions under which noise in the E-cells results in phase walkthrough of the I-cells. Our analysis is not very accurate quantitatively (we will explain why); however, it does reveal that slower PING rhythms are much more vulnerable to noise in the E-cells than faster ones.

Rhythms in the Presence of Noise

585

E−cells

0 40

0 0

B

160

I−cells

I−cells

E−cells

A

160

100 200 300

0 40

0 0

100 200 300

Figure 8: Example illustrating bistability. Suppression of the E-cells (A) or a rhythm (B) are possible with the same parameter values (gEI = 0.05, gIE = 0.2, gII = 0, IE = 0.03, II = 0.0025), different initial conditions.

In our simulations, we introduce random spiking of the E-cells as follows. At randomly selected times, ti1 < ti2 < ... , 1 ≤ i ≤ NE , the ith E-cell is forced to spike; that is, the value of θ associated with it is instantaneously reset to −π , and the synaptic gating variable s associated with it is set to 1. We assume that the time intervals between forced spikes, ti,k+1 − ti,k , 1 ≤ i ≤ NE , k = 1, 2, ..., are independent of each other and exponentially distributed, with a common expected value TSE . (The subscript S stands for “stochastic.”) We define νSE =

1000 . TSE

(8.1)

This is the average frequency of the random spiking of the E-cells. Figures 9C, 9D, and 9F (discussed in detail shortly) show examples of rhythms that persist in spite of random spiking in the E-cells. When an Ecell spikes out of order, it is temporarily out of synchrony with the bulk of the E-cells. It may or may not participate in the next population spike of the E-cells. However, it is brought back into synchrony with the bulk of the E-cells by the next population spike of the I-cells. If a large fraction of the E-cells spikes out of order, the number of Ecells participating in the population spikes may be reduced significantly. The population spikes of the E-cells may then no longer suffice to trigger population spikes of the I-cells. This effect is neglected in the analysis given

586

C. Borgers ¨ and N. Kopell

below, as is justified if gEI is sufficiently large and/or if the ratio νSE /νP is sufficiently small. For example, in Figure 9D, a large fraction of all E-cells spike out of order, but gEI is so large that those E-cells that participate in the population spikes still suffice to prompt the I-cells. 8.1 Analysis. The low-frequency random spiking of some of the E-cells between population spikes generates extra excitatory drive to the I-cells. If each I-cell receives input from sufficiently many E-cells, this drive is nearly constant. For two special cases, we will now analyze when the extra drive to the I-cells results in phase walkthrough. We assume here that the I-cells are modeled as theta neurons. However, we find it convenient to use the dependent variable V instead of θ (see section 2.1). In the absence of synaptic input, the equation of an I-cell is then dV = 2V(V − 1) + QI dt (see equation 2.7). The relation between QI and II is II = 2QI − 1 (see equation 2.10). The random spiking of the E-cells approximately adds the term
TSE 1 1 − e−TSE /τE QSI = gEI e−t/τE dtVrev = gEI Vrev TSE 0 TSE /τE

(8.2)

to QI (compare equation 2.12). The average of e−t/τE over [0, TSE ] appears in this formula because the stochastic spiking of the E-cells is assumed to be asynchronous. We have also used the approximation Vrev − V ≈ Vrev in equation 8.2; this is reasonable because V is near 0 except during spikes. We are interested in low-frequency random spiking in the E-cells, so TSE  τE = 2, and therefore the term e−TSE /τE in equation 8.2 is negligible: QSI ≈

gEI τE Vrev . TSE

(8.3)

QSI is added to QI , so II = 2QI − 1 turns into II + 2QSI . Denoting by fI (I) the frequency of an I-cell receiving drive I, the condition under which phase walkthrough of the I-cells is avoided becomes approximately   2gEI τE Vrev fI II + ≤ νP (8.4) TSE √ (see equation 4.1). If gII = 0, then fI (I) = (1000/π) I by equations 2.3 and 2.5. Therefore, inequality 8.4 becomes  1000 2gEI τE Vrev II + ≤ νP π TSE

Rhythms in the Presence of Noise

587

or, equivalently, π 2 (νP /1000)2 − II νSE ≤ . 1000 2gEI τE Vrev

(8.5)

Inequality 8.5 was derived assuming gII = 0. For gII > 0, the rhythm can withstand more drive to the I-cells, and therefore more random spiking of the E-cells. In the special case gII = gIE , the condition under which phase walkthrough of the I-cells is avoided is II + 2QSI ≤ IE

(8.6)

(compare equation 4.4). With the approximation 8.3, inequality 8.6 is equivalent to νSE IE − II . ≤ 1000 2gEI τE Vrev

(8.7)

To highlight the similarity between this inequality and equation 8.5, we note that IE = π 2 (νE /1000)2 by equations 2.3 and 2.5, so inequality 8.7 can be written as νSE π 2 (νE /1000)2 − II . ≤ 1000 2gEI τE Vrev

(8.8)

Note that equation 8.8 is obtained from equation 8.5 if νP is replaced by νE . Both formulas show that slower rhythms (smaller νP or smaller νE ) are more vulnerable to noisy activity in the E-cells than faster ones. 8.2 Numerical Examples. 8.2.1 Too Much Noise in the E-Cells Leads to Phase Walkthrough of the I-Cells. Figure 9A shows results of a simulation with gEI = 0.05, gIE = 0.20, gII = 0, IE = 0.1 (νE ≈ 101), II = 0 .

(8.9)

(Other parameter values are as specified in section 2.) A rhythm at frequency νP ≈ 40 is seen. We remark that r = (3/2)gIE /IE = 3 here. Inserting the parameter values 8.9 in inequality 8.5, we find that the rhythm should hold up for νSE ≤ 12.15. This prediction is not very accurate. With νSE = 7, phase walkthrough occurs already, as shown in Figure 9B. (Note that Figure 9B is indeed a noisy analog of the sort of phase walkthrough depicted in Figure 4.) Thus, the breakdown occurs earlier in our simulation than predicted

588

C. Borgers ¨ and N. Kopell

E−cells

0 40

0 0

B

160

I−cells

I−cells

E−cells

A

160

0 40

0 0

100 200 300

E−cells

0 40

0 0

D

160

I−cells

I−cells

E−cells

C

160

0 40

0 0

100 200 300

E−cells

0 40

0 0

400 800 1200

100 200 300 F

160

I−cells

I−cells

E−cells

E

160

100 200 300

0 40

0 0

400 800 1200

Figure 9: (A) Gamma frequency PING rhythm (gEI = 0.05, gIE = 0.2, gII = 0, IE = 0.1, II = 0). (B) Low-frequency stochastic spiking in the E-cells leads to phase walkthrough in the I-cells. (C) Phase walkthrough is counteracted by reducing gEI 0.02. (D) I→I synapses (gII = gIE ) make the rhythm astonishingly robust to noise in the E-cells. (E) A slow PING rhythm (IE = 0.003; all other parameter values as in A) is highly sensitive to noise in the E-cells. (F) Noise sensitivity can be greatly reduced, even for the slow rhythm, by setting gII = gIE and reducing gEI to 0.01.

Rhythms in the Presence of Noise

589

by our theory. To understand this discrepancy, we took a detailed look at the simulation underlying Figure 9B, focusing on the three population spikes of the I-cells that occur out of order, not prompted by the E-cells, in Figure 9B. Immediately before each of these population spikes, the actual drive to the I-cells rises significantly above that predicted by formula 8.2, as a result of random fluctuations. Our simple theory neglects those fluctuations, assuming instead that QSI is constant. The fluctuations would be less important in significantly larger networks. Therefore, we would expect the predictions of inequality 8.5 to be more accurate in larger networks. For the parameter values of Figure 9A, Figure 11A illustrates the transition from low, inconsequential levels of noise in the E-cells to disruptive levels. The figure shows quantities ρE ∈ [0, 1] and ρI ∈ [0, 1], measuring the regularity of the spiking of the E- and I-cells, plotted as functions of the noise frequency νSE . The precise definitions of ρE and ρI are given in appendix B. The closer ρE and ρI are to 1, the more regular is the rhythm. The figure shows that the regularity of the rhythm is lost abruptly as νSE is raised above 6. If in fact it is correct that the rhythm in Figure 9B is disrupted essentially as a result of too much drive to the I-cells, then it ought to be possible to restore the rhythm by lowering gEI . Indeed, this is the case. When gEI is lowered to 0.02, the rhythm shown in Figure 9C is obtained. 8.2.2 At Gamma frequency, I→I Synapses Greatly Reduce Sensitivity to Noise in the E-Cells. Figures 5B and 5D show that in large portions of parameter space, I→I synapses greatly enlarge the amount of drive to the I-cells that PING rhythms can withstand before breaking down as a result of phase walkthrough. To confirm this by simulation, we use the parameter values of Figure 9A, but replace gII = 0 by gII = gIE . The resulting rhythm is astonishingly insensitive to noisy spiking in the E-cells. Formula 8.8 predicts that the rhythm should survive for νSE ≤ 77. Figure 9D shows a simulation with νSE = 70. The spike time rastergram for the E-cells looks quite noisy, of course, but the rhythm is still visible. (Because of stochastic fluctuations, the rhythm would not, in reality, remain intact if νSE were equal to 77.) 8.2.3 Sensitivity to Noise in the E-Cells Increases as the Rhythm Slows Down. We now present numerical simulations illustrating that slower PING rhythms are more sensitive to noisy spiking of the E-cells, as predicted by Figure 5A and formulas 8.5 and 8.8. Figure 9E shows a slow PING rhythm disrupted by a small amount of stochastic spiking in the E-cells. (Notice that the time window shown in Figures 9E and 9F is four times longer than that shown in Figures 9A–9D.) The parameter values of Figure 9E are those of Figure 9A, except that IE has been reduced from 0.1 to 0.003. The frequency of the resulting PING rhythm is about 10. Formula 8.5 predicts that this rhythm should be abolished by random spiking in the E-cells at an average frequency of about νSE = 0.76. Indeed, phase walkthrough occurs

590

C. Borgers ¨ and N. Kopell

for νSE = 0.5 already. This is shown in Figure 9E. Thus, phase walkthrough again occurs at a (somewhat) lower value of νSE than predicted by our theory. As before, the discrepancy is due primarily to the stochastic fluctuations in the drive to the I-cells generated by the random spiking of the E-cells. For this case, Figure 11B illustrates the transition from nearly inconsequential levels of noise in the E-cells to disruptive levels. The regularity of the rhythm is lost abruptly as νSE is raised above 0.4. It is not surprising that the maximum allowable value of νSE decreases as νP decreases. For instance, if the maximum allowable value of νSE is 6 for νP = 40 (see Figure 11A), one might expect it to be four times smaller for νP = 10. What is surprising is that it is not 4 but 16 times smaller (see Figure 11B). This is in agreement with inequality 8.5, as the right-hand side of that inequality, for II = 0, is proportional to (νP /1000)2 , not to νP /1000. 8.2.4 At Subgamma Frequencies, Careful Parameter Tuning Can Reduce Sensitivity to Noise in the E-Cells. At low frequencies, I→I synapses do not help much. Inequality 8.8 predicts that even with gII = gIE , phase walkthrough of the I-cells will occur as soon as νSE > 2.3. However, the robustness of the rhythm can be enhanced by reducing gEI . Figure 9F shows a simulation in which the parameter values are as in Figure 9E, except gII = gIE , gEI = 0.01, and νSE = 7. 9 Suppression of the E-Cells Resulting from Stochastic Spiking of the I-Cells In this section, we consider the effects of random spiking of the I-cells, for instance, as a result of stochastic external input. We show that noisy spiking of the I-cells disrupts PING rhythms by causing suppression of the E-cells. We give an approximate analysis of the conditions under which noise in the I-cells results in suppression of the E-cells. As in section 8, our analysis is not very accurate quantitatively, again because it neglects statistical fluctuations that are significant at least in small networks. It does reveal that the crucial quantity here is the product rτI : the larger rτI , the less noisy spiking in the I-cells can be tolerated. In our simulations, we enforce random spiking of the I-cells in the same way in which we enforced random spiking of the E-cells in section 8. In analogy with section 8, TSI denotes the expected time between two random spikes of a given I-cell, and νSI = 1000/TSI .

(9.1)

Figures 10B and 10F (discussed in detail shortly) show examples of rhythms that persist in spite of random spiking in the I-cells. When an I-cell spikes out of order, it is temporarily out of synchrony with the bulk of the I-cells. However, it is brought back into synchrony when the next

Rhythms in the Presence of Noise

591

population spike of the E-cells prompts a population spike of the I-cells. 9.1 Analysis. The low-frequency random spiking of some of the I-cells between population spikes generates inhibitory synaptic drive to the Ecells. If each E-cell receives input from sufficiently many I-cells, this drive is nearly constant. If it is strong enough, it leads to suppression of the E-cells, and thereby abolishes the rhythm (see Figure 10D). A necessary and sufficient condition for the E-cells to escape suppression is s<

1 , r

s=

1 TSI

(9.2)

with


TSI

e−t/τI dt =

0

1 − e−TSI /τI TSI /τI

(9.3)

(see equation 6.3, which is the same as equation 9.2, and equation 2.19, which is nearly the same as equation 9.3, the only difference being that the deterministic interspike interval TI in equation 2.19 has been replaced by the expected interspike interval TSI in equation 9.3). We are interested in lowfrequency random spiking of the I-cells, and thus assume TSI  τI = 10. Therefore, the term e−TSI /τI in equation 9.3 is negligible, and condition 9.2 becomes τI 1 < , TSI r or, equivalently, using equation 9.1, 1 νSI . < 1000 rτI

(9.4)

This formula shows that the sensitivity of PING rhythms to random spiking in the I-cells depends on the size of rτI . 9.2 Numerical Examples. 9.2.1 Too Much Noise in the I-Cells Leads to Suppression of the E-Cells. Figure 10A shows, once more, the simulation of Figure 9A. Figure 10B shows the result of adding random spiking of the I-cells at frequency νSI = 25 to the simulation in Figure 10A. The noisy spiking in the I-cells slows the rhythm (as it should, since it inhibits the E-cells), but it does not disrupt it. At νSI = 30 (not shown in Figure 10), the time intervals between population spikes become irregular. Formula 9.4 predicts that suppression of the E-cells should occur for νSI > 33, approximately. In our simulations, even for νSI = 40, the

592

C. Borgers ¨ and N. Kopell

E−cells

0 40

0 0

B

160

I−cells

I−cells

E−cells

A

160

0 40

0 0

100 200 300

E−cells

0 40

0 0

D

160

I−cells

I−cells

E−cells

C

160

0 40

0 0

400 800 1200

E−cells

0 40

0 0

400 800 1200

400 800 1200 F

160

I−cells

I−cells

E−cells

E

160

100 200 300

0 40

0 0

400 800 1200

Figure 10: (A) Gamma frequency PING rhythm with r = 3 (gEI = 0.05, gIE = 0.2, gII = 0, IE = 0.1, II = 0). (B) Considerable noise in the I-cells can be withstood by this rhythm. (C) Low-frequency PING rhythm with r = 60 (IE = 0.005; all other parameters as in A). (D) This rhythm is highly sensitive to noise in the I-cells. (E) Low-frequency PING rhythm with r = 3 (IE = 0.001, gIE = 0.002; all other parameter values as in A). (F) Considerable noise in the I-cells can be withstood by this rhythm.

Rhythms in the Presence of Noise

593

E-cells are not suppressed, but the intervals between population spikes of the E-cells are long and quite irregular. Thus, the regularity of the rhythm is disrupted earlier (for smaller νSI ) than predicted by our theory, but complete suppression of the E-cells occurs later (for larger νSI ) than predicted by our theory. This discrepancy between our theory and the numerical simulations is not hard to understand. In our theory, statistical fluctuations in the strength of inhibition received by the E-cells are neglected. These fluctuations make the rhythm irregular earlier (for smaller νSI ) than predicted by our theory. When the inhibition happens to be stronger than average, the E-cells are delayed, and when it happens to be weaker than average, the E-cells spike earlier. However, statistical fluctuations cause complete suppression of the E-cells to occur later (for larger νSI ) than predicted by our theory. Even when inhibition is strong enough, on the average, to suppress the E-cells, there are time windows when it happens to fall below the threshold required for suppression. Population spikes of the E-cells may occur during those time windows. For the parameter values of Figure 10A, Figure 11C illustrates the transition from low, inconsequential levels of noise in the I-cells to disruptive levels. The figure shows the regularity measures ρE and ρI (see appendix B) plotted as functions of νSI . 9.2.2 Sensitivity to Noise in the I-Cells Increases as Drive to the E-Cells Weakens. We lower IE to 0.005. The frequency of the PING rhythm decreases to about 12, and r rises to 60. Figure 10C shows the rhythm without any stochastic spiking. (Notice that the time window shown in parts C–F of Figure 10 is four times longer than that shown in parts A and B.) Adding stochastic spiking of the I-cells at average frequency νSI = 5, the E-cells are suppressed, and thereby the rhythm is abolished (see Figure 10D). For the parameter values of Figure 10C, Figure 11D illustrates the transition from low, inconsequential levels of noise in the I-cells to disruptive levels. Note that the value of IE is 20 times smaller in Figure 11D than in Figure 11C, and the value of r is therefore 20 times larger. According to inequality 9.4, the maximum frequency of noise in the I-cells that the rhythm can withstand should therefore be 20 times smaller in Figure 11D than in Figure 11C. Indeed, this is approximately the case. 9.2.3 For Weakly Driven E-Cells, Careful Parameter Tuning Can Reduce Sensitivity to Noise in the I-Cells. We have mentioned that PING rhythms with low values of IE can be made fairly noise insensitive by lowering gIE proportionally, ensuring that r does not become too large. To illustrate this, we lower IE even further, to 0.001, but also lower gIE to 0.002, bringing the value of r back to 3, as in Figure 10A. For reasons discussed in the next paragraph, we also use a smaller value of gEI here: gEI = 0.005. The frequency of the resulting rhythm is approximately 8 (see Figure 10E). Remarkably, the rhythm in Figure 10E survives stochastic spiking of the I-cells at frequency

594

C. Borgers ¨ and N. Kopell

A

B

ρE

1

0.5

0.5

1

1

0.5

0.5

ρI

ρI

ρ

E

1

0 2 4 6 8 10 νSE

0

C

ρE

ρE

1

0.5

0.5 1

ρI

1

ρI

1

D

1

0.5 0

0.5 νSE

50 ν

SI

100

0.5 0 1 2 3 4 5 νSI

Figure 11: Regularity measures ρE and ρI (defined in appendix B) as functions of noise frequency. (A) At gamma frequency (40 Hz), the rhythm can withstand 6 Hz noise in the E-cells. (B) At four times lower frequency (10 Hz), it can withstand no more than 0.4 Hz noise in the E-cells. (C) At gamma frequency (40 Hz), the regularity of the rhythm is lost if the frequency of the noise in the I-cells is greater than 40 Hz. (D) When IE is reduced twenty-fold, the rhythm can withstand no more than 1.5 Hz noise in the I-cells.

Rhythms in the Presence of Noise

595

5 (see Figure 10F). This is a result of the moderate (not very large) value of r. In the preceding simulation, we used a reduced value of gEI . In general, for small gIE , the PING rhythm is more rapidly established if gEI is small as well. (Of course, gEI must still be large enough for a population spike of the E-cells to trigger a population spike of the I-cells.) Section 8 suggests the following heuristic explanation of this numerical observation. In the initial phases of the simulation, before the rhythm is established, the population spikes are fuzzy (see, for instance, the beginning of the simulation in Figure 10E). If gIE is small, there is a considerable amount of nearly asynchronous activity of the E-cells between activity peaks. This activity results in nearly tonic drive to the I-cells, which may result in premature spiking of some of the I-cells, and may therefore make it more difficult for the rhythm to be established. Lowering gEI reduces this effect. 9.2.4 Effects of I→I Synapses. We have used gII = 0 throughout this section. As pointed out in earlier sections, I→I synapses generally stabilize PING rhythms by slowing disruptive spiking in the I-cells. However, in the experiments presented here, the disruptive spiking in the I-cells is forced, regardless of the value of gII . As a result, if gII were set to gIE , the results presented in this section would remain virtually unchanged. If the random spiking were instead generated by weaker random input pulses, not necessarily always inducing immediate spiking, then I→I synapses would indeed counteract the suppression of the E-cells by the I-cells. 10 Simulations Including Sparse Connectivity and Heterogeneity In this section, we present some numerical simulations including sparse connectivity and heterogeneity. We sparsen the connectivity as follows. The possible synaptic connections are considered one at a time. Each possible connection is removed with probability 0.5. If it is retained, its strength is doubled. For instance, the strength of an I→E connection is 0 with probability 0.5, and 2gIE /NI with probability 0.5. This is a moderate degree of sparseness. However, in earlier work (Borgers ¨ & Kopell, 2003), we showed that the effect of sparseness on the coherence of a PING rhythm is determined not by the fraction p of connections retained, but by pNE /(1 − p) and pNI /(1 − p). For instance, in a 10 times larger network (i.e., in a network of 1600 E-cells and 400 I-cells), we would get approximately the same effect if we removed connections with probability 10/11 and strengthened the retained connections eleven-fold. (Note that p/(1 − p) = 1/10 when p = 1/11.) In addition to sparseness, we introduce 20% heterogeneity in synaptic strengths. For instance, the strength of a given retained I→E synapse is not precisely 2gIE /NI , but rather a gaussian random number with mean 2gIE /NI and standard deviation 0.4gIE /NI . (This random number is negative with

596

C. Borgers ¨ and N. Kopell

very small but positive probability. If our program draws a negative strength for one of the synapses, the strength of that synapse is reset to zero.) Thus, gIE , gEI , and gII are no longer the total synaptic strengths, but they are (very close to) the expected total synaptic strengths. We also introduce 20% heterogeneity in the external drive to the E-cells. That is, the external drive received by a given E-cell is not precisely IE , but rather a gaussian random number with mean IE and standard deviation 0.20IE . Thus, IE is no longer the external drive to each E-cell, but it is the expected external drive to each E-cell. Similarly, we introduce 20% heterogeneity in the external drive to the I-cells. Finally, all cells are forced to spike at random times at average frequencies νSE (for the E-cells) and νSI (for the I-cells). At gamma frequency, PING is robust to heterogeneity, random connectivity, and noisy external drives. Figure 12A shows the results of a simulation with gIE = 0.2, gEI = 0.05, gII = 0.2, IE = 0.1 (νE ≈ 101), II = 0.05, νSE = 5, νSI = 5. Figure 12B shows, for the same simulation, quantities sE = sE (t) and sI = sI (t) measuring the average strength of excitatory and inhibitory synapses (see appendix B for precise definitions). The rhythm is clearly detectable in Figures 12A and 12B. Its frequency is approximately 40. Even with heterogeneity and random connectivity, it is possible to obtain PING rhythms at subgamma frequencies, but careful parameter tuning is required. As discussed in section 8.2, one should lower gEI , to avoid phase walkthrough as a result of the drive to the I-cells resulting from noisy activity of the E-cells. However, one cannot lower gEI too much, since the I-cells must respond promptly to population spikes of the E-cells. As discussed in section 9, one should lower both IE and gIE proportionally, keeping r constant. This avoids making the rhythm vulnerable to suppression of the E-cells by noisy spiking of the I-cells. The parameter values gIE = 0.004, gEI = 0.004, gII = 0.004, IE = 0.001 (νE ≈ 10), II = 0.0005, νSE = 2, νSI = 2 yield a PING rhythm at frequency νP ≈ 8 in the presence of sparse connectivity, heterogeneity, and noise. Sparseness and heterogeneity are introduced as described earlier. Figure 12C shows the spike times, and Figure 12D shows sE and sI for the same simulation. (Notice that the time window shown in Figures 12C and 12D is five times longer than that shown in Figures 12A and 12B.)

0.26

sE

160

I−cells

A

0 40

0 0

0 0.68

0 0

100 200 300

0.16

0 40

0 0.59

I−cells

B

s

E

160

100 200 300

sI

E−cells

597

sI

E−cells

Rhythms in the Presence of Noise

0 0

500 1000 1500

0 0

500 1000 1500

Figure 12: Sparse connectivity, heterogeneity, and noise do not necessarily abolish PING rhythm, either at gamma frequency or—with carefully tuned parameter values—at lower frequencies. (A) Gamma frequency PING rhythm (gEI = 0.05, gIE = 0.2, gII = 0.2, IE = 0.1, II = 0.05); spike times (left panel), and sE and sI as defined in appendix B (right panel). (B) Low-frequency PING rhythm (gEI = 0.004, gIE = 0.004, gII = 0.004, IE = 0.001, II = 0.0005); spike times (left panel), and sE and sI (right panel).

11 The Frequency Range in Which PING is Robust A major theme of this letter has been that PING rhythms at low frequencies, significantly below the gamma range, are easily abolished by noise, unless the parameter values are tuned very carefully. In fact, there is also a sense in which PING rhythms at high frequencies, above the gamma range, are not robust. We remarked earlier that one typically needs r ≥ 3 for synchronization in the presence of 20% heterogeneity in network parameter values. Combining

598

C. Borgers ¨ and N. Kopell

this with equation 3.15, and recalling q ≤ 1, we find TP ≥ (ln 3)τI ≈ 1.1τI . If τI = 10, then TP ≥ 11, which approximately means νP ≤ 91. Thus, in the presence of significant heterogeneity, PING rhythms cannot have a frequency above the gamma range. This argument is, of course, far from precise. In particular, nothing is really special about the value r = 3. Nevertheless, we do believe that the argument is correct in essence. In more intuitive language, when one attempts to drive the frequency of a PING rhythm above the gamma range by raising drive to the E-cells, one must also raise the strength of the I→E synapses in order to maintain synchronization. The rise in the strength of the I→E synapses brings the frequency back into the gamma range. 12 Summary and Discussion We have described and analyzed two distinct ways in which too much external drive to the I-cells can disrupt PING rhythms: The I-cells may synchronize, but get ahead of the E-cells (phase walkthrough of the I-cells), or the I-cells may not synchronize, and their activity may keep the E-cells from spiking altogether (suppression of the E-cells). Our analysis of the effects of deterministic drive to the I-cells casts light on the effects of stochastic drive to the E- or I-cells. If there is too much noisy spiking activity in the E-cells, the resulting rise in excitatory drive to the I-cells may lead to phase walkthrough; too much noisy spiking in the I-cells may result in suppression of the E-cells. I→I synapses reduce the spiking frequency of the I-cells, counteracting phase walkthrough of the I-cells, suppression of the E-cells, and the effects of noisy external drive on both I- and E-cells, and thereby enhancing the robustness of the rhythm. Our analysis also shows why PING rhythms are most robust when their frequency lies in the gamma range. Above the gamma range, synchronization breaks down easily when the E-cell population is heterogeneous. As the frequency is lowered, on the other hand, PING becomes increasingly sensitive to noisy spiking of the E-cells, and—unless the strength of the inhibitory synapses and the drive to the E-cells are calibrated carefully—more sensitive to noisy spiking of the I-cells as well. Gutkin and Ermentrout (1998) showed that noise sensitivity of type I model neurons is greater at lower frequencies than at higher frequencies, (see in particular Figure 6B of their article). Our point about noise sensitivity and frequency may seem similar at first but is in fact quite different. Gutkin and Ermentrout showed for a single theta neuron that noisy drive has a more severe effect when the intrinsic frequency of the neuron is low than when it is high. We have shown for an E/I network that the tonic component of the drive to the I-cells created by noisy spiking in the E-cells has a more severe effect when the population frequency is low than when it is high.

Rhythms in the Presence of Noise

599

We conclude with a brief discussion of the relation of our work to previous work on synchronization in networks of model neurons. Much of this work has addressed purely excitatory networks (e.g., Peskin, 1975; Mirollo & Strogatz, 1990; Somers & Kopell, 1993; Hansel, Mato, & Meunier, 1995; Crook, Ermentrout, & Bower, 1998; Bose, Kopell, & Terman, 2000; van Vreeswijk & Hansel, 2001; Acker, Kopell, & White, 2003), purely inhibitory networks (e.g., Wang & Rinzel, 1992; Golomb & Rinzel, 1993; Skinner, Kopell, & Marder, 1994; Chow, 1998; Chow, White, Ritt, & Kopell, 1998; Terman, Kopell, & Bose, 1998; White, Chow, Ritt, Soto-Tervino, & Kopell, 1998; Brunel & Hakim, 1999), or pairs of neurons that are either both excitatory, or both inhibitory (e.g. van Vreeswijk, Abbott, & Ermentrout, 1994; Kopell & Ermentrout, 2002). Synchronization in networks including both excitatory and inhibitory model neurons has been studied widely as well (e.g., Wang, Golomb, & Rinzel, 1995; Ermentrout & Kopell, 1998; Brunel, 2000a, 2000b; Whittington et al., 2000; Tiesinga et al., 2001; Hansel & Mato, 2001; van Vreeswijk & Hansel, 2001; Borgers ¨ & Kopell, 2003; Hansel & Mato, 2003). For more complete references, see Kopell and Ermentrout (2002) or Hansel and Mato (2003). Much of the work on (a)synchrony in neuronal networks has focused on states of asynchronous spiking and ways in which such states can lose stability (Abbott & Van Vreeswijk, 1993; Gerstner & van Hemmen, 1993; Hansel et al., 1995; Gerstner, 2000; Neltner Hansel, Mato, & Meunier, 2000; van Vreeswijk, 2000; Hansel & Mato, 2003). In this letter, we have taken a complementary point of view, focusing on states of nearly synchronous spiking, asking for which parameter values such states exist and how sensitive they are to changes in parameter values and noise. Hansel and Mato (2003) gave a comprehensive analysis of the bifurcations by which asynchronous states can lose stability in networks of excitatory and inhibitory neurons. They identified four codimension 1 bifurcations, of which three lead to oscillatory behavior. One of these (the one associated with crossing the curve L4 in Figure 11 of Hansel and Mato, 2003) corresponds to the emergence of PING and another (associated with the curve L2 in the same figure) to the emergence of ING. In drawing their phase diagrams, Hansel and Mato varied the strengths of synaptic conductances while adjusting external drives to keep the average firing rates of the E- and I-cell populations constant. By contrast, we have treated synaptic strengths and external drives as independent parameters. This difference in point of view complicates direct comparisons between the results of Hansel and Mato and ours. In particular, the lower panel of Figure 11 of Hansel and Mato shows that in a certain sense, I→I synapses counteract PING. For stronger I→I synapses, stronger coupling between the E- and I-cells is needed for the transition from asynchrony to PING (crossing the L4 boundary). At first sight, this appears to contradict our conclusion that I→I synapses promote PING by protecting against phase walkthrough of the I-cells and suppression of the E-cells. However, in fact, there is no contra-

600

C. Borgers ¨ and N. Kopell

diction, since different points in the phase plane depicted by Hansel and Mato correspond to different external drives. The frequency of PING oscillations was analyzed in a recent paper by Brunel and Wang (2003). We note that their study was concerned with oscillations driven by noisy input, whereas in this article, we have focused on oscillations driven by deterministic input (and, in some cases, disrupted by a noisy input component). However, not all discrepancies between their results and ours can plausibly be explained by this difference. For instance, equation 18 of Brunel and Wang (2003) determines the population frequency of the oscillation for E/I networks without E→E and I→I synapses. According to this equation, the frequency depends on synaptic delay, rise, and decay times. In our simulations, there are no synaptic delays, and synaptic rise times are very short. Furthermore, numerical results (not shown here) indicate that shortening the synaptic rise times much further has little impact on the results of our simulations, and the same holds even when the rhythm is driven by noise instead of deterministic tonic drive. This motivates consideration of the limit of equation 18 of Brunel and Wang (2003) as the synaptic delay and rise times tend to zero. Doing this, one finds the prediction that the population oscillation frequency should tend to infinity—in stark discrepancy with our numerical results. The reason for this and other discrepancies between the results of Brunel and Wang and ours remains to be determined. An important difference between the study of Brunel and Wang and ours lies in the choice of membrane time constants for the excitatory neurons. Brunel and Wang used linear integrate-and-fire neurons with time constants equal to 10 ms for inhibitory neurons and 20 ms for excitatory ones. Significantly shorter membrane time constants are believed to be appropriate for neocortical pyramidal neurons in vivo during high ongoing activity (Destexhe & Par´e, 1999; Destexhe, Rudolph, & Par´e, 2003). For theta neurons, as for real cortical neurons, the membrane time “constant” is not actually a constant, but depends on external input. In our simulation of gamma rhythms, the E-cells typically have rather short membrane time constants (on the order of few milliseconds) shortly after receiving input from the I-cells; this is what makes the stable river in Figure 3A strongly attracting and allows for synchronization of the E-cells within one gamma cycle. Appendix A: The Impact of Synchrony and Asynchrony on Downstream Effects In computing the suppression boundary, we assumed that the activity of the I-cells was completely asynchronous. The rationale for this assumption was that asynchrony maximizes the downstream effect of an assembly of inhibitory neurons. In this appendix, we prove a precise statement to this effect. We also prove a precise statement showing that synchrony maximizes

Rhythms in the Presence of Noise

601

the downstream effect of an assembly of excitatory neurons. In contrast with the main portion of this article, we use the linear integrate-and-fire model here. We have not so far generalized these results to the theta model. We begin with a precise version of the following claim. If N periodic current inputs succeed in making an integrate-and-fire neuron spike, then the same inputs would also succeed in making the neuron spike if they were synchronized. Theorem 1. Let ϕ ≥ 0 be periodic function with period T, and let t1 , ..., tN ∈ [0, T). Let τ > 0, and consider the initial value problems

N dV V  ϕ(t − ti ) =− + dt τ i=1

V(0) = 0 and dVˆ Vˆ = − + Nϕ(t). dt τ ˆ V(0) =0 Then ˆ ≥ sup V(t) . sup V(t) t≥0

t≥0

Denote by V(V0 ; t1 , ..., tN ; t) the solution of

Proof.

N V  dV ϕ(t − ti ). =− + dt τ i=1

V(0) = V0 . In general, V(V0 ; t1 , ..., tN ; t) =


t  N 0

 ϕ(s − ti ) e(s−t)/τ ds + V0 e−t/τ .

(A.1)

i=1

To find solutions with period T, we define V0,per > 0 to be the solution of
0

T



N  i=1

 ϕ(s − ti ) e(s−T)/τ ds + V0,per e−T/τ = V0,per .

602

C. Borgers ¨ and N. Kopell

We write Vper (t1 , ..., tN ; t) = V(V0,per ; t1 , ..., tN ; t) . From equation A.1, V(0; t1 , ..., tN ; t) < V(V0,per ; t1 , ..., tN ; t) for all t, but   lim V(0; t1 , ..., tN ; t) − V(V0,per ; t1 , ..., tN ; t) = 0 .

t→∞

Therefore, sup V(0; t1 , ..., tN ; t) = sup Vper (t1 , ..., tN ; t) . t≥0

t≥0

We denote by Vˆ per the uniquely determined solution with period T of dVˆ per Vˆ per =− + ϕ(t) . dt τ Then Vper (t1 , ..., tN ; t) =

N 

Vˆ per (t − ti ) .

i=1

Therefore, sup Vper (t1 , ..., tN ; t) ≤ N sup Vˆ per (t) , t≥0

t≥0

with “=” if ti = 0 for all i. Therefore, sup V(0; t1 , ..., tN ; t) = sup Vper (t1 , ..., tN ; t) t≥0

t≥0

≤ N sup Vˆ per (t) t≥0

= sup Vper (0, ..., 0; t) t≥0

= sup V(0; 0, ..., 0; t). t≥0

This proves the assertion.

Rhythms in the Presence of Noise

603

We next show that asynchrony minimizes the downstream effect of an assembly of excitatory neurons. More precisely, we will show that if a given asynchronous excitatory synaptic input succeeds in making an integrateand-fire neuron spike, then the same input, delivered phasically with a period T, will also succeed in making the neuron spike. Theorem 2. Let I ∈ IR, τ > 0, Vrev ∈ IR (think of Vrev as the reversal potential of an excitatory synapse), and let g = g(t) ≥ 0 be a function with period T. Define V = V(t) by dV V = − + I + g(Vrev − V) dt τ V(0) = 0 . Let g=

1 T




T

g(t)dt .

0

Define V = V(t) by V dV = − + I + g(Vrev − V) dt τ V(0) = 0 . Then sup V(t) ≤ sup V(t) . t≥0

t≥0

Proof. If supt≥0 V(t) = ∞, then we have nothing to prove, so assume supt≥0 V(t) < ∞. V dV = − + I + g(t)(Vrev − V) dt τ  sup V(t) ≥−

t≥0

τ



+ I + g(t) Vrev − sup V(t) .

(A.2)

t≥0

The right-hand side of this inequality is a periodic function of t and a lower bound on dV/dt. The assumption supt≥0 V(t) < ∞ then implies that the average of the right-hand side of equation A.2 is ≤ 0: sup V(t) −

t≥0

τ





+ I + g Vrev − sup V(t) ≤ 0 . t≥0

604

C. Borgers ¨ and N. Kopell

This means that dV/dt would be ≤ 0 if V ever reached supt≥0 V(t). So V cannot exceed supt≥0 V(t). This proves the assertion. Thinking of Vrev as the reversal potential of an inhibitory synapse, one can reinterpret the statement just proved as follows. If a given inhibitory synaptic input suppresses an integrate-and-fire neuron, then the same input, delivered asynchronously, will also suppress the neuron. That is, asynchrony maximizes the downstream effect of an assembly of inhibitory neurons. Appendix B: A Measure of Regular Rhythmicity We associate synaptic gating variables with the presynaptic neurons (see section 2.2). Let sE,i denote the synaptic gating variable associated with ith E-cell, 1 ≤ i ≤ NE , and let sE (t) be the average of sE,i over i ∈ {1, 2, ..., NE } and over the time interval [t − 5, t + 5]. We approximate the time average using the trapezoid method. The time averaging is needed in some of our noisier simulations (for instance, those of Figure 12) to eliminate small random fluctuations that would make automatic detection of the underlying rhythm difficult. We define sE,min + sE,max sE,min = min sE (t) , sE,max = max sE (t) , and sE,av = . t t 2 The minimum and maximum are taken over the second half of the simulation time interval to avoid the effects of initial transients. We consider those times t in the second half of the simulation time interval at which sE changes from values below sE,av to values above sE,av . We denote these times by tE,1 < tE,2 < ... < tE,nE . Our measure of (regular) rhythmicity is   min(tE,i+1 − tE,i ) ρE = max(tE,i+1 − tE,i )  0

if nE ≥ 3 , otherwise .

Here the minimum and maximum are taken over i ∈ {1, 2, ..., nE −1}. Clearly ρE ∈ [0, 1]; the closer ρE is to 1, the more regular is the rhythm. To visualize and measure rhythmicity of the I-cells, sI (t) and ρI are defined analogously. Acknowledgments We are grateful to Steve Epstein for reading the manuscript and providing helpful criticism. We also thank the referees for their detailed and thoughtful comments, which led to several improvements in the article. C.B. is

Rhythms in the Presence of Noise

605

supported by NSF grant DMS-0418832. N.K. is supported by NSF grant DMS-9706694 and NIH grant MH47150. References Abbott, L. F., & van Vreeswijk, C. (1993). Asynchronous state in networks of pulse-coupled oscillators. Phys. Rev. E, 48, 1483–1490. Acker, C., Kopell, N., & White, J. (2003). Synchronization of strongly coupled excitatory neurons: Relating network behavior to biophysics. J. Comp. Neurosci., 15, 71–90. Borgers, ¨ C., & Kopell, N. (2003). Synchronization in networks of excitatory and inhibitory neurons with sparse, random connectivity. Neural Comp., 15(3), 509–539. Bose, A., Kopell, N., & Terman, D. (2000). Almost synchronous solutions for pairs of neurons coupled by excitation. Physica D, 140, 69–94. Brunel, N. (2000a). Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons. J. Comp. Neurosci., 8, 183–208. Brunel, N. (2000b). Phase diagrams of sparsely connected networks of excitatory and inhibitory spiking neurons. Neurocomputing, 32, 307–312. Brunel, N., & Hakim, V. (1999). Fast global oscillations in networks of integrateand-fire neurons with low firing rates. Neural Comp., 11, 1621–1671. Brunel, N., & Wang, X.-J. (2003). What determines the frequency of fast network oscillations with irregular neural discharges? I. Synaptic dynamics and excitation-inhibition balance. J. Neurophysiol., 90, 415–430. Charpak, S., G¨ahwiler, B. H., Do, K. Q., & Knopfel, ¨ T. (1990). Potassium conductances in hippocampal neurons blocked by excitatory amino-acid transmitters. Nature, 347, 765–767. Chow, C. C. (1998). Phase-locking in weakly heterogeneous neuronal networks. Physica D, 118, 343–370. Chow, C. C., White, J. A., Ritt, J., & Kopell, N. (1998). Frequency control in synchronized networks of inhibitory neurons. J. Comp. Neurosci., 5, 407–420. Crook, S. M., Ermentrout, G. B., & Bower, J. M. (1998). Spike frequency adaptation affects the synchronization properties of networks of cortical oscillators. Neural Comp., 10, 837–854. Destexhe, A., & Par´e, D. (1999). Impact of network activity on the integrative properties of neocortical pyramidal neurons in vivo. J. Neurophysiol., 81, 1531– 1547. Destexhe, A., Rudolph, M., & Par´e, D. (2003). The high-conductance state of neocortical neurons in vivo. Nature Reviews Neurosci., 4, 739–751. Diener, F. (1985a). Propri´et´es asymptotiques des fleuves. C. R. Acad. Sci. Paris, 302, 55–58. Diener, M. (1985b). D´etermination et existence des fleuves en dimension 2. C. R. Acad. Sci. Paris, 301, 899–902. Engel, A. K., Konig, ¨ P., Kreiter, A. K., Schillen, T. B., & Singer, W. (1992). Temporal coding in the visual cortex: New vistas on integration in the nervous system. Trends Neurosci., 15, 218–226.

606

C. Borgers ¨ and N. Kopell

Ermentrout, G. B. (1996). Type I membranes, phase resetting curves, and synchrony. Neural Comp., 8, 879–1001. Ermentrout, G. B., & Kopell, N. (1986). Parabolic bursting in an excitable system coupled with a slow oscillation. SIAM J. Appl. Math, 46, 233–253. Ermentrout, G. B., & Kopell, N. (1998). Fine structure of neural spiking and synchronization in the presence of conduction delay. Proc. Natl. Acad. Sci. USA, 95, 1259–1264. Fisahn, A., Pike, F. G., Buhl, E. H., & Paulsen, O. (1998). Cholinergic induction of network oscillations at 40 Hz in the hippocampus in vitro. Nature, 394, 186–189. Fries, P., Neuenschwander, S., Engel, A. K., Goebel, R., & Singer, W. (2001). Rapid feature selective neuronal synchronization through correlated latency shifting. Nature Neurosci., 4, 194–200. Fries, P., Roelfsema, P. R., Engel, A. K., Konig, ¨ P., & Singer, W. (1997). Synchronization of oscillatory responses in visual cortex correlates with perception in interocular rivalry. Proc. Natl. Acad. Sci. USA, 94, 12699–12704. Gerstner, W. (2000). Population dynamics of spiking neurons: Fast transients, asynchronous state, and locking. Neural Comp., 12, 43–89. Gerstner, W., & van Hemmen, J. L. (1993). Coherence and incoherence in a globally coupled ensemble of pulse-emitting units. Phys. Rev. Lett., 71, 312– 315. Golomb, D., & Rinzel, J. (1993). Dynamics of globally coupled inhibitory neurons with heterogeneity. Phys. Rev. E, 48, 4810–4814. Gray, C. M., & McCormick, D. A. (1996). Chattering cells: Superficial pyramidal neurons contributing to the generation of synchronous oscillations in the visual cortex. Science, 274, 109–113. Gutkin, B. S., & Ermentrout, G. B. (1998). Dynamics of membrane excitability determine interspike interval variability: A link between spike generation mechanisms and cortical spike train statistics. Neural Comp., 10, 1047–1065. Hansel, D., & Mato, G. (2001). Existence and stability of persistent states in large neuronal networks. Phys. Rev. Lett., 86, 4175–4178. Hansel, D., & Mato, G. (2003). Asynchronous states and the emergence of synchrony in large networks of interacting excitatory and inhibitory neurons. Neural Comp., 15(1), 1–56. Hansel, D., Mato, G., & Meunier, G. (1995). Synchrony in excitatory neural networks. Neural Comp., 7, 307–337. Hasselmo, M. E. (1999). Neuromodulation: Acetylcholine and memory consolidation. Trends in Cognitive Sciences, 3 (9), 351–359. Hodgkin, A. L. (1948). The local changes associated with repetitive action in a nonmedullated axon. J. Physiol. (London), 107, 165–181. Hoppensteadt, F. C., & Izhikevich, E. M. (1997). Weakly connected neural networks. New York: Springer-Verlag. Izhikevich, E. M. (2000). Neural excitability, spiking and bursting. Int. J. Bifurcation and Chaos, 10, 1171–1266. Knowles, W. D., & Schwartzkroin, P. A. (1981). Local circuit synaptic interactions in hippocampal brain slices. J. Neurosci., 1, 318–322.

Rhythms in the Presence of Noise

607

Kopell, N., & Ermentrout, G. B. (2002). Mechanisms of phase-locking and frequency control in pairs of coupled neural oscillators. In B. Fiedler (Ed.), Handbook on dynamical systems (Vol. 2: Towards applications (pp. 3–54). Amsterdam: Elsevier. Latham, P. E., Richmond, B. J., Nelson, P. G., & Nirenberg, S. (2000). Intrinsic dynamics in neuronal networks. I. Theory. J. Neurophysiol., 83, 808–827. Mirollo, R. E., & Strogatz, S. H. (1990). Synchronization of pulse-coupled biological oscillators. SIAM J. Appl. Math., 50, 1645–1662. Neltner, L., Hansel, D., Mato, G., & Meunier, C. (2000). Synchrony in heterogeneous networks of spiking neurons. Neural Comput., 12, 1607–1641. Olufsen, M., Whittington, M. A., Camperi, M., & Kopell, N. (2003). New functions for the gamma rhythm: Population tuning and preprocessing for the beta rhythm. J. Comp. Neurosci., 14, 33–54. Peskin, C. S. (1975). Mathematical aspects of heart physiology. New York: Courant Institute of Mathematical Sciences, New York University. Pulvermuller, ¨ F., Birbaumer, N., Lutzenberger, W., & Mohr, B. (1997). Highfrequency brain activity: Its possible role in attention, gestalt processing and language. Progress in Neurobiology, 52, 427–445. Rinzel, J., & Ermentrout, G. B. (1998). Analysis of neural excitability and oscillations. In C. Koch, & I. Segev (Eds.), Methods in neuronal modeling (pp. 251–292), Cambridge, MA: MIT Press. Singer, W. (1999). Neuronal synchrony: A versatile code for the definition of relations. Neuron, 24, 49–65. Singer, W., & Gray, C. M. (1995). Visual feature integration and the temporal correlation hypothesis. Ann. Rev. Neurosci., 18, 555–586. Skinner, F., Kopell, N., & Marder, E. (1994). Mechanisms for oscillation and frequency control in networks of mutually inhibitory relaxation oscillators. J. Comp. Neurosci., 1, 69–87. Slotnick, S. D., Moo, L. R., Kraut, M. A., Lesser, R. P., & Hart, J. (2002). Interactions between thalamic and cortical rhythms during semantic memory recall in human. Proc. Natl. Acad. Sci. USA, 99(9), 6440–6443. Somers, D., & Kopell, N. (1993). Rapid synchronization through fast threshold modulation. Biol. Cybern., 68, 393–407. Storm, J. F. (1989). An afterhyperpolarisation of medium duration in rat hippocampal pyramidal cells. J. Physiol. (London), 409, 171–190. Tallon-Baudry, C., Bertrand, O., Peronnet, C. F., & Pernier, J. (1998). Induced gamma-band activity during the delay of a visual short-term memory task in humans. J. Neurosci., 18, 4244–4284. Tam´as, G., Buhl, E. H., Lorincz, ¨ A., & Somogyi, P. (2000). Proximally targeted GABAergic synapses and gap junctions synchronize cortical interneurons. Nature Neurosci., 3(4), 366–371. Terman, D., Kopell, N., & Bose, A. (1998). Dynamics of two mutually coupled slow inhibitory neurons. Physica D, 117, 241–275. Tiesinga, P. H. E., Fellous, J.-M., Jos´e, J. V., & Sejnowski, T. J. (2001). Computational model of carbachol-induced delta, theta, and gamma oscillations in the hippocampus. Hippocampus, 11, 251–274.

608

C. Borgers ¨ and N. Kopell

Tiitinen, H., Sinkkonen, J., Reinikainen, K., Alho, K., Lavikainen, J., & Naatanen, R. (1993). Selective attention enhances the auditory 40-Hz transient response in humans. Nature, 364, 59–60. Traub, R. D., Bibbig, A., Fisahn, A., LeBeau, F. E. N., Whittington, M. A., & Buhl, E. H. (2000). A model of gamma-frequency network oscillations induced in the rat CA3 region by carbachol in vitro. European J. Neuroci., 12, 4093–4106. Traub, R. D., Buhl, E. H., Gloveli, T., & Whittington, M. A. (2003). Fast rhythmic bursting can be induced in layer 2/3 cortical neurons by enhancing persistent Na+ conductance or by blocking BK channels. J. Neurophysiol., 89, 909–921. Traub, R. D., Kopell, N., Bibbig, A., Buhl, E. H., Lebeau, F. E. N., & Whittington, M. A. (2001). Gap junctions between interneuron dendrites can enhance long-range synchrony of gamma oscillations. J. Neurosci., 21, 9478–9486. Traub, R. D., Whittington, M. A., Colling, S. B., Buzs´aki, G., & Jefferys, J. G. R. (1996). Analysis of gamma rhythms in the rat hippocampus in vitro and in vivo. J. Physiol. (London), 493, 471–484. van Vreeswijk, C. (2000). Analysis of the asynchronous state in networks of strongly coupled oscillators. Phys. Rev. Lett., 84, 5110–5113. van Vreeswijk, C., Abbott, L. F., & Ermentrout, G. B. (1994). When inhibition not excitation synchronizes neural firing. J. Comp. Neurosci., 1, 313–321. van Vreeswijk, C., & Hansel, D. (2001). Patterns of synchrony in neural networks with spike adaptation. Neural Comp., 13, 959–992. Wang, X.-J., & Buzs´aki, G. (1996). Gamma oscillation by synaptic inhibition in a hippocampal interneuronal network model. J. Neurosci., 16, 6402–6413. Wang, X.-J., Golomb, D., & Rinzel, J. (1995). Emergent spindle oscillations and intermittent burst firing in a thalamic model: Specific neuronal mechanisms. Proc. Natl. Acad. Sci. USA, 2, 5577–5581. Wang, X.-J., & Rinzel, J. (1992). Alternating and synchronous rhythms in reciprocally inhibitory model neurons. Neural Comp., 4, 84–97. White, J., Chow, C., Ritt, J., Soto-Trevino, C., & Kopell, N. (1998). Synchronization and oscillatory dynamics in heterogeneous, mutually inhibited neurons. J. Comp. Neurosci., 5, 5–16. Whittington, M. A., Stanford, I. M., Colling, S. B., Jefferys, J. G. R., & Traub, R. D. (1997). Spatiotemporal patterns of gamma frequency oscillations tetanically induced in the rat hippocampal slice. J. Physiol. 502, 591–607. Whittington, M. A., Traub, R. D., & Jefferys, J. G. R. (1995). Synchronized oscillations in interneuron networks driven by metabotropic glutamate receptor activation. Nature, 373, 612–615. Whittington, M. A., Traub, R. D., Kopell, N., Ermentrout, B., & Buhl, E. H. (2000). Inhibition-based rhythms: experimental and mathematical observations on network dynamics. Int. J. Psychophysiol., 38, 315–336. Received February 17, 2004; accepted July 27, 2004.

LETTER

Communicated by Sebastian Seung

Supervised Learning Through Neuronal Response Modulation Christian D. Swinehart cds@brandeis. edu.

L.F. Abbott abbott@brandeis. edu. Volen Center and Department of Biology, Brandeis University, Waltham, MA 02454, U.S.A.

Neural networks that are trained to perform specific tasks must be developed through a supervised learning procedure. This normally takes the form of direct supervision of synaptic plasticity. We explore the idea that supervision takes place instead through the modulation of neuronal excitability. Such supervision can be done using conventional synaptic feedback pathways rather than requiring the hypothetical actions of unknown modulatory agents. During task learning, supervised response modulation guides Hebbian synaptic plasticity indirectly by establishing appropriate patterns of correlated network activity. This results in robust learning of function approximation tasks even when multiple output units representing different functions share large amounts of common input. Reward-based supervision is also studied, and a number of potential advantages of neuronal response modulation are identified. 1 Introduction Correlation-based, Hebbian mechanisms of synaptic plasticity have been used with considerable success to explain the spontaneous development of selectivity and sensory maps in neural circuits (see Miller, 1996). However, when such plasticity mechanisms are applied to the development of networks that perform specific functions, rather than simply represent input data, a problem arises. To guide correlation-based synaptic plasticity, the activity of a naive neural circuit must be correlated in a manner similar to that of the final, functioning circuit. But such correlations usually arise only after the synapses of the circuit have been appropriately adjusted. The consequence of this is a chicken-and-egg problem: Which comes first, correlations or synaptic modifications? The traditional answer to this question is that synaptic modifications come first, guided by a supervisor. The supervisor is a hypothetical neural circuit that assesses network performance, computes an error signal, and uses it to direct synaptic plasticity within the network. Such schemes work Neural Computation 17, 609–631 (2005)

c 2005 Massachusetts Institute of Technology


610

C. Swinehart and L. Abbott

extremely well for many tasks (Widrow & Stearns, 1985; Chauvin & Rumelhart, 1995; Hertz, Krogh, & Palmer, 1991; Dayan & Abbott, 2001), making them attractive models for learning in biological systems. However, for biological applications, it is important to identify the pathways through which the supervisory circuit controls synaptic plasticity. In some cases, such as climbing fiber input to cerebellar Purkinje cells, such a mechanism appears to be in place. In other systems, such as cerebral cortex, an appeal must be made to some form of modulatory (perhaps dopaminergic; see Schultz, Dayan, & Montague, 1997) control of synaptic plasticity that is largely conjectural. Furthermore, modulatory pathways tend to be slow and nonlocal, making them poorly suited for the rapid, precise control of synaptic plasticity needed during task learning. These considerations lead us to explore the possibility that supervision of synaptic plasticity takes place indirectly rather than directly. The scheme we study corresponds to correlations coming first. In other words, the supervisor modulates neuronal excitability in order to introduce correlations into network activity. These correlations then generate the synaptic plasticity needed to learn a task through Hebbian synaptic modifications that are not themselves subject to direct supervision. We are interested in supervision through response modulation because it is easy to see how this scheme could be realized in cortical circuitry. The response modulations that we consider can be generated through standard excitatory and inhibitory synaptic input. Therefore, in this scheme, the supervisory circuit can guide learning through the feedback projection pathways that are characteristic of cortical circuitry, and no appeal must be made to as-yetundiscovered forms of modulation. It is important to realize that we are not proposing this scheme as an algorithmic improvement. Indeed, such indirect supervision of synaptic plasticity has disadvantages, and an important element of our study is to determine how detrimental these are. In summary, we consider a network in which synaptic plasticity is purely Hebbian, a form typically used in unsupervised learning applications. We ask whether it is possible to implement supervised learning in such a network solely by communicating error signals to the network along conventional excitatory and inhibitory feedback pathways that modulate neuronal responsiveness but do not directly affect synaptic plasticity. Such a scheme is not optimal, so its virtues are not efficiency or elegance. Rather, we take this minimalist approach so that we can determine whether these wellestablished elements of cortical circuitry provide a sufficient basis for implementing supervised learning. 2 Response Modulation and Synaptic Plasticity Neural networks used for supervised learning consist of units with nonlinear response functions connected together through interactions characterized by synaptic weights. The response ri of network unit i to an input Ii is

Supervised Learning Through Neuronal Response Modulation

611

typically determined by a sigmoidal function, ri =

1
. 1 + exp −gi (Ii − si )

(2.1)

In biophysical terms, this can be thought of as the normalized firing rate generated by an input current Ii . The parameter si , which we call the shift, controls the value of Ii at which ri reaches one-half its maximal value, while gi , which we call the gain, determines the slope of the firing rate versus input curve at this point. Input currents are typically computed by multiplying presynaptic responses by synaptic weight factors and summing over all inputs. The role of the supervisor is to compute an error by comparing actual and desired network output and to use this error to direct the modification of network parameters such that network performance improves. Conventionally, the major targets of this process are the synaptic weights. For example, weights can be modified to produce a stochastic gradient descent of the error function. We deviate from this procedure by employing a standard Hebbian synaptic modification rule that is not directly affected by the supervisor. At each stimulus presentation, the synaptic weight connecting unit i with response ri to unit a with response Ra , wai , is augmented by a term proportional to the product of the pre- and postsynaptic activities, wai → wai + w Ra ri ,

(2.2)

where the parameter w controls the learning rate. In addition, to prevent the runaway excitation that results from this positive feedback rule, divisive normalization is included. This consisted of dividing all the weights by factors that maintain the sums (for all a) N 

wai = α

(2.3)

i=0

at a constant value α. The important point here is that neither of the above rules, 2.2 or 2.3, involves the error function or any other form of supervisory signal. All the supervision in our network takes place at the level of the gain and shift parameters governing the input-output function of equation 2.1. It is not unusual for supervised learning schemes to modify such parameters, particularly shift parameters. Furthermore, in our scheme, supervision of shift and gain parameters takes place through the same type of stochastic gradient-descent procedure used in conventional supervised learning algorithms. The novel element in our approach is that these parameters are the only targets of supervised modification. The reason that we restrict supervision to the shift and gain parameters of neuronal response functions

612

C. Swinehart and L. Abbott

is that, unlike the supervision of synaptic plasticity, such supervision can be accomplished by ordinary, fast excitatory and inhibitory synapses from neurons of the supervisory circuit onto neurons of the function-approximation network. Changes in the shift variable si correspond to having the supervisor provide either net excitatory or net inhibitory input to network neuron i. It has been shown that balanced, parallel modulations of excitatory and inhibitory input can modify the gain of a postsynaptic neuron (Doiron, Longtin, Berman, & Maler, 2001; Chance, Abbott, & Reyes, 2002; Prescott & De Koninck, 2003), and on the basis of this result we argue that the supervisor can also control and modify the gain variable gi . In summary, the supervisory circuit in our model can modify both the shift and the gain variables for each of the neurons in the network (though in our examples, only the input neurons are modulated, and modulating the output units alone is not effective) through normal excitatory and inhibitory synaptic pathways. To reiterate what was said in section 1, our goal is not to introduce a new algorithm, but rather to see if existing algorithms can still operate when supervision is restricted to well-established cortical pathways. 3 Function Approximation We apply the proposed mechanism of supervised learning to function approximation, a well-studied task in the artificial neural network literature with obvious applications to biological systems (Poggio, 1990). In this task, network neurons are driven by a stimulus characterized by a single variable θ . The goal of learning is to produce a network output that matches a specified function or set of functions of θ. This is a very easy task for neural network learning that can be accomplished with a single layer of synapses modified, for example, by a delta learning rule (Widrow & Hoff, 1960). We consider this task because it allows us to illustrate clearly the features and limitations of the scheme we are studying. Specifically, we consider a two-layer feedforward network architecture with purely excitatory connections, as shown in Figure 1. The network consists of N input units, responding to a stimulus variable θ (which takes values in the range from 0 to 2π), that drive M output units. The input units of the network, indicated by the lower row of circles in Figure 1, are driven by currents that are gaussian functions of the difference between θ and a preferred stimulus value, which is different for each input unit. Specifically, the input to unit i, Ii , is given by Ii = G(θ − θi ) + G(θ − θi − 2π) + G(θ − θi + 2π ) ,

(3.1)

where

 2 θ G(θ) = 1.5 exp − − 0.5 . 2

(3.2)

Supervised Learning Through Neuronal Response Modulation

DjiejiAVnZg

613

hjeZgk^hdg

=ZWW^VcEaVhi^X^in

>cejiAVnZg

Jk`dlclj Figure 1: The function approximation network. Input units (lower row of circles) receive input tuned to the value of a stimulus variable θ, as indicated by the gaussian curves. The input units drive output units, shown at the top of the figure, through synaptic connections that are subject to Hebbian plasticity. A supervisor modifies the response properties of the input units through feedback projections. The task is to induce the firing rates of the output units to match specified functions of the stimulus variable.

The three terms appearing in equation 3.1 impose an approximate periodicity on the network, which is convenient (though not essential) because it removes edge effects. The values of the preferred stimulus parameters, θi for i = 1, 2, . . . , N, are uniformly distributed over the range from 0 to 2π . The response of input unit i to stimulus θ, ri (θ ), is given in terms of the input Ii by equation 2.1. The output of the network consists of the firing rates of the units appearing at the top of Figure 1. These are determined by the same firing-rate function as in equation 2.1, but their inputs are given by a weighted sum of the firing rates of the input units. Specifically, using Ra (θ ) to denote the response of output unit a (for a = 1, 2, . . . , M) to stimulus θ , Ra (θ) =

 1 + exp −ga



1 N 

 .

(3.3)

wai ri (θ) − sa

i=1

Here, wai is the weight of the connection from input unit i to output unit a. When we consider networks with a single output unit, we drop the output index and denote the weight from input i simply as wi . The goal of learning for this network is to match the outputs Ra (θ), as closely as possible, to a set of stimulus-dependent target functions Fa (θ).

614

C. Swinehart and L. Abbott

3.1 Action of the Supervisor. The supervisor in our network model computes an error by comparing the firing rates of the output units to the values of the target functions for each stimulus. It uses a stochastic gradientdescent algorithm to adjust the gain and shift values for the input units of the network of Figure 1 in such a way that the error, E(θ) =

M 1 (Ra (θ) − Fa (θ))2 , 2 a=1

(3.4)

is reduced after each stimulus presentation. Here, Ra is the response of output unit a, and Fa is the target response for that unit. As stated above, synaptic weights in the network are subject to Hebbian synaptic plasticity as described by equations 2.2 and 2.3, with w = 0.03 and α = 5.5. Error-based supervision is used to vary the shift and gain parameters (si and gi ) that control neuronal responsiveness. These are set to the initial values si = 1 and gi = 3 for all units, but are then changed by the supervised learning algorithm. During each run, a stimulus value θ is chosen randomly in the range from 0 to 2π, and the resulting output rates are computed. Then the shifts and gains for all the input units of the network are updated according to the rules si → si − s

∂E(θ) ∂si

and

gi → gi − g

∂E(θ) , ∂gi

(3.5)

where s and g are small parameters that control the rate of response modulation. For our simulations, these took the values s = g = 0.2/M. This process is repeated until performance stops improving. We could also adjust the corresponding parameters sa and ga for the output units, but for the examples we give, this is unnecessary. Instead, these have been held at their initial values sa = 1 and ga = 3 for all a. The adjustment of output shifts and gains is unnecessary in the examples we present because we have chosen parameters so that the mean of the output response, averaged across all stimuli, is equal to the stimulus average of the target function. This is not essential; it was done primarily to simplify the presentation. It is useful to compare and contrast our approach with the conventional use of the delta rule in this situation. In the conventional approach in which synaptic plasticity is supervised, the error in equation 3.4 is differentiated with respect to the synaptic weight wai . This weight is then updated according to the rule (assuming a gain of one) wai → −w

∂E = w (Fa − Ra ) Ra ri , ∂wai

(3.6)

where Ra stands for the derivative of the response of output unit a with

Supervised Learning Through Neuronal Response Modulation

H]^[i

9%


:%

8dbW^cZY

=`i`e^IXk\

8%

615





=`i`e^IXk\



Figure 2: Effects of shift and gain modulations on firing rates and response tuning curves. The upper plots show neural responses as a function of the input current I, and the lower plots show them as a function of the stimulus parameter θ. (A) Changing the shift variable slides the response current curve to the left or right and moves the tuned response up and down. (B) Changing the gain variable changes the slope of the response current curve and has a roughly multiplicative effect on the tuned response. (C) Changing both variables changes the width of the tuned response.

respect to its input current. The term (Fa − Ra )Ra can be thought of as an error signal sent to output unit a that, in conjunction with the presynaptic firing rate ri , controls modification of the weight wai . In contrast, the error signals in our scheme, given by the derivatives in equation 3.5, are “sent” to the input units of the network rather than to the output units. Furthermore, these guide the modification of parameters affecting neuronal responses, not synaptic weights. Although the supervised learning rules in equations 3.5 and 3.6 may look similar in terms of mathematical abstraction, we stress that the modification described by equation 3.5 can be generated by normal, ionotropic synaptic transmission from the supervisory circuit to the targeted neuron, whereas those of equation 3.6 cannot. This is why we are considering such a modified form of delta rule learning. The ability to change both the shift and gain variables that determine neuronal excitability provides considerable flexibility in modulated neuronal responsiveness. The different effects of shift and gain modulations on the firing rate of a model neuron, both as a function of its input current and of the stimulus variable, are shown in Figure 2. Changing the

616

C. Swinehart and L. Abbott

shift parameter translates the firing-rate curve right and left or, plotted as a function of the stimulus variable, shifts the tuning curve up and down (see Figure 2A). Changing the gain variable modifies the slope of the firingrate curve and modulates the firing-rate tuning curves in a roughly multiplicative manner (see Figure 2B). Adjusting both variables allows the width of the tuning curve to be changed without an iceberg effect (see Figure 2C). 4 Results In studying supervised learning through response modulation, we separately consider networks with a single output unit and networks with multiple output units. Obviously, the case of a single output unit provides less of a challenge than multiple outputs to any learning algorithm. Nevertheless, we consider it here because it provides a clear example of the interaction between supervised response modulation and unsupervised synaptic plasticity. We begin by showing that supervised response modulation, acting by itself without any accompanying synaptic plasticity, leads to a solution of the function approximation problem with a single output unit. This has some implications for network switching. However, it does not provide a satisfactory long-term solution because the supervisor-induced modulations do not produce any permanent changes in the network. This means that the task can be performed only, even after learning, with continuous input from the supervisor. This problem is resolved by adding Hebbian synaptic plasticity to the learning scheme. This allows the supervisor-induced modulations to be transferred into changes of synaptic strength. Ultimately, this transfer allows the network to function properly even in the absence of supervisory input. Supervised learning through response modulation is more difficult in networks with multiple output units. In the multi-output case, situations often arise in which response modulation, acting without synaptic plasticity, cannot solve the function approximation task. As an example, consider an input unit that projects to two output units that are supposed to represent two different functions. For one of these functions, it might be appropriate to enhance the response of this input unit, while for the other, it may be necessary to decrease its responsiveness. Clearly, without access to the separate synapses that connect this single input unit to its multiple output targets, both of these criteria cannot be satisfied. In such situations, Hebbian plasticity does not merely act as a way of transferring supervisory modulation into permanent network changes; it must act in concert with response modulation for the task to be learned at all. This is indeed what happens. We find that a combination of supervised response modulation and unsupervised synaptic plasticity allows networks with multiple outputs to compute multiple functions, provided that the connection probability between the input and output layers is less than about 95%.

Supervised Learning Through Neuronal Response Modulation

617

4.1 Networks with a Single Output Unit. 4.1.1 Learning and Switching Through Response Modulation. Because response modulation is the pathway through which supervision affects network responses in our studies, it is useful to start off by considering what happens when response modulation acts alone, without the Hebbian synaptic plasticity that will be added later. Therefore, we begin the study of networks with a single output unit by showing that function approximation can be accomplished solely on the basis of response modulation. For Figure 3, synaptic connection strengths were held fixed, while a gradient-descent supervisor varied the shifts and gains of the input units. In other words, we used equation 3.5 but not equation 2.2 during learning. Figure 3A shows the initial state of the network in which the output response is independent of the stimulus (upper panel), because all the input units have identical shifts and gains, as revealed by the identically shaped response curves in the lower panel. After the gradient-descent response modulation algorithm has acted, the output response matches the target function (upper panel of Figure 3B) due to the modulation of responses revealed by the modified response curves seen in the lower panel. To match the cosine-like target response, the input units selective for stimuli near zero and 2π have been upregulated by the supervisor, while those selective for stimuli near π have been downregulated. Using supervised response modulation, the network can approximate a wide variety of functions (some examples are shown, along with the distributions of shift and gain values that produce them, in Figure 4). It is important to keep in mind that the distributions of shift and gain variables shown in the left column of this figure could arise from specific excitatory and inhibitory inputs generated by a supervisor circuit. Thus, each function computed by the network corresponds to a specific pattern of activity within the hypothetical supervisory circuit. If these patterns of activity are remembered and later recreated within the supervisor circuit, this will induce the function approximation network to compute the target function related to that pattern of activity. Thus, after learning has taken place, the supervisor can act as a controller, rapidly switching the input-output relationship of the function approximation network between prelearned states. Although we do not consider this form of switching further in this article, it provides an interesting mechanism by which one neural circuit can control, activate, and switch the function of another (for a related discussion, see Lukashin, Wilcox, & Georgopoulos, 1994). In this network, the input unit responses act as basis functions for representing the output response. Because they do not provide a complete set for arbitrarily high frequencies, there are limits to the types of functions that can be accurately approximated. Limitations arise when the target function varies rapidly, as seen in Figure 4. Although these limitations exist, they are less severe than they would be in a function approximation network

618

C. Swinehart and L. Abbott

FlkglkIXk\

&

'

2

(2

&

$.

$.

$,

$,

$*

$*

$(

$(

&

6[iZgBdYjaVi^dc

+

+ &

@eglkIXk\j

'

>c^i^VaHiViZ

'

2

(2

&

2

(2

2

(2

Figure 3: Function approximation by supervised response modulation acting alone without synaptic plasticity. Upper curves show the response of the single output unit to various stimulus values (dots) and the target function (line). The lower panels show a sampling of the responses of the 230 input units as a function of the stimulus value. (A) State of the network before learning. All input responses have the same shifts and gains, and the output is independent of the stimulus. (B) State of the network after learning. The output unit responses match the target function due to modulation of the input unit responses.

that relied solely on synaptic modification. This is because the tuning curve narrowing seen in Figure 2C can somewhat ameliorate problems with approximating rapidly varying functions. In the following examples, we choose to approximate sinusoidally varying functions and do not present examples with other types of functions. All the networks shown can produce equivalent results with any target functions for which the input responses provide an adequate basis. 4.1.2 Transfer of Learning to Synapses. In the previous section, we considered supervised response modulation acting alone. We now add to this a Hebbian plasticity mechanism. In other words, we now use both equation 3.5 and equation 2.2 during learning. The combined effect of supervised response modulation and unsupervised synaptic plasticity is illustrated in Figure 5. As before, the network is initialized with uniform weights and all shifts and gains set to the same values. The supervisor then modifies response properties to minimize the output error. Early on during the learn-

Supervised Learning Through Neuronal Response Modulation >ceji;^g^c\GViZh

'

IXk\

J_`]k

'

'

;jcXi^dc6eegdm^bVi^dc

IXk\

BdYjaVi^dcHiViZ

8%

619

+

+

$.

>X`e

9%

-$,'

-$.

>X`e

.$&

&

2

(2

IXk\

J_`]k

>X`e

-$,(

&

&

2

(2

2

(2

'

+

-$,

(2

+

'

$.

2

IXk\

-$,

& '

+

'

$,

(2

IXk\

J_`]k

:% '$(

2

'

'

&

&

IXk\

-$,&

+

2

(2

&

Figure 4: Examples of learning through response modulation. Three different functions are approximated by response modulation. The left column shows the shift and gain variables for all 460 of the input units of the network, each dot representing one input unit. Gains and shifts can also be seen in the sampling of input unit response curves in the middle column. The right column shows the output unit response (dots) and target function (line) plotted against the stimulus value.

ing process (top row of plots), the performance of the network relies almost entirely on the response modulation of the input units produced by the supervisor (illustrated by the distribution of input responses in the top row, left panel). At this point, the weights have hardly changed from their initial values (as seen in the top row, center panel). However, as the simulation progresses, the weight changes become progressively larger (second row, center panel), and the response modulations become progressively smaller (second row, left panel). Ultimately, the weights take on the cosine shape of the target function (third row, center panel), and the responses are almost uniform for all the input units (third row, left panel), as they were at the beginning of the learning process. Note that a stable equilibrium is reached when Hebbian modification and response modulation act together. Once the response modulation and Hebbian plasticity have equilibrated, the supervisory input can be removed altogether, returning all shifts and

620

C. Swinehart and L. Abbott

>ceji;^g^c\GViZh

( '

$)

2

(2

(2

& (

(2

&

2

(2

2

(2

2

(2

IXk\

&

IXk\

+

2

Gi\]\ii\[

(2

& '

N\`^_k

IXk\

2

(2

'

'

n`k_flk $) jlg\im`jfi

&

(2

N\`^_k

IXk\

(2

$,

;^cVa

2

Gi\]\ii\[

'

2

2

+

(

$) &

&

& '

N\`^_k

IXk\

2

$,

AViZ

(2

'

$) &

2

Gi\]\ii\[

(

$,

B^Y

&

+

IXk\

&

IXk\

IXk\

:Vgan

;jcXi^dc6eegdm^bVi^dc

'

N\`^_k

$,

LZ^\]ih

+

2

Gi\]\ii\[

(2



Figure 5: Supervised response modulation along with unsupervised synaptic plasticity allows for the transfer of learning to the synapses. The left column shows a sampling of the responses of the 230 input units as a function of the stimulus value. The middle column depicts the synaptic weights of the network plotted as a function of the preferred stimulus value for the presynaptic neuron. The right column shows the responses of the output unit (dots) and the target function (line), plotted against the stimulus value. The top three rows of plots, from top to bottom, show the gradual transfer of learning as the network changes from relying primarily on response modulation (top row of plots) to relying primarily on the pattern of modified synaptic weights (third row of plots). The bottom row illustrates that the network can perform fairly well even when response modulation is totally eliminated, once the appropriate pattern of synaptic weights has been established.

Supervised Learning Through Neuronal Response Modulation

621

gains to their default values, and yet the network can still generate a good approximation of the target function (bottom row of Figure 5) (although, for stability, this necessitates the deactivation of Hebbian plasticity). Unsupervised synaptic plasticity thus allows the supervisor to contribute progressively less as the burden of representing the target function is taken up by the synapses. Supervised response modulation plays three critical roles in guiding the Hebbian development of synapses capable of performing the function approximation task. First, because supervised response modulation acting alone can solve the task, the supervisor can act through the input units to effectively clamp the output to the correct response profile while Hebbian plasticity is taking place. Second, by increasing the responsiveness of appropriate input units while clamping the output to the target function, supervised response modulation sets up the appropriate pattern of correlation across the synapses of the network to guide Hebbian modification. For example, input units that are important contributors to the correct output response will be pushed to high levels of responsiveness by the supervisor, enhancing their correlations with the correctly clamped output unit. This causes the synapses connecting such units to the output to grow rapidly. Input units not needed for the task will be made unresponsive by the supervisor, so their synapses to the output unit will not be enhanced by the Hebbian modification rule. Instead, these synapses will be weakened due to the synaptic normalization constraint. Finally, we consider a third role for supervised response modulation on the basis of an analysis of Hebbian modification. The form of synaptic plasticity we are using, Hebbian synaptic modification in conjunction with divisive normalization, ultimately sets synaptic weights in this case equal to αFri  wi =  , Frj 

(4.1)

j

where Fri  =

1 2π



2π

dθ F(θ)ri (θ) .

(4.2)

0

To simplify the analysis, we consider a linear approximation for the response function of the output unit rather than the full sigmoidal form of equation 3.3. In this case and for these weights, the condition that the output response matches the target function, R=

N  i=0

wai ri = F ,

(4.3)

622

C. Swinehart and L. Abbott

requires that α

N  i=1

ri (θ)ri (θ  ) = δ(θ − θ  )

N  Frj  .

(4.4)

j=0

The third role of supervised response modulation is to make this equation as near to an equality as possible. The accuracy with which the sum on the left side of this equation can match a δ function profile depends on the narrowness of the tuning curves of the input units. This places a limit on the degree to which rapidly varying target functions can be reproduced, but modifications in the gain and shift variables can improve this situation by narrowing the input tuning curves. More important, supervised response modulation acts to ensure that the normalization condition implied by equation 4.4 is met, and this is what ultimately allows Hebbian plasticity to solve the problem (Salinas & Abbott, 2000). Thus, by acting on these multiple levels, supervised response modulation guides Hebbian plasticity to a solution of the function approximation task. 4.2 Networks with Multiple Output Units. Networks with multiple output units present a greater challenge to the form of supervised learning we are proposing than do single-output networks. In particular, situations frequently arise where the representation of different functions by different output units cannot be achieved by response modulation alone due to shared input. Two cases are simple to analyze. If the connectivity between the input and output units is all-to-all with equal weights, response modulation alone is clearly unable to produce different responses in the output units. With all-to-all coupling, all the output units receive the same total drive, and whatever modulation is done at the input level affects all of the output units in the same way. Unsupervised synaptic plasticity does not help because the input correlation structure seen by the synapses to each output unit is identical, so the synapses will all be modified in an identical manner. Basically, the problem with all-to-all coupling is symmetry; all the output units are equivalent, and supervised modulation of input responses is not sufficient to break this symmetry and allow the output units to respond differently to the stimulus. Although we have assumed that the synapses take identical values, setting the initial synaptic weights to different values does not fix this problem. At the opposite extreme, if the coupling from input to output units is so sparse that each input unit projects to just a single output unit, the situation reduces to multiple copies of the single-output case, and the analysis becomes a trivial extension of what was done in the previous section. In this section, we consider intermediate cases where the input-to-output connectivity is not all-to-all, but there is nevertheless considerable overlap in the input to different output units. We start by considering the case of twooutput units and construct networks with various amounts of overlap in

Supervised Learning Through Neuronal Response Modulation

GZhedchZBdYjaVi^dc6adcZ 

+

+



2

2



GBl^i]=ZWW^VcAZVgc^c\ 

.*d[^cejihh]VgZY

+

IXk\

IXk\

IXk\



GZhedchZBdYjaVi^dcl^i] =ZWW^VcAZVgc^c\

623

2

2



2

2

Figure 6: Unsupervised synaptic plasticity allows learning to succeed where interference causes supervised response modulation, acting without synaptic plasticity, to fail. In the left panel, a network of 460 input units connected with 88% overlap to two output units was simulated with response modulation acting without synaptic plasticity. The responses of the two output units, denoted by filled and open dots, failed to match the two target functions, indicated by dashed and solid curves, as a function of the stimulus value. When Hebbian plasticity was included, the two output responses accurately matched the target functions (middle panel). The combination of supervised response modulation and Hebbian plasticity continued to produce accurate outputs until the common input to the output units was increased to 95% (right panel).

the projections they receive from the input units. For an overlap of q, the number of input units that project to both output units is qN, and the number that project to only a single output unit is (1 − q)N. We ask whether, in such cases, unsupervised synaptic plasticity can exploit small differences in the drive to each output unit to break the symmetry and allow the output units to represent different functions. Figure 6 illustrates the ability of unsupervised synaptic plasticity to play the role of a symmetry-breaking mechanism. In the first panel, supervised learning acting without synaptic plasticity has set the shifts and gains to their optimal values, but due to the degree of interference caused by shared input, the approximation is quite poor, and neither target function has been matched. The network has essentially split the difference between the two functions, with only small disparities between the responses of the two output units. However, when Hebbian plasticity is activated, it is able to exploit and amplify these small differences to improve performance dramatically. This ultimately leads to a match of the two different target functions (see the center panel of Figure 6). At this point, the supervisory input is no longer necessary (provided that the Hebbian process is halted). Thus, the combination of supervised response modulation and unsupervised synaptic plasticity allows the network to perform this task at a level that could not be achieved via response modulation alone. The problem of indirectly supervising the plasticity of NM synapses by modulating only N neurons might at first appear to be a crippling limitation

624

C. Swinehart and L. Abbott

of response modulation. The example of Figure 6 shows at least one case in which this problem is not nearly as severe as might have been imagined. Separation of the two output units could still be achieved when they shared up to 95% of their inputs. However, it is critical to the success of supervision by response modulation that the requirement of a unique component for the input to each output unit scale appropriately as the size of the network and the number of output units increase. Initially, we investigate this issue by varying the amount of shared input in a network with two output units. The result of varying the proportion of shared inputs in two-output networks of different sizes, when both supervised response modulation and unsupervised synaptic plasticity are active, is shown in Figure 7. In this figure and in Figure 8, performance is quantified by computing the error of equation 3.4, divided by the number of output units. The left panel of this figure shows that when learning occurs via response modulation alone without synaptic plasticity, errors begin to grow once the proportion of shared inputs exceeds 50% (dashed curves in Figure 7A). Performance is virtually identical for different input population sizes. With Hebbian plasticity included, the required number of unique inputs decreases dramatically, with little error accumulating until 90% to 95% of inputs are shared (solid curves in Figures 7A and 7B). The point of the transition from small errors to large errors appears to be roughly the same for all the network sizes studied (see Figure 7B). The main effect of increasing the number of input units is to make the transition point, where the function approximation network fails, sharper. This suggests that a discontinuous phase transition occurs at a critical percentage of about 94% shared input in the N → ∞ limit. We now extend these results to networks with more than two output units. In this case, the proportion of shared inputs (q) is not appropriate for describing all the different possibilities for sharing projections from the input units. Instead, we use the connection probability (p) to characterize the networks we study. To construct these networks, we introduce a connection between any one of the N input units and any one of the M output units with probability p. If such a connection is formed, it is subject to Hebbian plasticity. If no connection forms during this stochastic initial wiring, the connection remains absent for the entire duration of the simulation. The connection probability controls the sparseness of the network in that small values of p correspond to sparse connectivity. Figure 8 shows function approximation errors for networks with different numbers of output units as a function of the connection probability p, for two sizes of input unit populations. In this case, both response modulation and synaptic plasticity are activated. As in the two-output case, interference does not become a serious impediment to learning until p reaches the .9 to .95 range, indicating that truly unique inputs are not necessary. Rather, the requirement is a certain degree of sparseness. Also noteworthy is the fact that the output population size can approach one-third of the total number

Supervised Learning Through Neuronal Response Modulation

$+

'

?dfkjFefkbWj_edI_p[

$-

D\Xe=`eXc
$-

D\Xe=`eXc
&

'(. (+& +&& '&&& (&&& *&&&

$+

CeZkbWj_ed CeZkbWj_ed>[XX

625

?dfkjFefkbWj_edI_p[

'(. (+& +&& '&&& (&&& *&&&

CeZkbWj_ed>[XX

$)

$)

$'

$' $+

$,

$-

$.

$/

Gifgfik`fef]@eglkjJ_Xi\[

'

$.

$.*

$..

$/(

$/,

Gifgfik`fef]@eglkjJ_Xi\[

'

Figure 7: Error as a function of the proportion of shared input for a two-output network with different numbers of input units. Dashed lines show the mean error per output unit resulting from supervised response modulation without synaptic plasticity. Solid lines are from runs that also employed Hebbian plasticity. Individual lines correspond to different numbers of input units (N). The functions being approximated are cosine and sine. (A) Network performance degrades as the proportion of inputs projecting to both outputs increases. (B) Detailed view of the results for supervised response modulation with Hebbian plasticity shown in panel A.

of inputs before performance begins to suffer from interference (provided p values are not too large). Our results indicate that connection probability is the dominant factor that controls whether a network with multiple output units, using both supervised response modulation and Hebbian plasticity, can function properly. The required sparseness in the connectivity is not stringent. Furthermore, approximation of multiple functions is possible even when output population size is a significant fraction of the total input population size. The analysis of networks with multiple output units is more difficult than in the single-output case, but some of the same basic principles apply. In this case, the supervisor cannot clamp the output units to their target functions, but the existence of even a small number of symmetry-breaking synapses is sufficient to break this impasse. These synapses initially get quite strong and drive the output units away from the degenerate state in which they are all the same, which starts off the combined response modulation Hebbian learning process. Through this process, the bulk of the synapses ultimately come to obey the multi-output generalization of equation 4.1, αFa ri  wai =  . Fa rj 

(4.5)

j

Similar to the result in the one-output case and making the same linear

626

HbVaa>cejiEdejaVi^dc-4)'' EkfkjFefkbWj_edI_p[

+ '& (& )& +& -&

$+

9%

$-

D\Xe=`eXc
$-

D\Xe=`eXc
8%

C. Swinehart and L. Abbott

AVg\Z>cejiEdejaVi^dc-4+'' EkfkjFefkbWj_edI_p[

$+

$)

+ '& (& )& +& -+

$)

$'

$'

$-)

$.)

$..

$/)

:fee\Zk`feGifYXY`c`kp2

$/. '

$-)

$.)

$..

$/)

:fee\Zk`feGifYXY`c`kp2

$/. '

Figure 8: Error as a function of network connection probability for different numbers of output units. The functions being approximated are cosines with phases ranging from 0 to 2π, equally spaced between the output units. (A) A network with 200 input units. Error increases rapidly for p > 0.93 regardless of the number of output units. For 50 and 70 output units, the error is larger for all p values. (B) For a population of 400 input units, the results are similar, except that overall performance only degrades when there are 75 output units.

approximation, matching of the target function with these synaptic weights requires that α

N  i=1

ri (θ)ri (θ  ) = δ(θ − θ  )

N  Fa rj  .

(4.6)

j=0

Subject to the same constraints on the approximation of the δ function, these equations represent M constraints that need to be satisfied by appropriate adjustment of the 2N shift and gain variables of the input units, which should be possible to satisfy when M < N. The key to making the combined response modulation synaptic plasticity scheme work is that Hebbian modification reduces the problem of setting pNM synaptic weights to the problem of satisfying the M constraints appearing above, and this can be done by the supervision through its control of the 2N shift and gain variables. 4.3 A Stochastic Supervisor. The supervisor used in the simulations discussed thus far employed a gradient-descent algorithm to modify intrinsic response properties on the basis of the error generated by each stimulus. A biological supervisor circuit is more likely to operate under a reinforcementbased scheme. As a first attempt at constructing such a supervisor, we have implemented a model using a stochastic search guided only by a reward signal that reflects network performance. Related ideas have been applied to the supervision of synaptic plasticity (Barto, Sutton, & Anderson, 1983; Mazzoni, Anderson, & Jordan, 1991; Jabri & Flower, 1992; Williams, 1992;

Supervised Learning Through Neuronal Response Modulation

&

)

'

[dZ

627

'

[dZ

IXk\

J_`]k

( ' &

+

ijWhj [dZ

($+

)

>X`e

)$+

&

*

2# Jk`dlclj

(2#

Figure 9: Learning under the random walk supervisor. (A) The paths in modulation space of three input units controlled by the supervisor. All three units started with the same shifts and gains (marked Start), but these then diverged as the supervisor found values that accomplished the function approximation task (End). (B) The end result is a good approximation (dots) of the target function (line) by the output unit as a function of stimulus value. In this simulation, 230 input units drove a single output unit.

Cauwnberghs, 1993; Doya & Sejnowski, 1995; O’Reilly, 1996; Xie & Seung, 2004; Seung, 2003). For stochastic reward-based supervision, two N-dimensional “modification” vectors, vs and vg , of unit length were generated randomly—one for shifts and one for gains. For all i values, the shift and gain of unit i was incremented by an amount proportional to component i of the appropriate modification vector, si → si + vsi

and

g

gi → gi + vi .

(4.7)

Simulations were divided into epochs of 20 stimulus presentations and error evaluations. After each epoch, the sum of the 20 errors was compared to the summed error from the previous epoch. If this total error was less than it was previously, the modification vectors were left unchanged. If the summed error increased from the previous epoch, new modification vectors, vs and vg , were generated randomly. In either case, the resulting modification vector was then used to further increment the shifts and gains, as described above. In this study of random walk learning through response modulation, we do not include any Hebbian synaptic modification. This strategy has the effect of steadily, although slowly, reducing the average error. The paths through modulation space of three input units over the course of a run are plotted in Figure 9A. The improvement in performance can be seen in the reduction in the lengths of the line segments seen in the traces. At the beginning of the run, most of these segments are relatively long as the network makes coarse adjustments to approach

628

C. Swinehart and L. Abbott

the target function. Later, more frequent trajectory changes appear as the network approaches a solution and makes fine adjustments. Figure 9B illustrates that this crude strategy is capable of solving the task, given enough time (in this case 600 iterations). Thus, a random walk supervisor strategy that requires much less information and algorithmic sophistication than gradient descent can, at least in simple cases, provide adequate supervision. There are clearly severe limitations on the sizes of networks that can be trained by this random walk algorithm. As the network grows in size, the algorithm gets prohibitively slow. In section 5, we propose ways that this problem might be addressed to achieve better scaling of performance with network size. 5 Discussion The novel feature of the supervised learning scheme we have proposed is that supervision takes place at the level of neuronal responsiveness rather than synaptic plasticity. Two apparent disadvantages of this scheme—that it does not lead to permanent network modification and that it severely limits the number of elements being supervised—appear to be far less severe than might have been imagined at first. By guiding synaptic plasticity that is otherwise unsupervised, supervised response modulation can lead to permanent changes that allow a network to operate effectively, even in the absence of supervision. Furthermore, Hebbian plasticity can take advantage of small inhomogeneities in randomly coupled networks to allow independent changes in synapses to output units that share presynaptic input. Given that it works, there are some potential advantages of supervising neuronal excitability rather than synaptic plasticity. First, supervision can occur through ordinary feedback projections that can act rapidly and can target individual neurons independently. Additional advantages concern the nature of the supervisory circuit. We have not attempted to construct a realistic model of this circuit, but we envision it as a network capable of maintaining a continuum of stable, self-sustained patterns of activity (Compte, Brunel, Goldman-Rakic, & Wang, 2000; Seung, Lee, Reis, & Tank, 2000). Such networks tend to drift, especially if provided with noisy input. Thus, it might be possible to implement the random walk supervisor as a network with self-sustained activity and random drift, with the rate of drift controlled by noisy inputs that are suppressed by reward. Whatever the form of the supervisory circuit, modulating neuronal responsiveness instead of synaptic plasticity has a number of tactical advantages. We considered two approaches to supervision: gradient descent, which involves more information and mathematical analysis than we would expect from a neural circuit, and a random walk model that uses less. A real circuit should lie somewhere between these extremes. From the point of

Supervised Learning Through Neuronal Response Modulation

629

view of the supervisor, the fact that there are far fewer neurons than synapses to supervise changes from a disadvantage to an advantage. The supervisor must search in the space of the parameters it is modifying for a solution to the problem at hand. By reducing the dimension of this space, supervision of neuronal responses, provided that it works (and we have shown that it does), is far easier than supervision of more numerous synapses. Another advantage of supervising neuronal responses is that the supervisor can monitor the activities that it is modulating in a way that is impossible with supervised synaptic plasticity. It is almost inevitable that the function approximation network, which receives input from the supervisor circuit, would also send projections to it. Such reciprocal connectivity is a typical feature of neuroanatomy. These projections allow the supervisor to monitor the activity of the units it is supervising and use this information to guide learning. For example, this information could be used to reduce the dimension of the space in which the supervisor must search for solutions of the task being learned. Consider, for example, two input units in the function approximation network that have almost totally overlapping response profiles. It is rather wasteful for the supervisor to vary the response properties of these two neurons independently, and yet this is what was done in the random walk model we studied. A more “intelligent” supervisor would use information about the correlations between the units it is modulating to find strategies that are most likely to produce large changes in the network being supervised, and to avoid wasting time generating modulations that have little effect. Thus, projections from the supervised units to the supervisor could be part of a secondary modulatory process that allows the supervisor to learn about learning. Acknowledgments This research was supported by the National Science Foundation (IBN0235463) and NSF-DGE-9972756 and the Sloan-Swartz Center for Theoretical Neurobiology at Brandeis University. References Barto, A. B., Sutton, R. S., & Anderson, C. W. (1983). Neuron-like adaptive elements that can solve difficult learning control problems.IEEE Trans. on Systems, Man and Cybernetics, 13, 834–846. Cauwnberghs, G. (1993). A fast stochastic error-descent algorithm for supervised learning and optimization. In Col. Giles, S. J. Hanson, & J. D. Cowan (Eds.), Advances in neural information processing stystems, 5 (pp. 244–251). San Mateo, CA: Morgan Kaufmann. Chance, F. S., Abbott, L. F., & Reyes, A. D. (2002). Gain modulation through background synaptic input. Neuron, 35, 773–782.

630

C. Swinehart and L. Abbott

Chauvin, Y., & Rumelhart, D. E., (Eds.) (1995). Back propagation: Theory, architectures, and applications. Hillsdale, NJ: Erlbaum. Compte, A., Brunel, N., Goldman-Rakic, P. S., & Wang X. J. (2000). Synaptic mechanisms and network dynamics underlying spatial working memory in a cortical network model. Cereb. Cortex, 10, 910–923. Dayan, P., & Abbott, L. F. (2001). Theoretical neuroscience: Computational and mathematical modeling of neural Systems. Cambridge, MA: MIT Press. Doiron, B., Longtin, A., Berman, N., & Maler, L. (2001). Subtractive and divisive inhibition: Effect of voltage-dependent inhibitory conductances and noise. Neural Comput., 13, 227–248. Doya, K., & Sejnowski, T. J. (1995). A novel reinforcement model of birdsong vocalization learning. In G. Tesauro, D. Touretzky, & T. Leen (Eds.) Advances in nueral information processing systems, 7 (pp. 101–108) Cambridge, MA: MIT Press. Hertz, J., Krogh, A., & Palmer, R. G. (1991). Introduction to the theory of neural computation. Redwood City, CA: Addison-Wesley. Jabri, M., & Flower, B. (1992). Weight perturbation—an optimal architecture and learning technique for analog VLSI feedforward and recurrent multilayer networks. IEEE Trans. on Neural Networks, 3, 154–157. Lukashin, A. V., Wilcox, G. L., & Georgopoulos A. P. (1994). Overlapping neural networks for multiple motor engrams. Proc. Natl. Acad. Sci. U.S.A., 9, 8651– 8654. Mazzoni, P., Andersen R. A., & Jordan M. I. (1991). A more biologically plausible learning rule for neural networks. Proc. Natl. Acad. Sci. U.S.A., 88, 4433–4437. Miller, K. D. (1996). Receptive fields and maps in the visual cortex: Models of ocular dominance and orientation columns. In E. Domany, J. L. van Hemmen, & K. Schulten (Eds.), Models of neural networks (Vol. 3, pp. 55–78). New York: Springer-Verlag. O’Reilly, R. C. (1996). Biologically plausible error-driven learning using local activation differences: The generalized recirculation algorithm. Neural Computation, 8, 895–938. Poggio, T. (1990). A theory of how the brain might work. Cold Spring Harbor Symposium on Quantitative Biology, 55, 899–910. Prescott, S. A., & De Koninck, Y. (2003). Gain control of firing rate by shunting inhibition: Roles of synaptic noise and dendritic saturation. Proc. Natl. Acad. Sci. U.S.A., 100, 2076–2081. Salinas, E., & Abbott, L. F. (2000). Do simple cells in primary visual cortex form a tight frame? Neural Comp., 12, 313–336. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593–1599. Seung, S. (2003). Learning in spiking neural networks by reinforcement of stochastic synapatic transmission. Neuron, 40, 1063–1073. Seung, H. S., Lee, D. D., Reis, B. Y., Tank, D. W. (2000). Stability of the memory of eye position in a recurrent network of conductance-based model neurons, Neuron 26, 259–271. Widrow, B., & Hoff, M. E. (1960) Adaptive switching circuits. WESCON Convention Report, 4, 96–104.

Supervised Learning Through Neuronal Response Modulation

631

Widrow, B., & Stearns, S. D. (1985). Adaptive signal processing. Englewood Cliffs, NJ: Prentice Hall. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229–256. Xie, X., & Seung, S. (2004). Learning in neural networks by reinforcement of irregular spiking. Physical Review, 69, 041909. Received October 2, 2003; accepted July 28, 2004.

LETTER

Communicated by Bard Ermentrout

The Combined Effects of Inhibitory and Electrical Synapses in Synchrony Benjamin Pfeuty [email protected] Neurophysique et Physiologie du Syst`eme Moteur, Universit´e Ren´e Descartes, 75270 Paris Cedex 06, France

Germ´an Mato [email protected] Comisi´on Nacional de Energ´ıa At´omica and CONICET Centro At´omico Bariloche and Instituto Balseiro, R. N.,Argentina

David Golomb [email protected] Department of Physiology and Zlotowski Center for Neuroscience, Faculty of Health Sciences, Ben Gurion University of the Negev, 84105, Be’er-Sheva, Israel

David Hansel [email protected] Interdisciplinary Center for Neural Computation, Hebrew University, 91904 Jerusalem, Israel, and Neurophysique et Physiologie du Syst`eme Moteur, Universit´e Ren´e Descartes, 75270 Paris Cedex 06, France

Recent experimental results have shown that GABAergic interneurons in the central nervous system are frequently connected via electrical synapses. Hence, depending on the area or the subpopulation, interneurons interact via inhibitory synapses or electrical synapses alone or via both types of interactions. The theoretical work presented here addresses the significance of these different modes of interactions for the interneuron networks dynamics. We consider the simplest system in which this issue can be investigated in models or in experiments: a pair of neurons, interacting via electrical synapses, inhibitory synapses, or both, and activated by the injection of a noisy external current. Assuming that the couplings and the noise are weak, we derive an analytical expression relating the cross-correlation (CC) of the activity of the two neurons to the phase response function of the neurons. When electrical and inhibitory interactions are not too strong, they combine their effect in a linear manner. In this regime, the effect of electrical and inhibitory interactions when combined can be deduced knowing the effects of each of the interactions separately. As a consequence, depending on intrinsic neuronal properNeural Computation 17, 633–670 (2005)

c 2005 Massachusetts Institute of Technology


634

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

ties, electrical and inhibitory synapses may cooperate, both promoting synchrony, or may compete, with one promoting synchrony while the other impedes it. In contrast, for sufficiently strong couplings, the two types of synapses combine in a nonlinear fashion. Remarkably, we find that in this regime, combining electrical synapses with inhibition amplifies synchrony, whereas electrical synapses alone would desynchronize the activity of the neurons. We apply our theory to predict how the shape of the CC of two neurons changes as a function of ionic channel conductances, focusing on the effect of persistent sodium conductance, of the firing rate of the neurons and the nature and the strength of their interactions. These predictions may be tested using dynamic clamp techniques. 1 Introduction Electrical synapses have long been known to exist in invertebrates (Watanabe, 1958; Furshpan & Potter, 1959), but only recently has evidence of their ubiquity been unequivocally found in the mammalian brain. Remarkably, electrical synapses in the CNS are frequently found to connect GABAergic interneurons. This is, for instance, the case in the striatum (Kita, Kosaka, & Heizmann, 1990), the hippocampus (Venance et al., 2000), the cerebellum (Mann-Metzer & Yarom, 1999), the reticular thalamic nucleus (Landisman et al., 2002), and the neocortex (Gibson, Beierlein, & Connors, 1999; Galarreta & Hestrin 1999; Fukuda & Kosaka, 2000). In the neocortex, electrical as well as inhibitory synapses exist between fast spiking (FS) interneurons (Gibson et al., 1999; Galarreta & Hestrin, 2002) or between multipolar bursting neurons (Blatow et al., 2003). In contrast, low threshold spiking (LTS) interneurons interact mostly via electrical synapses, the inhibitory synapses between them being sparse (Gibson et al., 1999; Amitai et al., 2002). FS and LTS neurons are interacting reciprocally via inhibition but not via electrical synapses. Recent experiments have shown that electrical synapses may be involved in synchronizing neural activity. Blocking of inhibition and excitation does not reduce the synchronization of interneurons in the molecular layer of the cerebellum between which electrical synapses have been identified (MannMetzer & Yarom, 1999). In slices of neocortex, in which LTS cells are activated by ACPD, the synchronous activity of LTS remains after blocking inhibition (Beierlein, Gibson, & Connors, 2000). Reciprocally, blockade of gap junctions through pharmacological agent (Draguhn et al., 1998; PerezVelazquez, Valiante, & Carlen, 1994; Traub et al., 2001; Friedman & Strowbridge, 2003) or genetic manipulations (Deans, Gibson, Sellitto, Connors, & Paul, 2001; Hormuzdi et al., 2001; Blatow et al., 2003) reduces the synchronized activity in cortex or in hippocampus. Other studies suggest that electrical synapses alone are not sufficient for getting synchronous activity but that they are necessary. This has been shown in preparations of cortical slices where electrical synapses or inhibitory ones are not able to synchro-

Inhibitory and Electrical Synapses in Synchrony

635

nize neuronal activity (Tamas, Buhl, Lorincz, & Somogyi, 2000; Blatow et al., 2003). In contrast with these results, it has been recently reported that in inspiratory motoneurons, synchronous activity depends on inhibition and is strongly enhanced in the presence of CBX, a blocker of electrical synapses (Bou-Flores & Berger, 2001). Therefore, electrical synapses desynchronize neural activity, in this case. These experimental facts raise the following issues. How do intrinsic cellular properties affect synchrony of neurons coupled via inhibitory or electrical synapses? What is the significance of the combination of electrical and inhibitory couplings for neuronal dynamics? The goal of this work is to address these issues in the simplest system in which they can be answered in models or in experiments: a pair of neurons, interacting via electrical synapses, inhibitory synapses, or both, and activated by the injection of a noisy external current. The pattern of firing of a pair of model neurons can be characterized by the cross-correlations (CCs) of their activity, which measure the probability that the two neurons fire with some given time delay. Of particular interest are the locations and the width of the peaks and the troughs of the CCs. The main limitation in measuring CCs in experiments is the stationarity of the data. This puts constraints on the number of spikes one must accumulate. Simulations of neuronal models can be useful to estimate how long the recordings should be to get reliable estimate of the CCs. Exploring in a systematic way the space of parameters of a model using only numerical simulations is not the best way to discover the general rules underlying the dependence of the CCs on neuronal properties. Here, we derive an analytical expression for the CC of a pair of spiking neurons under the assumptions of weak interactions and weak noise. It provides valuable information about the way noise and intrinsic and synaptic properties affect the shape of the CCs. To our knowledge, this is the first time such an analytical expression has been obtained. Only a few theoretical works have been devoted to neurons coupled via electrical synapses (Sherman & Rinzel, 1992; Chow & Kopell, 2000; Pfeuty, Mato, Golomb, & Hansel, 2003). Recently we combined analytical and numerical studies to investigate how cellular properties affect synchronization of activity of neurons coupled via electrical synapses (Pfeuty et al., 2003). We showed that the stability of the asynchronous state in a large network of tonically firing neurons and the way it is affected by the average neuronal firing rate depend crucially on the shape of the neuronal phase response function (for the definition of the PRF, see section 2). Our theory predicts that sodium or calcium currents impede synchrony of neurons coupled via electrical synapses, whereas potassium currents promote it. This study also shows that predictions for experiments derived in the framework of the leaky integrate-and-fire (LIF) model should be considered with caution. Indeed, we found that large networks of electrically coupled LIF neurons and conductance-based neurons behave in qualitatively different

636

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

ways unless strong potassium conductances are involved in the dynamics of the conductance-based model. This previous work focused on the conditions for antisynchronous states in pairs of neurons and for stability of asynchronous states in large networks. The analysis was also restricted to a single model. In this article, we complement that study by investigating in detail how the features of the CCs of a pair of electrically coupled neurons depend on their intrinsic properties. We also study in a two-compartment model how the location of the synapses, on the soma or on the dendrite, affects the CCs. In contrast to the case of electrically coupled neurons, synchrony of neurons coupled via chemical synapses has been extensively studied. Many of the results that have shaped our concepts regarding this issue have been obtained in the framework of the LIF model (Abbott & van Vreeswijk, 1993; van Vreeswijk, Abbott, & Ermentrout, 1994; Hansel, Mato, & Meunier, 1995; Brunel & Hakim, 1999) or of a modified version of it, the spike response model (Gerstner & Kistler, 2002). Several works have also considered neuronal models with more realistic biophysics (Wang & Rinzel, 1992; Hansel, Mato, & Meunier, 1983; Hansel et al., 1995; van Vreeswijk et al., 1994; Wang & Buzs´aki 1996; Ermentrout, 1996; Crook, Ermentrout, & Bower, 1998a, 1998b; Golomb, Hansel, & Mato, 2001; Ermentrout, Pascal, & Gutkin, 2001; Acker, Kopell, & White, 2003). However, few general results relate intrinsic properties and synchronization behavior. Moreover, these results deal mostly with the stability of fully synchronized states and not the stability of asynchronous state in large networks or antiphase locking of a pair of neurons. In this article, we provide new results regarding how intrinsic properties affect antiphase locking and bistability between in-phase and antiphase locking for a pair of reciprocally coupled inhibitory neurons. Another issue investigated here is the interplay of inhibitory and electrical synapses. This issue has been studied analytically and numerically in the framework of the LIF model (Lewis & Rinzel, 2003) or in specific conductance-based models (Skinner, Zhang, Velazquez, & Carlen, 1999; Traub et al., 2001; Bartos et al., 2002; Nomura, Fukai, & Aoyagi, 2003; Bem & Rinzel, 2004). However, general principles governing this interplay are lacking. When the coupling strengths are not overly strong, the two types of synapses interact linearly when combined. As a consequence, we show that depending on cellular properties, they can cooperate or act antagonistically. We derive rules to predict these effects. We also show that in the strong coupling regime, the two types of synapses can interact in a nonlinear manner. Indeed, we find a regime in which synchrony promoted by inhibition is amplified by electrical synapses, although these synapses alone do not promote synchrony. Finally, our work provides concrete predictions on the relationship between intrinsic properties and synchrony. We propose to test these predictions in experiments in which ionic currents or interactions are modified by

Inhibitory and Electrical Synapses in Synchrony

637

dynamic clamp (Sharp, O’Neil, Abbott, & Marder, 1993; Prinz, Abbott, & Marder, 2004). The letter is organized as follows. In section 2, we describe the methods employed in this work. We describe the two models that are the framework of our study: the quadratic integrate-and-fire model (QIF) and a conductance-based model that involves sodium and potassium currents. We also derive a general formula for the CCs of weakly coupled spiking neurons. Section 3 is devoted to the results obtained in the framework of the QIF model. We first consider the situation of weak or moderate couplings, and we rely on our analytical formula for the CCs. We checked with numerical simulations that these results extend over a substantial range of coupling strength and noise level. The effects specific to strong coupling are subsequently analyzed. The theory developed in the first part of the article allows one to predict the effects of calcium, potassium, or sodium currents. For the sake of concreteness, we focus in section 4 on the effect of a persistent sodium current. The case of other currents is briefly considered in section 5. 2 Methods 2.1 The Models. 2.1.1 Single Neuron Dynamics. Two models are considered in this work: the quadratic integrate-and-fire (QIF) model and a two-compartment conductance-based model. The QIF model is sufficiently simple that it can be studied with analytical calculations. In this model, the dynamical equation for the subthreshold membrane potential, v, is (Latham, Richmond, Nelson, & Nirenberg, 2000; Hansel & Mato, 2003; Pfeuty et al., 2003)

τm

dv = A(v − Vs )2 + Iext (t) − Ic , dt

(2.1)

where τm = 10 ms, A = 0.25, and Ic = 0 are constant parameters and Iext (t) is the total external current to the neuron. The parameter Vs is varied. The membrane potential is measured in mV. Hence, A is measured in mV−1 and the “current” in mV. If Iext is constant in time and smaller than Ic , v converges to a fixed point at large time. In contrast, if Iext > Ic , the solutions to equation 2.1 diverge in finite time. We define action potential firing as v reaching some threshold value VT from below. The membrane potential is then instantaneously reset to some value Vr < VT . In this article, we assume that Vr < Vs < VT . The membrane potential between two action potentials (Iext > Ic ) can be expressed analytically, solving equation 2.1 with the initial condition v(0) =

638

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

Vr . It reads: v(t) =




 √  t A(Iext − Ic ) (Iext − Ic )/A tan + α + Vs τm

(2.2)

where α is an integration constant determined by the condition that at t = 0, the membrane potential of the neuron is at its reset value, Vr : α = tan−1

 √

 Vr − Vs . (Iext − Ic )/A

(2.3)

The condition v(T) = VT determines the firing period, and the firing frequency of the neuron is f = 1/T. One finds tan−1 ( √ VT −Vs ) − tan−1 ( √ Vr −Vs ) (Iext −Ic )/A (Iext −Ic )/A . T = τm √ A (Iext − Ic )

(2.4)

√ Near firing onset, Iext >≈ Ic , the frequency f behaves like f ∝ Iext − Ic , whereas f increases linearly with Iext for large Iext . The subthreshold dynamics needs to be supplemented with a model for the suprathreshold part of the membrane potential time course. Assuming that the width of the action potentials is much smaller than the interspike intervals, we represent them by a δ-function at each time a spike is fired. Therefore, the membrane potential of the neuron can be written at time t, V(t) = v(t) + θ



δ(t − tspike ),

(2.5)

spikes

where θ measures the integral over time of the suprathreshold part of action potentials. For given Vr and VT , the membrane potential time course depends on the parameter Vs . This is shown in Figure 1A, where V(t) is plotted for Vr = 0, VT = 12 mV and different values of Vs . In all these examples, the external current is such that the neurons fire at f = 50 Hz. It is clear that the concavity of the potential time course depends on Vs . For Vs ≈ Vr , it is convex except right after a spike, whereas for Vs ≈ VT , it is concave except right before a spike. For Vs = (VT + Vr )/2, the membrane potential displays an inflexion point at half the firing period. The conductance-based model used in this work has two compartments. The somatic compartment has a leak current, IL , a fast sodium current, INa , a delayed rectifier potassium current, IK , and a persistent sodium current, INaP . Details about these currents are given in appendix A. The second compartment, which corresponds to the dendrite, is passive. For simplicity, we assume that the two compartments have equal areas. Therefore, the

Inhibitory and Electrical Synapses in Synchrony

Vs=2 mV

639

Vs=6 mV

Vs=10 mV

A

B 0.6

0.6

0.4

0.4

0.2

0.2

0.2

Z

0.6 0.4 0

0

0.5

φ

1

0

0

0.5

φ

1

0

0

0.5

φ

1

Figure 1: Voltage traces and PRFs of the QIF neuron. Voltage traces and PRFs were computed using equations 2.2 and 3.3, respectively. The external current is such that the firing rate is 50 Hz (τ0 = 10 msec). (A), Voltage traces of the neuron for three values of Vs (2 mV, 6 mV, 10 mV) in response to a constant current (0.77 mV, 0.98 mV,0.77 mV). Note the changes in the concavity of the voltage traces between the spikes when Vs increases. (B), The PRF, Z, for the values of Vs as in A. For Vs = 6 mV= VT /2, the phase response is symmetric around φ = 1/2. For Vs = 2 mV (resp. Vs = 10 mV), the response is skewed toward the left (resp. the right).

membrane potentials of the soma, V, and of the dendrite, Vd , follow the equations: dV = −IL − INa − INaP − IK + Iext − gc (V − Vd ), dt dVd C = −gld (Vd − Vld ) − gc (Vd − V), dt C

(2.6) (2.7)

where gc is the coupling conductance between the two compartments, gld is the leak conductance of the dendritic compartment, and Iext is an external current. 2.1.2 Inhibitory and Electrical Synaptic Couplings. Inhibitory interactions are taken into account in the QIF model by adding to the right-hand side of the dynamical equation for v, equation 2.1, an additional current of the

640

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

form Iinh (t) = ginh s(t).

(2.8)

Here, ginh is a negative constant parameter measuring the strength of the inhibitory coupling. The function s(t) is defined by s(t) =



f (t − tspike ),

(2.9)

spikes

where the summation extends over all the spikes fired by the presynaptic neuron at time tspike prior to time t and f (t) =

 −t/τ  1 e 1 − e−t/τ2 (t) τ1 − τ2

(2.10)

with τ1 and τ2 , the rise and the decay time of the synapse, respectively, and

the Heaviside function: (t) = 0 for t < 0 and (t) = 1 otherwise. Note that the integral of f (t) over time is normalized to 1. Each neuron receives s(t) from the other neuron. For simplicity, we limit our study of the conductance-based model to the case of inhibitory synapses located on the soma. Therefore, we incorporate inhibition in this model by adding to the right-hand side of the equation for V in equation 2.6 a current of the form Iinh (t) = −ginh s(V(t) − Vinh ),

(2.11)

where ginh > 0 is a constant conductance, Vinh is the reversal potential of the synapse, and s is the synaptic conductance dynamics defined in appendix A. The reversal potential of the inhibition is Vinh = −75 mV throughout the article. An electrical synapse between two neurons induces a synaptic current proportional to the difference of their membrane potential. In both the QIF and the conductance-based model, the effect on neuron i of an electrical synapse connecting neuron j and neuron i is taken into account by adding to the right-hand side of the dynamical equation of neuron i a current of the form Igap = −ggap (Vi − Vj ).

(2.12)

In the QIF model, Vi and Vj are the membrane potentials of the neurons defined by equations 2.1, and 2.5. In the conductance-based model, Vi and Vj stand for the potentials of the somatic or the dendritic compartments depending on the locations of the electrical synapses.

Inhibitory and Electrical Synapses in Synchrony

641

2.1.3 Noise. tochasticity is introduced in the dynamics of the QIF model by adding a gaussian white noise current, Inoise (zero mean, standard deviation, σ ), in the right-hand side of equation 2.1. In the conductance-based model, noise is incorporated in a similar way in the dynamics of the somatic compartment. 2.2 Cross-Correlations of Spike Trains. We characterize the synchrony properties of a pair of neurons by the CCs of their activity. Given the time series at which neuron i = 1, 2 fire action potentials, one can define a variable Si (t) by Si (t) = 1/δ

(2.13)

if neuron i has fired a spike in a time bin of size δ about time t and Si (t) = 0 otherwise. For a sufficiently small δ, the time average of Si , < Si (t) >, is the average firing rate of neuron i. The normalized CC of the spike trains of the two neurons is defined as C(τ ) =

< S1 (t)S2 (t + τ ) > . < S1 (t) >< S2 (t) >

(2.14)

It represents the density of probability that neuron 2 will fire a spike in some bin of size δ a delay τ after a spike of neuron 1. 2.2.1 Synchrony and Antisynchrony of a Pair of Neurons. A flat CC indicates that the two neurons fire independently, whereas a peak in the CC at some time shift indicates that the neurons have an increased probability to fire with some constant time delay. In particular, synchronous firing corresponds to an increased probability of the two neurons to fire simultaneously, that is, to a peak of the CC at τ = 0 in the CC. The model neurons considered in this article fire periodic trains of action potentials in response to constant input current. Noise in the input will decorrelate the periodic firing of action potentials. If the noise is not overly strong, this decorrelation requires several action potentials, and if the neurons fire with some correlation, the CCs will display a few oscillations with a period equal to the average firing period of the neurons, T. We will say that the two neurons tend to fire in antisynchrony if the CC of their activity displays a peak at τ = ±T/2. 2.3 Reduction to a Phase Model. 2.3.1 The Phase Dynamics of a Pair of Neurons. The study of the dynamics of a pair of interacting oscillatory neurons is greatly simplified if one assumes that their cellular properties are not too different, the noise they receive is small, and the coupling between them is weak. Under these assumptions, the dynamics of the neurons can be described in terms of two

642

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

phase variables—one for each neuron. In this reduced model, interactions between the neurons are taken into account by effective coupling functions that depend on their phase difference (Kuramoto, 1984; Ermentrout & Kopell, 1986; Hansel et al., 1993, 1995; van Vreeswijk et al., 1994) and the dynamics of the phases for the two neurons are given by the coupled differential equations, dφ1 = f1 + ginh inh (φ1 − φ2 ) + ggap gap (φ1 − φ2 ) + η1 (t) dt dφ2 = f2 + ginh inh (φ2 − φ1 ) + ggap gap (φ2 − φ1 ) + η2 (t), dt

(2.15) (2.16)

where f1 and f2 are the frequencies of the neurons when decoupled. In the following, we consider the homogeneous case : f1 = f2 = f . The phase coupling inh and gap , which corresponds to chemical and electrical interactions, respectively, depends on the phase response function (PRF) of the neuron, Z(φ). These can be computed from the convolution integrals with φ between 0 and 1:  inh (φ) = 

1

Z(u) s(u − φ) du

(2.17)

Z(u) (V(u) − V(u − φ)) du

(2.18)

0 1

gap (φ) = 0

where s(φ) =

1 e−φ/( f τ1 ) e−φ/( f τ2 ) ( − ) τ1 − τ2 1 − e−1/( f τ1 ) 1 − e−1/( f τ2 )

(2.19)

and V(φ) is the membrane potential expressed as a function of the phase variable. The functions gap (φ), inh (φ), and Z(φ) are periodic with period 1. The terms η1 (t) and η2 (t) in equations 2.15 and 2.16 are the effective noise on the phase dynamics that corresponds to the noise in the input received by the neurons. Assuming white and gaussian input noise with zero mean and standard deviation, σ , and generalizing the approach developed by Kuramoto (1991) in the context of the LIF model, one can show that for the QIF model, η1 (t) and η2 (t) are white gaussian noise with zero mean and standard deviation σφ :  σφ2

=σ

2

1

[Z(u)]2 du.

(2.20)

0

Therefore, the effective noise in the phase dynamics of the QIF neuron is proportional to the average of its PRF over a period.

Inhibitory and Electrical Synapses in Synchrony

643

2.3.2 Analytical Expression for the CC of a Pair of Neurons in the Weak Coupling and Weak Noise Limit. In the weak coupling, weak noise limit, the time elapsed at time t since the last spike fired by neuron i is simply the firing period, T, multiplied by the phase, φ(t), computed modulo 1 (mod(φi (t), 1)). Similarly, the time elapsed between the last spikes fired by neurons i and j is τ = T mod(φi (t) − φj (t), 1). In the weak coupling limit, the CC and the phase shifts’ probability distribution function, P0 (), are related by C(τ ) = P0

τ T

.

(2.21)

A more detailed proof of this equation is given in appendix B. Using equations 2.15 and 2.16, one finds that  = φ1 − φ2 satisfies the Langevin equation, d = − () + η(t), dt

(2.22)

where we have assumed that the two neurons fire at the same average firing rates. We have defined η(t) ≡ η1 (t) − η2 (t) and − () ≡ (() − (−)) with () = ginh inh () + ggap gap ().

(2.23)

The noise η(t) is the difference of two gaussian white noise, with zero average and standard deviation σφ . Therefore, √ it is a gaussian white noise with zero average and standard deviation 2σφ . Hence, the phase-shift distribution, P(, t), satisfies a Fokker-Planck equation (van Kampen, 1981): ∂2 ∂P(, t) ∂ P(, t) − = σφ2 [− () P(, t)]. 2 ∂t ∂ ∂

(2.24)

The stationary phase shifts distribution, P0 , is the solution of this equation, which satisfies the additional constraint ∂P0 (, t) = 0, ∂t

(2.25)

that is, 



P0 () = eG() [A



e−G( ) d + B],

(2.26)

0

where 



G() = 0

− ( ) d

σφ2

(2.27)

644

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

and A and B are two integration constants, which are determined by the

1 periodicity of P0 , that is, P0 (0) = P0 (1), and its normalization, 0 P0 ()d = 1. One finds P0 () = 1 0

eG()

eG( ) d

.

(2.28)

One can show that P0 is extremum for phase shifts  solutions of − () = 0.

(2.29)

These solutions for which d− () =  () < 0(resp. > 0) dφ

(2.30)

are maxima (resp. minima) of P0 . The function  is periodic with period 1. Therefore,  = 0, ±1/2 are always solutions of equation 2.29. Other solutions may also exist. Equations 2.29 and 2.30 can be easily interpreted. Indeed, in the absence of noise, when time goes to infinity, the two neurons phase-lock, and the possible phase shifts are the stable fixed point of equation 2.22 with η = 0. It is straightforward to see that these phase shifts are the solutions of equation 2.29 and that equation 2.30 is the condition for their stability. In the presence of noise, the phase shifts between the two neurons have an increase (resp. decrease) in probability density to be near the stable (resp. unstable) fixed points of the noiseless dynamics. In the limit σ → 0, P0 () is nonnegligable only for values of  in the vicinity of the maxima of G(). For chemical synapses, G() is continuously differentiable at least up to the second order. Expanding G around its maxima, one finds that P0 is a sum of √ gaussians centered around these points, with a width proportional to 1/ − (). Therefore, the greater is the stability of a fixed point in the absence of noise, the sharper is the probability distribution around when noise is present. This is also true for electrical synapses for  = 0. For  = 0, the phase-shift distribution of the QIF model has a cusp. This is because in the QIF model, the PRF is discontinuous at  = 0 and because we have modeled the spikes as a δ-function added to the subthreshold voltage. Note that the phase shift distribution depends on the couplings and the noise via the ratios ggap /σ 2 and ginh /σ 2 . However, the locations of the extrema of the phase shift distribution depend on only the coupling via the ratio ggap /gsyn and are independent of the noise. 3 Cross-Correlations in the Quadratic Integrate-and-Fire Model The QIF model is defined in section 2. Its dynamics are one-dimensional, with a temporal evolution of the voltage in its subthreshold range given by

Inhibitory and Electrical Synapses in Synchrony

645

one first-order differential equation. This equation is supplemented with a reset condition whenever the voltage reaches a threshold. In response to a suprathreshold step of current, the neuron fire spikes periodically. The QIF model is reminiscent of the more standard LIF model. However, in contrast with this model, the QIF dynamics of the voltage between the spikes are nonlinear. As a consequence of the quadratic nonlinearity in the dynamics, the temporal second-order derivative of the voltage of the QIF neuron vanishes during the interspike at a time that depends on a parameter, Vs (as shown in section 2). This parameter also controls the excitability of the neuron (see below). As we will see, this gives rise to a variety of synchronization regimes as Vs varies that do not occur in the LIF model but do occur in a conductance-based model with dynamics that are more faithful to neuronal biophysics. The goal of this section is to identify these regimes in the QIF model. We consider three situations: a pair of neurons interacting solely via inhibition, a pair of neurons interacting solely via electrical coupling, and dual coupling, in which the neurons interact via both types of synapses. We show that similar regimes are found in biophysically more realistic models. Because of the simplicity of the QIF model, many aspects of the dynamics of QIF networks (Hansel & Mato, 2003; Pfeuty et al., 2003) can be investigated using analytical techniques. In the case of a pair of neurons, an analytical formula can be established for their CC, assuming weak coupling and weakly noisy inputs (see equation 2.28). Using this formula, we conducted a systematic study of the ways in which the CC of the activity of a pair of QIF neurons depends on Vs , the firing frequency, and the nature of the neuronal coupling on synchrony. We present the results in the form of phase diagrams for the various regimes of parameters corresponding to qualitatively different shapes of CCs. Subsequently, we use numerical simulations to show that the results thus established hold for a broad range of coupling strengths and investigate new behaviors that may occur beyond this range. 3.1 Weakly Coupled Quadratic Integrate-and-Fire Neurons. 3.1.1 The Phase Response Function. The PRF of the QIF model can be computed analytically. It is given by (Ermentrout, 1981; Kuramoto, 1991; Pfeuty et al., 2003)  Z(φ) =

dv(φ) dφ

−1

,

(3.1)

where φ = t/T with T given by equation 2.4 and v(φ) =




 (Iext − Ic )/A tan

√  φT A(Iext − Ic ) + α + Vs . τm

(3.2)

646

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

Using equation 2.1, one finds Z(φ) =

τm A [v(φ) − Vs ]2 + Iext − Ic

.

(3.3)

Clearly Z(φ) > 0 for all φ. It is nonmonotonic and has one maximum, Z(φm ) at some φm , which depends on the frequency f and on Vs (for fixed Vr and VT ). Examples of the PRF, Z(φ), are plotted in Figure 1B for three values of Vs . For Vs < (Vr + VT )/2, φm is a decreasing function of f with φm → 1/2 in the limit f → 0 and φm → 0 in the limit f → ∞. For Vs > (VT + Vr )/2, φm increases with f . Still, in the limit f → 0, φm → 1/2 but φm → 1 for f → ∞. For Vs = (VT + Vr )/2, φm = 1/2, independently of f . Examples of the PRF are plotted in Figure 1B for three values of Vs . An analytical formula for the CC of a pair of QIF, equation 2.28, can be derived assuming weak coupling and weak noise. The results presented in this section were derived using this formula to compute the CCs. 3.1.2 The Three Generic Shapes of the CC for Inhibitory Coupling. Three examples of CCs computed using equation 2.28 are plotted in Figures 2A through 2C. The first example, displayed in Figure 2A, is for a high firing rate, 80 Hz, and a small Vs = 0.4 mV. In this case, C(0) and C(±T/2) are local minima, and C is maximum at time shifts ±τ , with τ ≈ T/6. Figures 2B and 2C for different values of Vs , Vs = 11.2 mV and Vs = 8 mV and the same firing, f = 20 Hz. In both cases, C(τ ) has a local maximum at τ = 0, but for Vs = 11.2 mV, the amplitude of this maximum is extremely small. It is much more pronounced for Vs = 8 mV. For Vs = 11.2 mV, C is also maximum at τ = ±T/2. Therefore, the pattern of firing of the neuronal pair is different in the two cases. For Vs = 8 mV, the neurons fire most of the time in synchrony, whereas for Vs = 11.2 mV, the neurons fire most of the time in antisynchrony. The three shapes of the CCs described in Figure 2 are typical. A phase diagram showing the domain of parameters corresponding to each shape can be computed. It is plotted in Figure 2D. In the dark gray region, the CC has only one maximum, located at τ = 0. For large Vs and sufficiently small frequency (gray-and-white striped region), the CC displays another peak at τ = ±T/2. However, C(0) decreases sharply below the boundary of the gray and white striped region. In most of this region, C(±T/2) is much larger than C(0), that is, the neurons are more likely to fire in antisynchrony than in synchrony. In contrast, for a small Vs and a large frequency (light gray region), the CC has two maxima, symmetrical around 0, at ±τ with 0 < τ < T/2. Figure 2E displays the CC against Vs and τ for a fixed frequency, f = 20 Hz. It shows that at this frequency, the CC is maximum around τ = 0 but broad at small Vs . The peak becomes more pronounced when Vs increases. However, when Vs is too large, the amplitude of this peak decreases, whereas a peak around τ = ±T/2 appears.

Inhibitory and Electrical Synapses in Synchrony

A C(τ)

B Vs =2mV, f=50Hz

2 1

Frequency (Hz)

2

C Vs =4mV, f=50Hz

0 τ/T

0.5

0 -0.5

0 τ/T

D

V =8mV, f=50Hz

s 4 3 2 1 0 0.5 -0.5

1

0 -0.5

100

647

0 τ/T

0.5

E

80 60

4

C(τ) 2

40

0 0

20 0

0.5

Vs (mV) 8 0

4

8 Vs (mV)

12

0

4

τ/T

12 -0.5

Figure 2: Cross-correlations of a pair of QIF neurons interacting via inhibitory synapses. (A–C) CCs for three values of Vs and f for ginh /σ 2 = 20. The synaptic time constants are τ1 = 1 msec, τ2 = 6 msec. (A): Vs = 0.4 mV, f = 80 Hz. The CC is peaked at ±T/6. (B) Vs = 8 mV, f = 20 Hz. The CC has only one peak located at τ = 0. (C) Vs = 11.2 mV, f = 20 Hz. The CC is peaked at τ = ±T/2 (antiphase). It also exhibits a small peak at τ = 0 (see inset). (D) The phase diagram. Regions in which the CCs have different shapes have different gray codes. Dark gray: CCs with a peak at τ = 0. White: CCs with a peak at τ = ±T/2. Light gray: CCs with peaks at ±τ with 0 < τ < T/2. Dark gray and white: CCs with a peak at τ = 0 and at τ = ±T/2. (E) The CCs as a function of Vs for f = 20 Hz.

The phase diagram in Figure 2 was computed for τ1 = 1 msec, τ2 = 6 msec. Similar phase diagrams are found for other values of τ1 and τ2 , but the positions of the transition lines are different. The slower the inhibition, the smaller the region where CC displays a peak around τ = ±T/2 (grayand-white striped region) and the larger the region where CC has a maxima at ±τ with 0 < τ < T/2 (red). 3.1.3 Electrical Couplings Mediate Synchrony or Antisynchrony Depending on Vs and the Firing Frequency. A detailed study of C(τ ) for a fixed frequency, f = 50 Hz, shows that when Vs increases from Vs = Vr ≡ 0 to Vs = VT ≡

648

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

12 mV, the shape of the CC undergoes a continuous transformation. This is shown in Figure 2E, where the CC is plotted against Vs and τ for a fixed frequency, f = 50 Hz. For a sufficiently small Vs , the CC is peaked around τ = ±T/2 and has a trough at τ = 0. When Vs increases, the trough persists, but when Vs becomes sufficiently large, the CC is no longer maximum at τ = ±T/2. The two maxima, symmetrical around 0, are then located at some intermediate time delay, the distance of which decreases when Vs increases. The trough at τ = 0 is also reduced, and the distribution becomes flatter when Vs increases. For some value of Vs , the two maxima merge, and the CC becomes monomodal at τ = 0. The CC keeps this shape, becoming sharper and then again flatter, as Vs increases. Therefore, for f = 50 Hz, the CC can have three qualitatively different shapes when Vs changes. Examples are plotted in Figures 3A through 3C. The phase diagram plotted in Figure 3D as a function of Vs and f shows that the three types of CCs described above for f = 50 Hz are also found for other values of the firing rate. It also shows that for sufficiently small frequency, a fourth type of CC is found for Vs sufficiently close to VT . It is maximum at τ = 0 but also at τ = ±T/2. The closer Vs is to VT , the larger the amplitude of the CC near τ = ±T/2 (not shown). The interactions induced by electrical synapses are diffusive. In contrast to chemical synapses, the synaptic currents they induced do not have intrinsic kinetics but depend on the membrane potential time course of the preand postsynaptic neurons. Subsequently, for electrical synapses, the current depends on the size and the shape of the action potential. In our model, this means that the amplitude of the spike, θ, may affect the synchronization properties of the neurons. In fact, we found that the phase diagram is affected quantitatively only when θ is changed. When θ increases, the grayand-white striped region shrinks and the white region expands. However, it can be shown that the white region always remains confined to the left half-part of the phase diagram, where Vs < 6 mV. 3.1.4 Three Modes of Interplay Between Electrical and Inhibitory Synapses. The results of the previous sections demonstrate that the CCs of a pair of QIF neurons coupled via inhibition or electrical synapses depend crucially on the parameter Vs . Moreover, comparing the phase diagram in Figure 3 with the one in Figure 2 reveals that these two types of synapses may have similar or opposing effects on synchrony, depending on Vs and f . Therefore, it is likely that when combined, the two types of synapses can either cooperate in promoting synchrony or be antagonistic, depending on the parameters. Using equations 2.28, 2.27, and 2.23, one can see that the number of maxima and minima of C(τ ) and their location depend on the coupling constants via their relative value, ggap /gsyn alone. However, other properties of C, for example, the amplitude of the extrema and their widths, depend on the absolute values of ggap and gsyn .

Inhibitory and Electrical Synapses in Synchrony

A C(τ)

2

Vs =0.4mV, f=80Hz

1

Frequency (Hz)

2

C

Vs =8mV, f=20Hz

2

1

0 -0.5

100

B

649

0 τ/T

0.5

Vs =11.2mV, f=20Hz

1

0 -0.5

0 τ/T

D

0 -0.5

0.5

0 τ/T

0.5

E

80 60

C(τ)

40 20 0

2 1 0 0

0.5

Vs (mV) 8 0

4

8 Vs (mV)

12

0

4

τ/T

12 -0.5

Figure 3: CCs for a pair of QIF neurons interacting via electrical synapses. (A– C) CCs for three values of Vs and f = 50 Hz for ggap /σ 2 = 10. (A) Vs = 2 mV, f = 50 Hz. The CC is peaked at ±T/2. (B) Vs = 4 mV. The CC is peaked at τ ± T/6. (C) Vs = 8 mV. The CC is peaked at τ = 0. (D) The phase diagram. The codes are as in Figure 2. (E) Plot of the CCs as a function of Vs and τ/T for f = 50 Hz.

We first investigate how the combination of these two types of synapses affects the extrema of C and the corresponding phase diagram as a function of f and Vs . As the ratio ggap /ginh varies from 0 to ∞, the phase diagram gradually modifies from the one in Figure 3D to the one in Figure 2D. For example, Figure 4A plots the phase diagram for ggap /ginh = 1/2. The codes have the same meaning as in Figures 2D, and 3D. Comparing Figures 4A and 3D shows that adding inhibitory synapses to electrical synapses reduces the size of the region of antisynchronous firing (white region), and increases the domain of intermediate phaseshift (light gray region) as well as the domain of bistability (gray-and-white striped region). Comparing Figure 4A with Figure 2D, one sees that adding electrical synapses to inhibitory synapses reduces the size of the domain of synchronous firing (dark grey region). This

650

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

Frequency (Hz)

100

A

B

80 60

C(τ) 4

2 0 0

40 20 0

0.5

Vs (mV) 8 0

4

8 Vs (mV)

12

0

4

τ/T

12 -0.5

Figure 4: A pair of QIF neurons interacting via dual coupling. The coupling ratio ggap /ginh = 1/2. (A) Phase diagram. The parameters and the codes are the same as in Figures 2D and 3D. Note the small region at large Vs and small frequency in which the CCs have a peak at τ = 0 and at τ = ±T/2. (B) The CC as a function of Vs for f = 50 Hz.

also decreases the size of the bistability domain (gray-and-white striped region). Figure 4B displays the CC as a function of Vs and τ for a fixed frequency, f = 50 Hz. The CCs for electrical synapses, alone, dual synapses, or inhibition alone are compared (from left to right) in Figure 5 for different values of the parameters Vs and f . In the top row (Vs = 2 mV and f = 50 Hz), electrical synapses alone promote antisynchronous firing, whereas inhibition promotes synchronous firing. When combined together, the CC is maximum for some intermediate time delay (neither 0 nor ±T/2). Note also that the CC is less modulated for dual coupling than for electrical or inhibitory couplings alone. Therefore, for these values of Vs and f , starting from dual coupling and blocking inhibition increases dramatically the probability of antisynchronous firing while suppressing almost completely the probability of synchronous firing. Blocking electrical synapses improves synchronous firing. In the middle row (Vs = 8 mV, f = 50 Hz), both electrical synapses and inhibitory synapses promote synchronous firing. When together, the synapses cooperate to increase further the probability of synchronous firing and to sharpen the CC around τ = 0. Therefore, in this case, blocking electrical synapses or inhibition reduces the probability of the neurons of firing in synchrony. In the bottom row (Vs = 11.2 mV and f = 20 Hz), electrical synapses alone promote synchrony, but with inhibition alone, the maxima of C are at τ = 0, ±T/2, C(±T/2), being much larger than C(0) (in fact, the maximum at τ = 0 cannot be seen in the figure because of its scale). For dual

Inhibitory and Electrical Synapses in Synchrony

651

Figure 5: Different regimes of interplay of electrical and inhibitory couplings for QIF neurons. (Left column) Electrical synapses (same parameters as in Figure 2). (Right column) Inhibitory synapses (same parameters as in Figure 3). (Middle column) Dual coupling. (Top row), Vs = 2 mV, f = 50 Hz. Antagonistic effects of electrical and inhibitory synapses on synchrony: electrical synapses impede synchrony, but inhibitory synapses promote it. (Middle row), Vs = 8 mV, f = 50 Hz: Electrical and inhibitory synapses cooperate. Bottom row, Vs = 11.2 mV, f = 20 Hz. Antagonistic effect of electrical and inhibitory synapses on synchrony: electrical synapses promote synchrony, but inhibitory synapses impede it.

coupling, the CC is maximum for τ = 0, ±T/2, with a larger correlation at τ = 0 than at τ = ±T/2. However, C(0) is substantially smaller in comparison to the case of electrical coupling alone. Hence, in this case, starting with neurons interacting via dual coupling and blocking inhibition would increase the probability of synchronous firing, whereas blocking electrical synapses would reduce it substantially and increase the probability of antisynchronous firing.

652

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

In summary, the weak coupling theory yields three parameter regimes for the interplay of electrical and inhibitory synapses in synchrony. In one regime, the synapses exhibit cooperative interplay, both of them promoting synchrony. In the other regimes the synapses act antagonistically. This may happen in two ways: because electrical synapses oppose synchrony but inhibitory synapses promote it or because electrical synapses promote synchrony but the net effect of the inhibitory interactions opposes it. Note that these three modes of interplay stem from the dependence of the CCs on Vs for electrical or inhibitory synapses alone combined with the fact that the function G() that determines the CCs for dual coupling, equation 2.28, depends linearly on the two interactions. 3.2 Beyond Weak Coupling and Weak Noise: The Nonlinear Interplay Between Electrical and Inhibitory Synapses. The analytical formula for the CCs used above in the derivation of the phase diagrams was established assuming weakly coupled neurons and weakly noisy inputs. Do the effects of Vs on the CCs found in this limit for inhibitory or electrical synapses alone still persist when the coupling and the noise are not weak? In the weak coupling limit, electrical and inhibitory synapses interact in a linear manner when they are combined. What are the nonlinear effects that occur at strong couplings? To answer these questions, we performed numerical simulations. Figure 6 displays the CCs obtained in numerical simulations (circles) for two values of Vs , Vs = 2 mV and Vs = 8 mV, with various coupling strengths and three combinations of the interactions (electrical, inhibitory, and dual). The noise was also varied to keep the ratios ginh /σ 2 , ggap /σ 2 fixed to 10 and 20, respectively. This is because the weak coupling theory predicts that the CCs depend on σ 2 and on ginh and ggap , via these ratios, as can be seen from equations 2.21 and 2.28. The solid lines in Figure 6 correspond to the CCs computed from the latter equations for the same values of these ratios. In the left column of Figure 6, which corresponds to relatively weak electrical and inhibitory couplings corresponding to postsynaptic potential of an amplitude of 0.2 mV, the simulations agree perfectly with the results from the weak coupling analysis. The middle column corresponds to coupling strengths, which are four times larger than in the left column. The agreement remains excellent except for dual coupling at Vs = 2 mV. Indeed, in this case, the weak coupling theory predicts that the CC should display a trough at zero time shift and that the firing probability should be maximum close to T/8. In contrast, in the simulations, the CC is maximum at zero time shift. Results for further increase of coupling strength by a factor of 3 are displayed in the right column in Figure 6. This corresponds to postsynaptic potentials of a typical size of 2 mV. For Vs = 8 mV, weak coupling theory predicts synchronous firing for electrical synapses alone. Synchrony is also found in the simulations, but

Inhibitory and Electrical Synapses in Synchrony

653

Figure 6: CCs of a pair of QIF neurons for different coupling strengths. Circles and dash lines: Results from simulations (averaging over 100 sec of activity). Solid lines: Predictions from weak coupling theory. The strength of the electrical synapses is from left to right: ggap = 0.0025, 0.01 and 0.03 corresponding, respectively, to coupling coefficients of 0.02 mS/cm2 , 0.06 mS/cm2 , and 0.16 mS/cm2 . The strength of inhibitory synapses is from left to right: ginh = 0.005 mS/cm2 , 0.02 mS/cm2 , and 0.06 mS/cm2 . The noise and the external current were adjusted to keep constant ggap /σ 2 = 10 and ginh /σ 2 = 20. The firing frequency is fixed (50 Hz). (A) Vs = 2 mV. (B) Vs = 8 mV.

the CC is much more peaked than predicted. This is due to direct threshold crossing, which is rare when the coupling is weak but contributes a great deal to the firing of the postsynaptic neuron for strong electrical coupling. For inhibitory coupling, the agreement between the theory and the simulations remains reasonable. Combining the two interactions sharpens the

654

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

CC, as predicted by the weak coupling theory, but quantitative agreement is poor. For Vs = 2 mV, the agreement is excellent for inhibitory coupling. For electrical coupling, the theory predicts antisynchronous firing. However, in the simulations, the CC is flat—the probability of firing is almost homogeneous, as it would be without coupling. Remarkably, in spite of the fact that electrical synapses are not able to induce synchrony, they strongly amplify synchrony and sharpen the CC when they are added to the inhibitory synapses (compare the CC in the middle line, right column). This nonlinear interplay between the two types of synapses requires strong couplings. 3.3 Predicting the Shape of the CCs from the Shape of the Phase Response Function. Our study of the QIF model in the weak coupling limit relies on the analytical formula, equation 2.28, which relates the CC to the synaptic couplings and the neuronal PRFs. The main effect on the PRF of changing Vs is to shift the location of its maximum. It varies continuously from 0 to 1, as Vs increases from Vr to VT (see Figure 1). Since equation 2.28 holds for any neuronal dynamics, our analysis of the QIF model suggests that more generally, synchrony may be predicted from the shape of the PRF independent of the details of the neuronal dynamics. These predictions can be summarized as follows. For electrical synapses, a pair of neurons with PRF skewed to the right tends to fire synchronously when interacting via electrical synapses. In contrast, neurons with PRF sufficiently skewed to the left tend to fire antisynchronously. For inhibitory synapses, CCs always have a peak at τ = 0 unless the PRF is too skewed to the left and the firing rate, f , of the neurons is too large. When the PRF is strongly skewed to the right and f is sufficiently small, the CC is also peaked around τ = ±T/2. The latter peaks may be much higher than the peak at τ = 0, indicating that the net effect of inhibition opposes synchrony. The faster the synapses, the larger the domain of parameters in which this happens. Our simulations of the QIF model indicate that these predictions from the weak coupling theory remain relevant, at least qualitatively, within a broad range of coupling strengths. For weak enough synapses, the interplay of electrical and inhibitory interactions is essentially linear. This property stems from the general formula, equation 2.28, and therefore it is not restricted to the QIF model. Hence, we expect that for general neuronal dynamics, the shapes of the CC for dual coupling can be predicted from the effect of each synapse taken separately, provided the interactions are not too strong. When the interactions are strong, nonlinear effects start to be significant. Nevertheless, knowing the PRF of the neurons can serve to qualitatively predict the interplay of the two types of interactions. For instance, from the analysis of the QIF model, we predict that for neurons with a PRF skewed to the left, adding strong electrical synapses to inhibitory synapses amplifies synchronous firing, although singly electrical synapses would not lead to synchrony. This effect

Inhibitory and Electrical Synapses in Synchrony

655

is a consequence of the rapid direct threshold crossing induced by strong electrical synapses when they act at a time when the postsynaptic neuron is not far from threshold. Therefore, one may expect that such synergetic interplay of electrical and inhibitory interactions is a general effect that occurs in conductance-based models and in real neurons. 4 Cross-Correlations in a Conductance-Based Model The results of the previous section suggest that the shape of the CC of a pair of neurons can be related to their intrinsic properties, provided one knows how ionic channels and neuronal morphology shape their PRF. In this section, we verify that this indeed holds in the framework of a twocompartment conductance-based model neuron. The somatic compartment incorporates one fast and one persistent sodium current and a delayed rectifier potassium current. The dendritic compartment is passive. Details of the model are given in section 2. To be specific, we focus on the effects of the persistent sodium current and the dendritic compartment on the synchrony of the neurons. The results presented in this section lead to experimentally testable predictions, as explained in section 5. 4.1 Increasing gNaP in the Conductance-Based Model and Decreasing Vs in the QIF Model Have Similar Effects on the PRF. The effect of INaP on the voltage trace and the PRF of the neuron is shown in Figures 7A and 7B. For all the considered values of gNaP , Z(φ) is positive for all φ. It is also nonmonotonic and monomodal. The PRF is small just before, during, and just after the action potential. The position of the maximum depends on gNaP . For gNaP = 0, it is in the second half of the period at φ ≈ 2/3. As a result, the global shape of the PRF is skewed toward the second half of the period of the neuron. The response of the neuron increases with gNaP , especially in the first half of the period. This shifts the maximum of Z(φ) toward smaller φ. For large enough gNaP , the PRF is skewed toward the left. The PRF also depends on the firing rate, f . For a given parameter set, the maximum of Z moves closer to 1/2 when f goes to 0; in that limit, the function Z becomes proportional to 1 − cos 4π φ (Ermentrout, 1996), as in the QIF model. Therefore, increasing gNaP in our conductance-based model or decreasing Vs in the QIF model affects the PRF of the neurons in a similar way. Hence, we may expect that this should also affect the CCs similarly. We show that this is indeed the case. 4.2 The Effect of the Persistent Sodium Current on the Cross-Correlations. In this section, we assume that gc = 0 (in this case, the model has a single compartment). Figure 8 plots the CCs for three values of gNap — when the neurons are coupled solely via electrical synapses (left column), chemical synapses (right column), or both synapses (middle column). The synaptic conductances were ggap = 0.005 mS/cm2 and ginh = 0.02 mS/cm2 .

656

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

Figure 7: Voltage traces and phase-response functions of the conductance-based model neuron are shaped by the persistent sodium current. The soma and the dendrite are decoupled (gc = 0). (A) Voltage traces. (B) PRF. The PRFs were obtained by computing the delay on the spike times induced by small perturbations (see Hansel et al., 1993). The external current was adjusted to keep the firing constant, f = 50 Hz: The external currents are 1.1 µA/cm2 , −0.3 µA/cm2 , and −1.04 µA/cm2 , respectively, for gNaP = 0, 0.2 mS/cm2 , and 0.7 mS/cm2 .

The corresponding spikelets and IPSPs are also shown in Figure 8A. Their peak amplitudes are about 0.3 mV and 0.4 mV, respectively. The results obtained for gNaP = 0.7 mS/cm2 , f = 50 Hz are similar to those obtained for the QIF model with Vs = 2 mV and the same firing frequency (see Figure 5 left line). In both cases, electrical synapses promote antisynchrony, whereas inhibition alone promotes synchrony. For dual coupling, the two trends compete, and the peak of the CC around τ = 0 is very flat. Reducing gNaP to gNaP = 0.2 mS/cm2 but keeping the frequency constant, f = 50 Hz, electrical synapses or inhibitory synapses alone promote synchronous firing. For dual coupling, the two types of synapses cooperate, leading to more synchronous activity than when each of them is considered alone. This is similar to what was obtained in the QIF model for Vs = 2, f = 20 Hz (see Figure 5, middle row). For even smaller gNaP and also reducing the frequency ( f = 20 Hz) (see Figure 8B, bottom row) leads to CCs that are similar to those for Vs = 11.2 mV and f = 20 Hz in the QIF (bottom row in Figure 5). In both cases, electrical synapses alone favor synchronous firing. Adding inhibitory interactions substantially reduces the probability of the neurons’ firing simultaneously. This is because for gNaP = 0, electrical and inhibitory synapses act antagonistically, since the latter promote antisynchrony (see Figure 8B, bottom row, right column). We also investigated the effect of gNaP on the CCs when the synapses are strong. Results are shown in Figure 9 for synaptic conductances six times

Inhibitory and Electrical Synapses in Synchrony

657

Figure 8: Effects of the persistent sodium current and the firing frequency on the CCs in the conductance-based model. (A) Postsynaptic potentials due to one presynaptic spike are shown (for gNaP = 0). The soma and the dendrite are decoupled (gc = 0). (Left) Electrical coupling, ggap = 0.005 mS/cm2 . (Right) Inhibitory coupling, ginh = 0.02 mS/cm2 . (Middle) Dual coupling. (B) CCs are computed over 100 sec of activity for three values of gNaP . The SD of the noise is σ = 0.15 mV/ms1/2 . (Top row) gNaP = 0.7 mS/cm2 , f = 50 Hz. (Iext = −1.04 µA/cm2 ). Electrical and inhibitory synapses have antagonistic effects on synchrony. Electrical synapses suppress synchrony and inhibition-induced synchrony. (Middle row) gNaP = 0.2 mS/cm2 , f = 50 Hz (Iext = −0.30 µA/cm2 ). Electrical and chemical synapses cooperate. (Bottom row) gNaP = 0, f = 20 Hz, (Iext = 0.56 µA/cm2 ). Electrical and inhibitory synapses have antagonistic effects on synchrony. Inhibitory synapses suppress synchrony, but electrical synapses promote it.

658

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

larger than in the results depicted in Figure 8, namely, for ggap = 0.03 mS/cm2 and ginh = 0.12 mS/cm2 . The amplitude of the corresponding spikelets and inhibitory postsynaptic potentials is about 2.5 mV (see Figure 9A). The coupling coefficient for electrical synapses is 20%. This corresponds to synapses with strengths in the upper physiological range. Our numerical simulations confirmed that the crucial influence of the persistent sodium current on the CCs also occurs for such strong synapses. Indeed, for gNaP = 0.2 mS/cm2 , the neurons fire in synchrony, and the CC has a peak at τ = 0 (see Figure 9C, left panel). In contrast, for gNaP = 0.7 mS/cm2 (see Figure 9B, left panel), the central peak of the CC is suppressed. Note, however, that the peak is sharper and larger in Figure 9C than in Figure 5B, and the peaks of the CC in Figure 9A are around τ = ±3T/8 and not at τ = ±T/2, as in the corresponding case for weaker coupling (see Figure 5). The right panels in Figures 9A and on B show the CCs for strong inhibition. Reducing gNaP has a minor effect on the CC peak value, as in the corresponding case in Figure 5. However, in contrast to weak coupling, this also decreases the firing probability around τ = T/4 and increases it at τ = T/2. This did not happen at weaker coupling (see Figure 5). The most significant difference between the CCs at weak and strong couplings occurs when electrical and inhibitory interactions are combined and gNaP is sufficiently large. Indeed, in this case, the two types of synapses show synergy. As in the QIF model, for small Vs , synchronous firing is completely suppressed for electrical synapses alone. But when added to inhibition, the peak of the CC at τ = 0 is amplified. 4.3 Synchrony of Neurons Coupled via Electrical Synapses Decreases When the Somato-Dendritic Coupling Increases. We now assume gNaP = 0 and discuss the effect of the dendrite on synchrony. Therefore, we assume gc = 0. In this case, perturbations acting on the somatic or the dendritic compartment affect the firing of the neuron differently. Therefore, two PRFs can be defined, one, Zsoma , for perturbations on the soma and the other, Zdendrite , for perturbations on the dendrite. These two functions are plotted in Figure 10A for our two-compartment model assuming gNaP = 0, a somato-dendritic coupling conductance gc = 0.3 mS/cm2 and for a firing frequency, f = 50 Hz. The somatic PRF for gc = 0 and f = 50 Hz is also plotted on the same figure (dashed line). The coupling with the dendritic compartment affects the shape of the somatic PRF by skewing it toward the first half of the period (compare the somatic PRFs for gc = 0.3 mS/cm2 and gc = 0). This is because the voltage perturbation at the soma backpropagates to the dendrite, and from there reverberates back to the soma. Therefore, the overall response to a perturbation at the soma is the sum of two contributions. The first one is the same as though the dendrite was absent. The second one, which is due to the dendrite, is a delayed and broadened version of the first one. Together, these

Inhibitory and Electrical Synapses in Synchrony

659

Figure 9: Nonlinear interplay between strong electrical and inhibitory synapses for conductance-based neurons. Synaptic conductances are ggap = 0.03 mS/cm2 and ginh = 0.12 mS/cm2 . The SD of the noise is adjusted to obtain ggap /σ 2 and ginh /σ 2 to ≈ 1.1 and ≈ 4.4. The external current is such that f = 50 Hz. (A) Postsynaptic potential induced by a presynaptic action potential for (left to right) electrical coupling alone, dual coupling, and inhibitory coupling alone. (B) gNaP = 0.7 mS/cm2 (Iext = −1.04 µA/cm2 ). Electrical synapses alone reduce probability to fire in-phase (top), but combined with inhibitory synapses they amplify synchronous firing. (see right panels). (C) gNaP = 0.2 mS/cm2 (Iext = −0.30 µA/cm2 ). Electrical synapses strongly amplify the peak in-phase with or without inhibitory synapses.

two contributions lead to a response that is stronger and extends earlier in the limit cycle. When the perturbation is on the dendrite, the PRF is skewed to the left, as shown in Figure 10A (solid line) compared to the somatic PRF (dash-dot line). This is because the dendritic compartment delays and filters the perturbation before it can affect the firing of action potentials in the somatic compartment.

660

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

0.4

A ES in dendrites

Z

ES in soma with dendrites

0.2 0

ES in soma without dendrites

0

φ

1

B

4

C(τ)

C(τ)

4 2 0 -0.5

0 τ/T

0.5

C 50 Hz

10 Hz

2

0

0 τ/T

Figure 10: Effect of the dendritic compartment on the PRF and on the CCs. (A) The somatic PRF for gc = 0, Iext = 1.02 µA/cm2 (dash line) and gc = 0.3 mS/cm2 , (dash-dot line); the dendritic PRF for gc = 0.3 mS/cm2 , Iext = 1.50 µA/cm2 ) (solid line). f = 50 Hz in all the cases. (B) The CC for somatic electrical synapses for gc = 0 (dashed line) and gc = 0.3 mS/cm2 (dash-dot line) and for dendritic synapses for gc = 0.3 mS/cm2 (solid line). The firing frequency is f = 50 Hz. (C) CCs with dendritic synapses for two values of the firing frequencies f = 50 Hz (solid line) and f = 10 Hz (circles).

Therefore, adding a dendritic compartment to the soma shapes the PRF similarly to reducing Vs in the QIF model or increasing gNaP in the single compartment model. Hence, we expect that for electrical synapses located on the soma, synchrony decreases with gc . We also expect that neurons are less synchronized if the electrical synapses are on the dendrites than if they are on the soma. This is confirmed by the results displayed in Figure 10B, where the CCs of two neurons firing at 50 Hz are compared for somatic electrical synapses for gc = 0.3mS/cm2 (dash-dot line) and gc = 0 (dashed line) and dendritic synapses (solid line). In the first two cases, the CC is peaked around τ = 0. However, it is much broader and smaller, gc = 0.3 mS/cm2 , than for gc = 0. Moving the synapses to the dendrites reduces the synchrony further. In the example shown in Figure 10B (solid line), it even leads to antisynchronous firing. The transition to antisynchrony induced by electrical

Inhibitory and Electrical Synapses in Synchrony

661

synapses when the dendritic compartment is added is basically similar to the transition to antisynchrony in the QIF model when Vs is reduced. As in the QIF, this effect is firing rate dependent. This is shown in Figure 10C, where the CC is plotted for f = 50 Hz and f = 10 Hz. In the latter case, the CC, although broad, is maximum at τ = 0 in contrast with f = 50 Hz, where antisynchrony occurs. 5 Discussion 5.1 Relating CCs to Cellular and Synaptic Properties. We have shown that the CCs of the activity of a pair of tonically spiking neurons interacting via electrical and/or inhibitory synapses can be predicted if one knows the shape of their PRF. Subsequently, one can predict how cellular properties affect the CCs if one knows how the former shape the PRF. Strictly speaking, the PRF is defined only when neurons are weakly interacting. However, in our model, weak coupling predictions remain valid at least qualitatively for spikelet amplitudes and IPSPs as large as 2 to 2.5 mV. These values are similar to those observed in vitro (Galarreta & Hestrin, 2001, 2002). In the case of the QIF model, we focused on the effect of the parameter Vs . The effects of Vr and VT can be studied in a similar way. In conductancebased models as in real neurons ionic channels determine the PRF (Crook et al., 1998a; Ermentrout et al., 2001; Oprisan & Canavier, 2002; Acker et al, 2003; Pfeuty et al., 2003). We have focused here on the effect of INaP or of a passive dendrite (modeled as a single compartment), which shifts the maximum of the PRF toward the left. Other inward currents, such as calcium currents, have the same effects on the PRF and must affect synchrony in a similar way. Potassium currents, in contrast, skew the PRF to the right (Ermentrout et al., 2001; Pfeuty et al., 2003). Therefore, we predict that potassium and sodium currents have mirror effects on synchrony (Pfeuty et al., 2003). We have verified these predictions in simulations of conductance-based models (results not shown). We have found that electrical synapses located on the dendritic compartment of the conductance-based neuron are less synchronizing than if they are located on the soma. One contribution to this effect is the additional delay induced by the dendritic compartment, which is like shifting the PRF to the left. One can similarly predict the effect of conductions or the synaptic delays in the inhibitory interactions. Although we did not include these delays in our model, our approach can be straighforwardly extended to take them into account (Izhikevich, 1998). For given cellular properties, this would amount to shifting the PRF of the neurons to the left, as would be the effect of increasing gNaP without delays. 5.2 Linear and Nonlinear Interplays Between Electrical and Inhibitory Synapses. When the interactions are not too strong, the CCs of the activity of two neurons are very well approximated by our analytical result, equa-

662

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

tions 2.21 and 2.28, which relates the CC to the effective phase interaction, . For dual coupling,  is the sum of two contributions due to each of the interactions. In this sense, the interactions combine their effect linearly. This implies that the CC for dual coupling is the normalized product of the CCs for each interaction alone. When the interactions are sufficiently strong, substantial deviations from equations 2.21 and 2.28 occurs due, for instance, to nonlinear interaction between electrical and inhibitory synapses. This is what we have found when electrical synapses amplify synchronous firing of a pair of neurons when combined with inhibition, although the CC would display a trough at zero time delay with electrical synapses alone due to the intrinsic properties of the neurons. A counterpart of this synergetic effect in a large network of such neurons is enhancement of synchrony by electrical synapses when combined with inhibition, although alone the latter would promote asynchronous firing (results not shown). This effect should not be confused with the more trivial fact that combining electrical and inhibitory synapses enhances synchrony when the neuronal properties are such that both interactions alone promote synchronous firing. 5.3 Relation to Previous Studies. 5.3.1 Modeling Works. Previous work has addressed the role of cellular properties in inhibition-mediated synchrony focusing on the interplay between postinhibitory rebound (PIR) and inhibition. They have shown that PIR stabilizes in-phase synchrony (Wang & Rinzel, 1992; Golomb, Wang & Rinzel, 1994) for sufficiently slow and strong inhibition. The synchronizing role of inhibition studied here is essentially different; it occurs for QIF neurons that do not display PIR, and it does not require strong coupling. Rather, it is similar to the mechanism considered by van Vreeswijk et al. (1994) and Hansel et al. (1995). Our study helps clarify how intrinsic neuronal properties affect synchrony in this mechanism. Previous studies of conductance-based models have shown that a pair of electrically coupled neurons may fire in synchrony or in antisynchrony depending on the model or the firing rate (Sherman & Rinzel, 1992; Han, Kurrer, & Kuramoto, 1995; Chow & Kopell, 2000). For the compartmental model studied by Alvarez, Chow, van Bockstaele, and Williams (2002), electrical synapses located on the soma synchronize the spikes of the neurons in the whole range of frequency investigated. In contrast, for synapses on the dendrites, synchrony occurs only for sufficiently small firing rates. Our work provides a general framework to understand this diversity of behaviors. Synchrony of a pair of LIF neurons interacting via electrical and/or inhibitory synapses has been studied by Lewis and Rinzel (2003). The PRF of the LIF increases monotonically with the phase, regardless of the cellular parameters (Kuramoto, 1991; Hansel et al., 1995). Such a shape is found in the QIF model for Vs → VT . Therefore, in this limit, the properties of

Inhibitory and Electrical Synapses in Synchrony

663

the QIF model resemble those of the LIF. For instance, in the regions in the bottom-right of the QIF phase diagrams for electrical synapses (see Figure 3) and for inhibitory synapses (see Figure 2), the CCs have a peak at τ = 0 and at τ = ±T/2. Changing the size of the spike θ and the inhibitory time constants modifies the extent of these regions and their overlap. As a consequence, for large Vs and small f , the interplay of the two types of synapses depends on these parameters. For sufficiently large θ and fast inhibition (the case considered in Figure 4) C(T/2) decreases and C(0) increases when electrical coupling is added to inhibitory coupling, compared to the case of inhibition alone. The opposite effect occurs for slow inhibition and small θ (result not shown). This is similar to what Lewis and Rinzel (2003) found for the LIF, but it occurs only in a restricted domain of the phase diagrams of the QIF. The dynamics of networks consisting of a large number of inhibitory interneurons coupled also via electrical synapses have been investigated recently (Traub et al., 2001; Bartos et al., 2002). Bartos et al. found that in a network of Wang-Buszaki (WB) neurons, electrical synapses enhanced synchrony even in the absence of inhibition. This agrees with our work, since one can check that the PRF of WB neurons is qualitatively similar to the one of our model in the absence of INaP . Traub et al. found that in a network of multicompartmental neurons coupled with inhibition, electrical synapses located on the dendrites improve synchrony. The mechanism underlying this effect is not clear since Traub et al. did not study their model when inhibition is suppressed.

5.3.2 Experimental Work. Recent experimental work suggests that electrically coupled pairs of LTS and pairs of FS cells have different synchronization properties (Mancilla, Lewis, Pinto, Rinzel, & Connors, 2002). Alvarez et al. (2002) have shown that synchrony of neurons in locus coeruleus (LC) interacting via electrical synapses is affected differentially by the firing rate of the neurons in adult and neonatal slices. It would be interesting to investigate experimentally whether these differences in synchronization properties can be explained by differences in the neuronal PRF, as our work suggests. The synchronizing effect of electrical synapses has been also assessed in neocortex and hippocampus by showing a reduction in synchronous oscillations in the gamma frequency range after suppressing Cx36-based electrical synapses (Deans et al., 2001; Hormuzdi et al., 2001). In the work of Tamas et al. (2000), both inhibitory and electrical synapses were found to be required to obtain tight synchrony in a neocortical slice. Although all of these studies support the role of electrical synapses in synchrony, the underlying mechanism remains to be defined in the light of our work, which shows that synchrony occurs in different ways depending on the intrinsic properties of the neurons and on the strength of the interactions.

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

τ/T

τ/T

gNaP

C

τ/T

τ/T

gNaP

C(t) τ/T

C(t)

τ/T

C(t)

τ/T

Large gNaP C(t)

C(t)

C(t)

Frequency

C(t)

τ/T

C(t)

C(t)

C(t)

τ/T

Inhibitory coupling

τ/T

C(t)

B

Electrical coupling

C(t)

Frequency

A

ggap

664

τ/T

τ/T

ginh

Figure 11: Predictions for the dependence of the shape of CCs on cellular and synaptic properties. (A) Qualitative predictions for the effect of changing gNaP and the firing frequency for the CCs of a pair of neurons interacting solely via electrical synapses. The coupling strength is fixed. (B) Qualitative predictions for a pair of neurons interacting solely via inhibitory synapses. The synaptic coupling conductance is fixed. (C) Qualitative predictions for the effects of the inhibitory and electrical coupling strength on the shapes of the CCs for dual coupling. The persistent-sodium conductance and the firing rates of the neurons are fixed and sufficiently large.

5.4 Predictions for Experiments. The dynamic clamp technique (Sharp et al., 1993; Prinz et al., 2004; Brizzi et al., 2004) can be used to modify artificially the cellular properties of neurons as well as the couplings between neurons. Therefore, it can serve to study how the dynamics of neurons depend on their intrinsic properties or the nature of their interactions (Prinz et al., 2004; Perez-Velazquez, Carlen, & Skinner, 2001). This approach can be used to test experimentally the predictions of this work, for instance, in pairs of cortical interneurons activated by injection of external noisy currents. Pairs of interneurons interact frequently via electrical and inhibitory synapses. A first experiment would be to record from pairs coupled only via electrical or inhibitory synapses to study how their CCs vary as a function of the average firing rates. We predict different behaviors, as schematically depicted in Figures 11A and 11B. Which of these behaviors actually occurs depends on the balance between inward and outward currents in the cells and on the location of the synapses on their dendritic trees. A second experiment would be to study the effect of the modifications of the cellular properties. We focus here on the effect of INaP . Depending on the outcomes of the first experiment, the effect would have to be tested by either increasing or decreasing the importance of this current (a decrease can be achieved by simulating a persistent sodium current with a negative conductance). We also predict that the effect of combination between inhibitory and electrical coupling on CCs depends on their strength as depicted in Figure 11C. Of particular interest would be to test the nonlinear interplay of these interactions.

Inhibitory and Electrical Synapses in Synchrony

665

Appendix A: Parameters of the Conductance-Based Model The dynamics of the membrane potentials of the somatic and dendritic compartments of our conductance-based model are given by C

dV = Iext −IL −INa −IK −INaP −gc (V − Vd )+Inoise +Igap +Iinh dt

(A.1)

C

dVd = −gl (Vd − Vld ) − gsd(Vd − V) + Igap , dt

(A.2)

where IL = −gL (V − VL ) is a leak current, Iext is a constant external current, Inoise a gaussian white noise with zero mean, and standard deviation, σ . The synaptic current received by the neuron i from the neuron j is Iinh = −ginh sj (Vi − Vinh ) for the inhibitory current, and Igap = −ggap (Vi − Vj ) or Igap = −ggap (Vdi − Vdj ) for current due to an electrical synapse respectively located between somatic or dendritic compartments. INa = gNa m3∞ h (V−VNa ) and IK = gK n4 (V−VK ) are the spike-generating currents, and INaP = gNaP p∞ (V − VNa ) is a noninactivating (persistent) sodium current. The kinetics of the gating variable h, n, s are given by dx = αx (V)(1 − x) − βx (V)x, dt

(A.3)

with x = h, n, s and αh (V) = 0.21 e−(V+58)/20 , βh (V) = 3./(1 + e−(V+28)/10 ), αs (V) = 50 (1 + tanh(V/4)), βs (V) = 1/τinh The activation functions, m∞ and p∞ , are given by m∞ (V) = αm (V)/(αm (V) + βm (V)), where αm (V) = 0.1(V + 35)/(1 − e−(V+35)/10 ), βm (V) = 4e−(V+60)/18 , and p∞ (V) = 1/(1 + e−(V+40)/6 ). Throughout this work, the parameters gNa = 35 mS/cm2 , VNa = 55 mV, gK = 10 mS/cm2 , VK = −75 mV, gL = 0.15 mS/cm2 , VL = −65 mV, gld = 0.15 mS/cm2 , Vld = −65 mV, Vinh = −75 mV, τinh = 6 msec and C = 1 µF/cm2 are kept constant. The conductance gNaP varies from 0 to 0.7mS/cm2 , and the conductance between the dendritic and somatic compartment is either gc = 0 (one-compartment model) or 0.3 mS/cm2 . Appendix B: Relationship Between the Phase-Shift Distribution Function and the Cross-Correlation in the Weak Coupling and Weak Noise Limit The normalized cross-correlation of the spike trains is (see equation 2.14) C(t2 − t1 ) =

< S1 (t1 )S2 (t2 ) > , < S1 (t) >< S2 (t) >

(B.1)

where S1 (t) is defined as 1/δ if there is a spike in a bin with a size δ and 0 otherwise. The average firing rate of the neuron is < S1 (t) >= P1 (t) = f .

666

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

We define the joint probability P1,2 (t1 , t2 ) as the probability that neuron 1 fires between t1 and t1 + δ and neuron 2 fires between t2 and t2 + δ. Then, < S1 (t1 )S2 (t2 ) >= P1,2 (t1 , t2 ) = P2|1 (t2 |t1 ) P(t1 ),

(B.2)

where P2|1 is the conditional probability that neuron 2 fires between t2 and t2 + δ given that neuron 1 fired between t1 and t1 + δ. Assuming stationarity, this function depends solely on the difference τ = t2 −t1 : P2|1 (t2 |t1 ) = P˜2|1 (τ ), and hence < S1 (t1 )S2 (t1 + τ ) > P˜2|1 (τ ) = . < S1 (t) >< S2 (t) > f

(B.3)

Switching from a function of time to a function of phase, we use the equation     P˜2|1 (τ )dτ = P0 Tt d Tτ , to obtain C(τ ) ≡

τ < S1 (t1 )S2 (t1 + τ ) > = P0 , < S1 (t) >< S2 (t) > T

(B.4)

where P0 is the probability distribution of the phase shift between the two neurons. Acknowledgments We thank B. Connors, C. Landisman, and C. van Vreeswijk for useful discussions and C. Meunier for careful reading of this manuscript. This work was supported in part by a NATO PST-CLG (reference 977683), the ACI “Neurosciences integratives et computationnelles” (Minist`ere de la Recherche, France), and project SECyT-ECOS A99E01. D.G. was supported by the Israel Science Foundation (grant no. 657/01). References Abbott, L. F., & van Vreeswijk, C. (1993). Asynchronous states in networks of pulse-coupled oscillators. Phys. Rev. E, 48, 1483–1490. Acker, C. D., Kopell, N., & White, J. A. (2003). Synchronization of strongly coupled excitatory neurons: Relating network behavior to biophysics. J. Comput. Neurosci., 15, 71–90. Alvarez, L. F., Chow, C., van Bockstaele, E. J., & Williams, J. T. (2002). Frequencydependent synchrony in locus coeruleus: Role of electrotonic coupling. Proc. Natl. Acad. Sci., 99, 4032–4036. Amitai, Y., Gibson, J. R., Patrick, A., Ho, B., Connors, B. W., & Golomb, D. (2002). Spatial organization of electrically coupled network of interneurons in neocortex. J. Neurosci., 22, 4142–4152. Bartos, M., Vida, I., Frotscher, M., Meyer, A., Monyer, H., Geiger, J. R., & Jonas, P. (2002). Fast synaptic inhibition promotes synchronized gamma oscillations in hippocampal interneuron networks. Proc. Natl. Acad. Sci., 99, 13222–13227.

Inhibitory and Electrical Synapses in Synchrony

667

Beierlein, M., Gibson, J. R., & Connors, B. W. (2000). A network of electrically coupled interneurons drives synchronized inhibition in neocortex. Nat. Neurosci., 3, 904–910. Bem, T., & Rinzel, J. (2004). Short duty cycle destabilizes a half-center oscillator, but gap junctions can restabilize the anti-phase pattern. J. Neurophysiol., 91, 693–703. Blatow, M., Rozov, A., Katona, I., Hormuzdi, S. G., Meyer, A. H., Whittington, M. A., Caputi, A., & Monyer, H. (2003). A novel network of multipolar bursting interneurons generates theta frequency oscillations in neocortex. Neuron, 38, 805–817. Bou-Flores, C., & Berger, A. J. (2001). Gap junctions and inhibitory synapses modulate inspiratory motoneuron synchronization. J. Neurophysiol., 85, 1543– 1551. Brizzi, L., Meunier, C., Zytnicki, D., Donnet, M., Hansel, D., Lamotte D’Incamps, B., & van Vreeswijk, C. (2004). How shunting inhibition affects the discharge of lumbar motoneurones. A dynamic clamp study in anaesthetized cats. J. Physiol, 558, 671–683. Brunel, N., & Hakim, V. (1999). Fast global oscillations in networks of integrateand-fire neurons with low firing rates. Neural Comput., 11, 1621–1671. Chow, C. C., & Kopell, N. (2000). Dynamics of spiking neurons with electrical coupling. Neural Comput., 12, 1643–1678. Crook, S. M., Ermentrout, G. B., & Bower, J. M. (1998a). Spike frequency adaptation affects the synchronization properties of networks of cortical oscillations. Neural Comput., 10, 837–854. Crook, S. M., Ermentrout, G. B., & Bower, J. M. (1998b). Dendritic and synaptic effects in systems of coupled cortical oscillators. J. Comput. Neurosci., 5, 315– 329. Deans, M. R., Gibson, J. R., Sellitto, C., Connors, B. W., & Paul, D. L. (2001). Synchronous activity of inhibitory networks in neocortex requires electrical synapses containing connexin36. Neuron, 31, 477–485. Draguhn, A., Traub, R. D., Schmitz, D., & Jefferys, J. G. (1998). Electrical coupling underlies high-frequency oscillations in the hippocampus in vitro. Nature, 394, 189–192. Ermentrout, B. (1981). n:m phase-locking of weakly coupled oscillators. J. Math. Biol., 12, 327–342. Ermentrout, B. (1996). Type I membranes, phase resetting curves and synchrony. Neural Comput., 8, 979-1001. Ermentrout, B., & Kopell, N. (1986). Parabolic bursting in an excitable system coupled with a slow oscillation. SIAM J. Appl. Math., 46, 233–253. Ermentrout, B., Pascal, M., & Gutkin, B. (2001). The effects of spike frequency adaptation and negative feedback on the synchronization of neural oscillators. Neural Comput., 13, 1285–1310. Friedman, D., & Strowbridge, B. W. (2003). Both electrical and chemical synapses mediate fast network oscillations in the olfactory bulb. J. Neurophysiol., 89, 2601–2610.

668

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

Fukuda, T., & Kosaka, T. (2000). Gap-junction coupling linking the dendritic network of GABAergic neurons in the hippocampus. J. Neurosci., 20, 1519– 1528. Furshpan, E. J., & Potter, D. D. (1959). Transmission at the giant motor synapses of the crayfish. J. Physiol., 145, 289–325. Galarreta, M., & Hestrin, S. (1999). A network of fast spiking cells in the neocortex connected by electrical synapses. Nature, 402, 72–75. Galarreta, M., & Hestrin, S. (2001). Electrical synapses between GABA-releasing neurons. Nature Neurosci., 2, 425–433. Galarreta, M., & Hestrin, S. (2002). Electrical and chemical synapses among parvalbumin fast-spiking GABAergic interneurons in adult mouse neocortex. Proc. Natl. Acad. Sci. USA., 19, 12438–12443. Gerstner, W., & Kistler W. M. (2002). Spiking neuron models: Single neurons, populations, plasticity. Cambridge: Cambridge University Press. Gibson, J. R., Beierlein, M., & Connors, B. (1999). Two networks of inhibitory neurons electrically coupled. Nature, 402, 75–79. Golomb, D., Hansel, D., & Mato, G. (2001). Mechanisms of synchrony of neural activity in large networks. In F. Moss & S. Gielen (Eds.), Handbook of biological physics, Vol. 4: Neuro-informatics and neural modeling (pp. 887–968). Amsterdam: Elsevier Science. Golomb, D., Wang, X. J., & Rinzel, J. (1994). Synchronization properties of spindle oscillations in a thalamic reticular nucleus model. J. Neurophysiol., 72, 1109– 1126. Han, S. K., Kurrer, C., & Kuramoto, Y. (1995). Dephasing and bursting in coupled neural oscillators. Phys. Rev. Lett., 75, 3190–3193 Hansel, D., & Mato, G. (2003) Asynchronous states and the emergence of synchrony in large networks of interacting excitatory and inhibitory neurons. Neural Comput., 15, 1–56. Hansel, D., Mato, G., & Meunier, C. (1993). Phase dynamics for weakly coupled Hodgkin-Huxley neurons. Europhys. Lett., 23, 367–372. Hansel, D., Mato, G., & Meunier, C. (1995). Synchrony in excitatory neural networks. Neural Comput., 7, 307–337. Hormuzdi, S. G., Pais, I., Lebeau, F. E .N., Towers, S. K., Rozov, A., Buhl, E., Whittington, M. A., & Monyer, H. (2001). Impaired electrical signaling disrupts gamma frequency oscillations in connexin 36-deficient mice. Neuron, 31, 487–495. Izhikevich, E. M. (1998). Phase models with explicit time delays. Phys. Rev. E, 58, 905–908 Kita, H., Kosaka, T., & Heizmann, C. W. (1990). Parvalbumin-immunoreactive neurons in the rat neostriatum: A light and electron microscopic study. Brain Res., 536, 1–15. Kuramoto, Y. (1984). Chemical oscillations, waves and turbulence. New York: Springer. Kuramoto, Y. (1991). Collective synchronization of pulse-coupled oscillators and excitable units. Physica D, 50, 15–30.

Inhibitory and Electrical Synapses in Synchrony

669

Landisman, C. E., Long, M. A., Beierlein, M., Deans, M. R., Paul, D. L., & Connors, B. W. (2002). Electrical synapses in the thalamic reticular nucleus. J. Neurosci., 22, 1002–1009. Latham, P. E., Richmond, B. J., Nelson P. G., & Nirenberg, S. (2000). Intrinsic dynamics in neuronal networks. I. Theory. J. Neurophysiol., 83, 808–827. Lewis, T., & Rinzel, J. (2003). Dynamics of spiking neurons connected by both inhibitory and electrical coupling. J. Comput. Neurosci., 14, 283–309. Mancilla, J. G., Lewis, T., Pinto, D. J., Rinzel, J., & Connors, B. W. (2002). Firing dynamics of single and coupled pairs of inhibitory interneurons in neocortex. Program No. 840.13. Abstract Viewer/Itinerary Planner. Washington, DC: Society for Neuroscience. Available at: http://apu.sfn.org. Mann-Metzer, P., & Yarom, Y. (1999). Electrical coupling interacts with intrinsic properties to generate synchronized activity in cerebellar networks of inhibitory interneurons. J. Neurosci., 19, 3298–3306. Nomura, M., Fukai, T., & Aoyagi, T. (2003). Synchrony of fast-spiking interneurons interconnected by GABAergic and electrical synapses. Neural Comput., 15, 2179–2198. Oprisan, S. A., & Canavier, C. C. (2002). The influence of limit cycle topology on the phase resetting curve. Neural Comput, 14, 1027–1057. Perez-Velazquez, J. L., Carlen, P. L. & Skinner F. K. (2001). Artificial electrotonic coupling affects neuronal firing patterns depending upon cellular characteristics. Neuroscience, 103, 841–849. Perez-Velazquez, J. L., Valiante, T. A., & Carlen, P. L. (1994). Modulation of gap junctional mechanisms during calcium-free induced field burst activity: A possible role for electrotonic coupling in epileptogenesis. J. Neurosci., 14, 4308–4317. Pfeuty, B., Mato, G., Golomb, D., & Hansel, D. (2003). Electrical synapses and synchrony: The role of intrinsic currents. J. Neurosci, 23, 6280–6294. Prinz, A. A., Abbott, L. F., & Marder A. (2004). The dynamic clamp comes of age. Trends Neurosci., 27, 228–234. Sharp, A. A., O’Neil, M. B., Abbott, L. F., & Marder, E. (1993). The dynamic clamp: Artificial conductances in biological neurons. Trends Neurosci., 16, 389–394. Sherman, A., & Rinzel, J. (1992). Rhythmogenic effects of weak electrotonic coupling in neuronal models. Proc. Natl. Acad. Sci., 89, 2471–2474. Skinner, F. K., Zhang, L., Velazquez, J. L., & Carlen, P. L. (1999). Bursting in inhibitory interneuronal networks: A role for gap-junctional coupling. J. Neurophysiol., 81, 1274–1283. Tamas, G., Buhl, E. H., Lorincz, A., & Somogyi, P. (2000). Proximally targeted GABAergic synapses and gap-junctions precisely synchronize cortical interneurons. Nature Neurosci., 3, 366–371. Traub, R. D., Kopell, N., Bibbig, A., Buhl, E., Lebeau F. E. N., & Whittington, M. (2001). Gap junction between interneuron dendrites can enhance synchrony of gamma oscillations in distributed networks. J. Neurosci., 21, 9478–9486. van Kampen, N. G. (1981). Stochastic processes in physics and chemistry. Amsterdam: Elsevier Science. van Vreeswijk, C., Abbott, L. F., & Ermentrout, G. B. (1994). When inhibition not excitation synchronizes neural firing. J. Comput. Neurosci., 1, 313–321.

670

B. Pfeuty, G. Mato, D. Golomb, and D. Hansel

Venance, L., Rozov, A., Blatow, M., Burnashev, N., Feldmeyer, D., & Monyer, H. (2000). Connexin expression in electrically coupled postnatal rat brain neurons. Proc. Natl. Acad. Sci., 97, 10260–10265. Wang, X. J., & Busz´aki, J. (1996). Gamma oscillation by synaptic inhibition in a hippocampal interneuronal network model. J. Neurosci., 16, 6402–6413. Wang, X. J., & Rinzel, J. (1992). Alternating and synchronous rhythms in reciprocally inhibitory model neurons. Neural Comput., 4, 84–117 Watanabe, A. (1958). The interaction of electrical activity among neurons of lobster cardiac ganglion. Jap. J. Physiol., 8, 305–318. Received June 1, 2004; accepted August 3, 2004.

LETTER

Communicated by Chris Watkins

Spikernels: Predicting Arm Movements by Embedding Population Spike Rate Patterns in Inner-Product Spaces Lavi Shpigelman [email protected] School of Computer Science and Engineering and Interdisciplinary Center for Neural Computation, Hebrew University, Jerusalem 91904, Israel

Yoram Singer [email protected] School of Computer Science and Engineering Hebrew University, Jerusalem 91904, Israel

Rony Paz [email protected]

Eilon Vaadia [email protected] Interdisciplinary Center for Neural Computation and Department of Physiology, Hadassah Medical School The Hebrew University Jerusalem, 91904, Israel

Inner-product operators, often referred to as kernels in statistical learning, define a mapping from some input space into a feature space. The focus of this letter is the construction of biologically motivated kernels for cortical activities. The kernels we derive, termed Spikernels, map spike count sequences into an abstract vector space in which we can perform various prediction tasks. We discuss in detail the derivation of Spikernels and describe an efficient algorithm for computing their value on any two sequences of neural population spike counts. We demonstrate the merits of our modeling approach by comparing the Spikernel to various standard kernels in the task of predicting hand movement velocities from cortical recordings. All of the kernels that we tested in our experiments outperform the standard scalar product used in linear regression, with the Spikernel consistently achieving the best performance.

1 Introduction Neuronal activity in primary motor cortex (MI) during multijoint arm reaching movements in 2D and 3D (Georgopoulos, Schwartz, & Kettner, 1986; Neural Computation 17, 671–690 (2005)

c 2005 Massachusetts Institute of Technology


672

L. Shpigelman, Y. Singer, R. Paz, and E. Vaadia

Georgopouls, Kettner, & Schwartz, 1988) and drawing movements (Schwartz, 1994) has been used extensively as a test bed for gaining understanding of neural computations in the brain. Most approaches assume that information is coded by firing rates, measured on various timescales. The tuning curve approach models the average firing rate of a cortical unit as a function of some external variable, like the frequency of an auditory stimulus or the direction of a planned movement. Many studies of motor cortical areas (Georgopoulos, Kalaska, & Massey, 1983; Georgopoulos et al., 1988; Moran & Schwartz, 1999; Schwartz, 1994; Laubach, Wessberg, & Nicolelis, 2000) showed that while single units are broadly tuned to movement direction, a relatively small population of cells (tens to hundreds) carries enough information to allow for accurate prediction. Such broad tuning can be found in many parts of the nervous system, suggesting that computation by distributed populations of cells is a general cortical feature. The population vector method (Georgopoulos et al., 1983, 1988) describes each cell’s firing rate as the dot product between that cell’s preferred direction and the direction of hand movement. The vector sum of preferred directions, weighted by the measured firing rates, is used both as a way of understanding what the cortical units encode and as a means for estimating the velocity vector. Several recent studies (Fu, Flament, Cotz, & Ebner, 1997; Donchin, Gribova, Steinberg, Bergman, & Vaadia, 1998; Schwartz, 1994) propose that neurons can represent or process multiple parameters simultaneously, suggesting that it is the dynamic organization of the activity in neuronal populations that may represent temporal properties of behavior such as the computation of transformation from desired action in external coordinates to muscle activation patterns. A few studies (Vaadia, et al., 1995; Laubach, Schuler, & Nicolelis, 1999; Reihle, Grun, Diesmann, & Aersten, 1997) support the notion that neurons can associate and dissociate rapidly to functional groups in the process of performing a computational task. The concepts of simultaneous encoding of multiple parameters and dynamic representation in neuronal populations together could explain some of the conundrums in motor system physiology. These concepts also invite using of increasingly complex models for relating neural activity to behavior. Advances in computing power and recent developments of physiological recording methods allow recording of ever growing numbers of cortical units that can be used for real time analysis and modeling. These developments and new understandings have recently been used to reconstruct movements on the basis of neuronal activity in real time in an effort to facilitate the development of hybrid brain-machine interfaces that allow interaction between living brain tissue and artificial electronic or mechanical devices to produce brain-controlled movements (Chapin, Moxon, Markowitz, & Nicolelis, 1999; Laubach et al., 2000; Nicolelis, 2001; Wessberg et al., 2000; Laubach et al., 1999; Nicolelis, Ghazanfar, Faggin, Votaw, & Oliveira, 1997; Isaccs, Weber, & Schwartz, 2000).

Spikernels

673

Most of the current attempts that focus on predicting movement from cortical activity rely on modeling techniques that employ parametric models to describe a neuron’s tuning (instantaneous spike rate) as a function of current behavior. For instance, cosine-tuning estimation (population vector) is used by Taylor, Tillery, and Schwartz (2002), and linear regression was employed by Wessberg et al. (2000) and Serruya, Hatsopoulos, Paninski, Fellows, and Donoghue (2002). Brown, Frank, Tang, Quirk, and Wilson (1998) and Brockwell, Rojas, and Kass (2004) have applied filtering methods that, apart from assuming parametric models of spike rate generation, also incorporate a statistical model of the movement. An exception is the use of artificial neural nets (Wessberg et al, 2000) (though this study reports getting better results by linear regression). This article describes the tailoring of a kernel method for the task of predicting two-dimensional hand movements. The main difference between our method and the other approaches is that our nonparametric approach does not assume cell independence, imposes relatively few assumptions on the tuning properties of the cells and how the neural code is read, allowing representation of behavior in highly specific neural patterns, and, as we demonstrate later, results in better predictions on our test sets. However, due to the implicit manner in which neural activity is mapped to feature space, improved results are often achieved at the expense of understanding tuning of individual cells. We attempt to partially overcome this difficulty by describing the feature space that our kernel induces and by explaining the way our kernel parameters affect the features. The letter is organized as follows. In section 2, we describe the problem setting that this article is concerned with. In section 3, we introduce and explain the main mathematical tool that we use: the kernel operator. In section 4, we discuss the design and implementation of a biologically motivated kernel for neural activities. The experimental method is described in section 5. We report experimental results in section 6 and give conclusions in section 7. 2 Problem Setting Consider the case where we monitor instantaneous spike rates from q cortical units during physical motor behavior of a subject (the method used for translating recorded spike times into rate measures is explained in section 5). Our goal is to learn a predictive model of some behavior parameter (such as hand velocities, as described in section 5) with the cortical activity as the input. Formally, let S ∈ Rq×m be a sequence of instantaneous firing rates from q cortical units consisting of m time samples. We use S, T to denote sequences of firing rates and denote by len(S) the time length of a sequence S (len(S) = m in above definition). Si ∈ Rq designates the instantaneous firing rates of all neurons at time i in the sequence S and Si,k ∈ R the rate of neuron k. We also use Ss to denote the concatenation of S with one more sample s ∈ Rq . The instantaneous firing rate of a unit k in sample s is then

674

L. Shpigelman, Y. Singer, R. Paz, and E. Vaadia

sk ∈ R. We also need to employ a notation for subsequences. A consecutive subsequence of S from time t1 to time t2 is denoted St1 :t2 . Finally, throughout the work, we need to examine possibly nonconsecutive subsequences. We denote by i ∈ Rn a vector of time indices into the sequence S such that 1 ≤ i1 < i2 < · · · < in ≤ len(S). Si ∈ Rq×n is then an ordered selection of spike rates (of all neurons) from times i. Let y ∈ Rm denote a parameter of the movement that we would like to predict (e.g., the movement velocity in the horizontal direction, vx ). Our goal is to learn an approximation
y of the form f: Rq×m → Rm from neural firing rates to movement parameter. In general, information about movement can be found in neural activity both before and after the time of movement itself. Our long-term plan, though, is to design a model that can be used for controlling a neural prosthesis. We therefore confine ourselves to causal predictors that use an l (l ∈ Z) long window of neural activity ending  at time step t, S(t−l+1):t , to predict yt ∈ R. We would like to make
yt = ft S(t−l+1):t as close as possible (in a sense that is explained in the sequel) to yt . 3 Kernel Method for Regression The major mathematical tool employed in this article is kernel operators. Kernel operators allow algorithms whose interface to the data is limited to scalar products to employ complicated premappings of the data into feature spaces by use of kernels. Formally, a kernel is an inner-product operator K: X × X → R where X is some arbitrary vector space. In this work, X is the space of neural activities (specifically, population spike rates). An explicit way to describe K is via a mapping φ: X → H from X to a feature space H such that K(x, x ) = φ(x) · φ(x ). Given a kernel operator, we can use it to perform various statistical learning tasks. One such task is support vector regression (SVR) (see, e.g., Smola & Scholkopf, ¨ 2004) which attempts to find a regression function for target values that is linear if observed in the (typically very high-dimensional) feature space mapped by the kernel. We give here a brief description of SVR for clarity. SVR employs the ε-insensitive loss function (Vapnik, 1995). Formally, this loss is defined as       yt − f (S(t−l+1):t ) = max 0, yt − f (S(t−l+1):t ) − ε . ε Examples that fall within ε of the target value (yt ) do not contribute to the error. Those that fall further away contribute linearly to the loss (see Figure 1, left). The form of the regression model is   f (S(t−l+1):t ) = w · φ S(t−l+1):t + b,

(3.1)

which is linear in the feature space H and is implicitly defined by the kernel. Combined with a loss function, f defines a hyperslab of width ε centered at the estimate (see Figure 1, right, for a single-dimensional illustration).

Spikernels

675 y f(x)

Loss

ε

ε

y − yˆ

φ(x)

Figure 1: Illustration of the ε-insensitive loss (left) and an ε-insensitive area around a linear regression in a single-dimensional feature space (right). Examples that fall within distance ε have zero loss (shaded area).

For each trial i and time index t designating a frame within the trial, we received a pair, (Si(t−l+1):t , yit ), consisting of spike rates and a corresponding target value. The optimization problem employed by SVR is arg min w

    1  w2 + C yit − f Si(t−l+1):t  , ε 2 i,t

(3.2)

where ·2 is the squared vector norm. This minimization casts a trade-off (weighed by C ∈ R) between a small empirical error and a regularization term that favors low sensitivity to feature values. Let φ(x) denote the feature vector that is implicitly implemented by kernel function K(·, x). Then there exists a regression model equivalent to the one given in equation 3.1, is a minimum of equation 3.2, and takes the form  f (T) = At,i K(Si(t−l+1):t , T) + b. t,i

Here, T is the observed neural activity, and A is a (typically sparse) matrix of Lagrange multipliers determined by the minimization problem in equation 3.2. In summary, SVR solves a quadratic optimization problem aimed at finding a linear regressor in an induced high-dimensional feature space. This regressor is the best penalized estimate for the ε-insensitive loss function where the penalty is the squared norm of the regressor’s weights. 4 Spikernels The quality of kernel-based learning is highly dependent on how the data are embedded in the feature space via the kernel operator. For this reason, several studies have been devoted to developing new kernels (Jaakola & Haussler, 1998; Genton, 2001; Lodhi, Shawe-Taylor, Cristianini, & Watkins, 2000). In fact, a good kernel could render the work of a classification or regression algorithm trivial. With this in mind, we develop a kernel for neural spike rate activity.

676

L. Shpigelman, Y. Singer, R. Paz, and E. Vaadia

R a te

Pattern A Pattern B

T im e

(a) Bin by bin P attern A P attern B

P attern A P attern B

R a te

R a te

Time of Interest

T im e

(b) Time warp

T im e

(c) Time of interest

Figure 2: Illustrative examples of pattern similarities. (a) Bin-by-bin comparison yields small differences. (b) Patterns with large bin-by-bin differences that can be eliminated with some time warping. (c) Patterns whose suffix (time of interest) is similar and prefix is different.

4.1 Motivation. Our goal in developing a kernel for spike trains is to map similar patterns to nearby areas of the feature space. In the description of our kernel, we attempt to capture some well-accepted notions on similarities between spike trains. We make the following assumptions regarding similarities between spike patterns: • A commonly made assumption is that firing patterns that have small differences in a bin-by-bin comparison are similar. This is the basis of linear regression. In Figure 2a, we show an example of two patterns that are bin-wise similar. • A cortical population may display highly specific patterns to represent specific information such as a particular stimulus or hand movement.

Spikernels

677

Though in this work we make use of spike counts within each time bin, we assume that a pattern of spike counts may be specific to an external stimulus or action and that the manner in which these patterns change as a function of the behavior may be highly nonlinear (Segev & Rall, 1998). Many models of neuronal computation are based on this assumption (see, e.g., Pouget & Snyder, 2000). • Two patterns may be quite different from a simple bin-wise perspective, yet if they are aligned using a nonlinear time distortion or shifting, the similarity becomes apparent. An illustration of such patterns is given in Figure 2b. In comparing patterns, we would like to induce a higher score when the time shifts are small. Some biological plausibility for time-warped integration of inputs may stem from the spatial distribution of synaptic inputs on cortical cells’ dendrites. In order for two signals to be integrated along a dendritic tree or at the axon hillock, their sources must be appropriately distributed in space and time, and this distribution may be sensitive to the current depolarization state of the tree (Magee, 2000). • Patterns that are associated with identical values of an external stimulus at time t may be similar at that time but rather different at t ±  when values of the external stimulus for these patterns are no longer similar (as illustrated in Figure 2c. We would like to give a higher similarity score to patterns that are similar around a time of interest (prediction time), even if they are rather different at other times. 4.2 Spikernel Definition. We describe our kernel construction, the Spikernel, by specifying the features that make up the feature space. Our construction of the feature space builds on the work of Haussler (1999), Watkins (1999), and Lodhi et al. (2000). We chose to directly specify the features constituting the kernel rather than describe the kernel from a functional point of view, as it provides some insight into the structure of the feature space. We first need to introduce a few more notations. Let S be a sequence of length l = len(S). The set of all possible n-long index vectors defining a subsequence of S is In,l = i: i ∈ Zn , 1 ≤ i1 < · · · < in ≤ l . Also, let d(α, β ), (α, β ∈ Rq ) denote a bin-wise distance over a pair of samples (population rates). We also overload notation and denote by  

n firing d (Si , U) = k=1 d Sik , Uk a distance between sequences. The sequence distance is the sum of distances over the samples. One such function (the d we explore here) is the squared 2 norm. Let µ, λ ∈ (0, 1). The U component of our (infinite) feature vector φ(S) is defined as

φnU (S) =



µd(Si ,U) λlen(S)−i1 ,

(4.1)

i∈In,len(S)

where i1 is the first entry in the index vector i. In words, φnU (S) is a sum

678

L. Shpigelman, Y. Singer, R. Paz, and E. Vaadia

over all n-long subsequences of S. Each subsequence Si is compared to U (the feature coordinate) and contributes to the feature value by adding a number that is proportional to the bin-wise similarity between Si and U and is inversely proportional to the amount of stretching Si needs to undergo. That is, the feature entry indexed by U measures how similar U is to the latter part of the time series S. This definition seems to fit our assumptions on neural coding for the following reasons: • A feature φnU (S) gets large values only if U is bin-wise similar to subsequences of the pattern S. • It allows for learning complex patterns: choosing small values of λ and µ (or concentrated d measures) means that each feature tends toward being either 1 or 0, depending on whether the feature coordinate U is almost identical to a suffix of S. • We allow gaps in the indexes defining subsequences, thus allowing for time warping. • Subpatterns that begin further from the required prediction time are penalized by an exponentially decaying weight; thus, more weight is given to activities close to the prediction time (the end of the sequence), which designates our time of interest. 4.3 Efficient Kernel Calculation. The definition of φ given by equation 4.1 requires the manipulation of an infinite feature space and an iteration over all subsequences that grows exponentially with the lengths of the sequences. Straightforward calculation of the feature values and performing the induced inner product is clearly impossible. Building on ideas from Lodhi et al. (2000), who describe a similar kernel (for finite sequence alphabets and no time of interest), we developed an indirect method for evaluating the kernel through a recursion that can be performed efficiently using dynamic programming. The solution we now describe is rather general, as it employs an arbitrary base feature vector ψ . We later describe our specific design choice for ψ . The definition in equation 4.1 is a special case of the following feature definition,

φnU (S)

=

 i∈In,len(S)



  len(S)−i 1 ψ U k S ik λ ,

(4.2)

k

where n specifies the length of the subsequences, and for the Spikernel, we substitute     d S ,U ψ U k S ik = µ i k k .

Spikernels

679

The feature vector of equation 4.2 satisfies the following recursion:  len(S)  i=1 λlen(S)−i+1 ψ (Si ) φn−1 (S1:i−1 ) len(S) ≥ n > 0 n φ (S) = 0 len(S) < n, n > 0 (4.3)  1 n=0 The recursive form is derived by decomposing the sum over all subsequences into a sum over the last subsequence index and a sum over the rest of the indices. The fact that φn (S) is the zero vector for sequences of length less than n is due to equation 4.2. The last value of φn when n = 0 is defined to be the all-one vector. The kernel that performs the dot product between two feature vectors is then defined as  Kn (S, T) = φn (S) · φn (T) = φnU (S)φnU (T)dU, Rq×n

where n designates subsequences of length n as before. We now plug in the recursive feature vector definition from equation 4.3:

  len(S)  n−1 n len(S)−i+1 λ ψ Un (Si ) φU1:n−1 (S1:i−1 ) K (S, T) = Rq×n



×

i=1

len(T) 

      dU. λlen(T)−j+1 ψ Un Tj φn−1 U1:n−1 T1:j−1

(4.4)

j=1

Next, we note that  n−1 n−1 φn−1 (S1:i−1 ) · φn−1 (T1:j−1 ) U (S1:i−1 )φU (T1:j−1 )dU = φ Rq×n−1   = Kn−1 S1:i−1 , T1:j−1 , and also  Rq

ψ Uk (Si )ψ Uk (Tj )dUk = ψ (Si ) · ψ (Tj ).

Defining K∗ (Si , Tj ) = ψ (Si )·ψ (Si ), we now insert the integration in equation 4.4 into the sums to get Kn (S, T) len(S)  len(T)      n−1   = λlen(S)+len(T)−i−j+2 ψ (Si )· ψ Tj φ (S1:i−1 )· φn−1 T1:j−1 i=1 j=1 len(S)  len(T) 

=

i=1 j=1

    λlen(S)+len(T)−i−j+2 K∗ Si , Tj Kn−1 S1:i−1 , T1:j−1 .

(4.5)

680

L. Shpigelman, Y. Singer, R. Paz, and E. Vaadia

We have thus derived a recursive definition of the kernel, which inherits the initial conditions from equation 4.3: ∀S, T K0 (S, T) = 1   if min len(S), len(T) < i Ki (S, T) = 0.

(4.6)

The kernel function K∗ (·, ·) may be any kernel operator and operates on population spike rates sampled at single time points. To obtain the Spikernel described above, we chose K∗ (α, β ) =

 Rq

µd(u,α) µ



d u,β

 du.

A more time-efficient description of the recursive computation of the kernel may be obtained if we notice that the sums in equation 4.5 are used, up to a constant λ, in calculating the kernel for prefixes of S and T. Caching these sums yields Kn (Ss, Tt) = λKn (S,Tt)+λKn (Ss,T)−λ2 K(S,T)+λ2 K∗ (s, t) Kn−1 (S,T) , where s and t are the last samples in the sequences (and the initial conditions of equation 4.6 still hold). Calculating Kn (S, T) using the above recursion is computationally cheaper than equation constructing a three-dimensional  4.5 and comprises  array, yielding an O len(S) len(T) n dynamic programming procedure. 4.4 Spikernel Variants. The kernels defined by equation 4.1 consider only patterns of fixed length (n). It makes sense to look at sub-sequences of various lengths. Since a linear combination of kernels is also a kernel, we can define our kernel to be K(S, T) =

n 

pi Ki (S, T),

i=1

Rn+

is a vector of weights. The weighted kernel summation can where p ∈ be interpreted as performing the dot product between vectors that take the √ √ form [ p1 φ1 (·), . . . , pn φn (·)], where φi (·) is the feature vector that kernel i K (·, ·) represents. In the results, we use this definition with pi = 1 (a better weighing was not explored). Different choices of K∗ (α, β ) allow different methods of comparing population rate values (once the time bins for comparison were chosen). We  2 q  2 chose d(α, β ) to be the squared 2 norm: α − β 2 = k=1 αk − β k . The kernel K∗ (·, ·) then becomes  2 1 α−β 2 K∗ (α, β ) = cµ 2 ,

Spikernels

with c =



681 π −2 ln µ

q

; however, c can (and did) set to 1 WLOG. Note that  2 − ln(µ) α−β  2 and is therefore a K∗ (α, β ) can be written as K∗ (α, β ) = e 2 simple gaussian. It is also worth noticing that if λ ≈ 0, Kn (S, T) → ce−

ln(µ) S−T22 2

.

That is, in this limit, no time warping is allowed, and the Spikernel is reduced to the standard exponential kernel (up to a constant c). 5 Experimental Method 5.1 Data Collection. The data used in this work were recorded from the primary motor cortex of a rhesus (Macaca mulatta) monkey (approximately 4.5 kg). The animal’s care and surgical procedures accorded with The NIH Guide for the Care and Use of Laboratory Animals (rev. 1996) and with the Hebrew University guidelines supervised by the institutional committee for animal care and use. The monkey sat in a dark chamber, and eight electrodes were introduced into each hemisphere. The electrode signals were amplified, filtered, and sorted (MCP-PLUS, MSD, Alpha-Omega, Nazareth, Israel). The data used in this report were recorded on four different days during a one-month period of daily sessions. Up to 16 microelectrodes were inserted on each day. Recordings include spike times of single units (isolated by signal fit to a series of windows) and multiunits (detection by threshold crossing). The monkey used two planar-movement manipulanda to control two cursors (× and + shapes) on the screen to perform a center-out reaching task. Each trial begun when the monkey centered both cursors on a central circle for 1.0 to 1.5 s. Either cursor could turn green, indicating the hand to be used in the trial (× for right arm and + for the left). Then, after an additional hold period of 1.0 to 1.5ss one of eight targets (0.8 cm diameter) appeared at a distance 4 cm from the origin. After another 0.7 to 1.0ss, the center circle disappeared (referred to as the Go Signal), and the monkey had to move and reach the target in less than 2 s to receive liquid reward. We obtained data from all channels that exhibited spiking activity (i.e., were not silent) during at least 90% of the trials. The number and types of recorded units and the number of trials used in each of the recording days are described in Table 1. At the end of each session, we examined the activity of neurons evoked by passive manipulation of the limbs and applied intracortical microstimulation (ICMS) to evoke movements. The data presented here were recorded in penetration sites where ICMS evoked shoulder and elbow movements. Identification of the recording area was also aided by MRI. More information can be found in (Paz, Boraud, Natan, Bergman, and Vaadia (2003).

682

L. Shpigelman, Y. Singer, R. Paz, and E. Vaadia

Table 1: Summary of the Data Sets Obtained.

Data Set

Number of Single Units

1 2 3 4

27 22 27 25

Number of Mean Number of Multiunit Spike Rate Successful Trials Channels (spikes/sec) 16 9 10 13

8.6 11.6 8.0 6.5

317 333 155 307

Net Cumulative Movement Time (sec) 529 550 308 509

The results that we present here refer to prediction of instantaneous hand velocities (in 2D) during the movement time (from Go Signal to Target Reach times) of both hands in successful trials. Note that some of the trials required movement of the left hand while keeping the right hand steady and vice versa. Therefore, although we considered only movement periods of the trials, we had to predict both movement and nonmovement for each hand. 5.2 Data Preprocessing and Modeling. The recordings of spike times were made at 1 ms precision, and hand positions were recorded at 500 Hz. Our kernel method is suited for use with windows of observed neural activity as time series and with the movement values at the ends of those windows. We therefore chose to partition spike trains into bins, count the spikes in each bin, and use a running window of such spike counts as our data with the hand velocities at the end of the window as the label. Thus, a labeled example (St , vt ) for time t consisted of the X (left-right) or Y (forward-backward) velocity for the left or right hand as the target label vt and the preceding window of spike counts from all q cortical units as the input sequence St . In one such preprocessing, we chose a running window of 10 bins of size 100 ms each (a 1-second-long window), and the time interval between two such windows was 100 ms. Two such consecutive examples would then have nine time bins of overlap. For example, the number of cortical units q in the first data set was 43 (27 single +16 multiple), and the total length of all the trials used in that data set is 529 seconds. Hence, in that session, with the above preprocessing, there are 5290 consecutive examples, where each is a 43 × 10 matrix of spike counts and there are 4 sequences of movement velocities (X/Y and R/L hand) of the same length. Before we could train an algorithm, we had to specify the time bin size, the number of bins, and the time interval between two observation windows. Furthermore, each kernel employs a few parameters (for the Spikernel, they are λ, µ, n, and pi , though we chose pi = 1 a priori) and the SVM regression setup requires setting two more parameters, ε and C. In order to test the algorithms, we first had to establish which parameters work best. We used the first half of data set 1 in fivefold cross-validation to choose the best

Spikernels

683

Table 2: Parameter Values Tried for Each Kernel. Kernel

Parameters

Spikernel

λ ∈ {0.5, 0.7, 0.9, 0.99}, µ ∈ {0.7, 0.9, 0.99, 0.999}, n ∈ {3, 5, 10}, pi = 1, C = 0.1, ε = 0.1 γ ∈ {0.1, 0.01, 0.001, 0.005, 0.0001}, C ∈ {1, 5, 10}, ε = 0.1 C ∈ {0.001, 10−4 , 10−5 }, ε = 0.1 C ∈ {10−4 , 10−5 , 10−6 }, ε = 0.1 C ∈ {100, 10, 1, 0.1, 0.01, 0.001, 10−4 }, ε = 0.1

Exponential Polynomial, second degree Polynomial, third degree Linear

Notes: Not all combinations were tried for the Spikernel. n was chosen to be  and some peripheral combinations of µ and λ were not tried either.

number of bins , 2

preprocessing and kernel parameters by trying a wide variety of values for each parameter, picking the combinations that produced the best predictions on the validation sets. For all kernels, the prediction quality as a function of the parameters was single peaked, and only that set of parameters was chosen per kernel for testing of the rest of the data. Having tuned the parameters, we then used the rest of data set 1 and the other three data sets to learn and predict the movement velocities. Again, we employed fivefold cross-validation on each data set to obtain accuracy results on all the examples. The five-fold cross-validation was produced by randomly splitting the trials into five groups: four of the five groups were used for training, and the rest of the data was used for evaluation. This process was repeated five times by using each fifth of the data once as a test set. The kernels that we tested are the Spikernel, the exponential kernel 2 K(S, T) = e−γ (S−T) , the homogeneous polynomial kernel K(S, T) = (S · T)d d = 2, 3, and standard scalar product kernel K(S, T) = S · T, which boils down to a linear regression. In our initial work, we also tested polynomial kernels of higher order and nonhomogeneous polynomial kernels, but as these kernels produced worse results than those we describe here, we did not pursue further testing. During parameter fitting (on the first half of data set 1), we tried all combinations of bin sizes {50ms, 100ms, 200ms} and number of bins {5, 10, 20} except for bin size and number of bins resulting in windows of length 2 s or more (since full information regarding the required movement was not available to the monkey 2S before the Go Signal and therefore neural activity at that time was probably not informative). The time interval between two consecutive examples was set to 100 ms. For each preprocessing, we then tried each kernel with parameter values described in Table 2. 6 Results Prediction samples of the Spikernel and linear regression are shown in Figure 3. The samples are of consecutive trials taken from data set 4 without

684

L. Shpigelman, Y. Singer, R. Paz, and E. Vaadia

v

y

1 0 −1 200

202

x

206

208

210

212

214

216

218

220

Go Signal / Target Reach Actual Spikernel Linear Regression

1

v

204

0 −1 200

202

204 206 208 210 212 214 216 cumulative prediction time from session start (sec)

218

220

Figure 3: Sample of actual movement, Spikernel prediction and linear regression prediction of left-hand velocity in data set 4 in a few trials: (top) vy , (bottom) vx (both normalized). Only intervals between Go Signal and Target Reach are shown (separated by vertical lines). Actual movement in thick gray, Spikernel in thick black, and linear regression in thin black. Time axis is seconds of accumulated prediction intervals from the beginning of the session.

prior inspection of their quality. We show the Spikernel and linear regression as prediction quality extremes. Both methods seem to underestimate the peak values and are somewhat noisy when there is no movement. The Spikernel prediction is better in terms of mean absolute error. It is also smoother than the linear regression. This may be due to the time warping quality of the features employed. We computed the correlation coefficient (r2 ) between the recorded and predicted velocities per fold for each kernel. The results are shown in Figure 4 as scatter plots. Each circle compares the Spikernel correlation coefficient scores to those of one of the other kernels. There are five folds for each of the four data sets and four correlation coefficients for each fold (one for each movement direction and hand), making up 80 scores (and circles in the plots) for each of the kernels compared with the Spikernel. Results above the diagonal are cases where the Spikernel outperformed. Note that though the correlation coefficient is informative and considered a standard performance indicator, the SVR algorithm minimizes the ε-insensitive loss. The results in terms of this loss are qualitatively the same (i.e., the kernels are ranked by this measure in the same manner), and we therefore omit them.

Spikernels

685

0.8

0.8

Spikernel − r

Spikernel − r

2

1

2

1

0.6

0.4

0.2

0

0

0.4

0.2

0.2 0.4 0.6 0.8 Exponential Kernel − r 2


0

1

0

0.2 0.4 0.6 0.8 Polynomial 2nd Deg. Kernel − r 2





1

0.8

0.8

1



Spikernel − r

2

1

2

Spikernel − r

0.6

0.6

0.4

0.2

0

0

0.6

0.4

0.2

0.2 0.4 0.6 0.8 Polynomial 3rd Deg. Kernel − r 2 

1

0

0

0.2 0.4 0.6 0.8 Linear Regression − r 2



1






Figure 4: Correlation coefficient comparisons of the Spikernel versus other kernels. Each scatter plot compares the Spikernel to one of the other kernels. Each circle shows the correlation coefficient values obtained by the Spikernel and by the other kernel in one fold of one of the data sets for one of the two axes of movement. (a) Compares the Spikernel to the exponential kernel. (b) Compares with the second-degree polynomial kernel. (c) Compares with the third-degree polynomial kernel. (d) Compares with linear regression. Results above the diagonals are cases where the Spikernel outperformed.

686

L. Shpigelman, Y. Singer, R. Paz, and E. Vaadia

Table 3: Chosen Preprocessing Values, Parameter Values, and Mean Correlation Coefficient Across Folds for Each Kernel in Each Data Set. Kernel

Kernel Parameters

Day 1

Day 2

Day 3

Day 4

Total

LhX RhX LhX RhX LhX RhX LhX RhX Mean LhY RhY LhY RhY LhY RhY LhY RhY

Type

Preprocessing

Spikernel

10 bins 100 ms

µ = .99 λ = .7 .71 .79 N = 5 C = 1 .74 .63

.77 .63 .87 .79

.75 .70 .67 .66

.69 .66 .79 .71

.72

Exponential

5 bins 200 ms

γ = .01 C = 5

.67 .76 .69 .45

.73 .57 .85 .74

.71 .67 .56 .61

.66 .63 .74 .65

.67

Polynomial second degree

10 bins 100 ms

C = 10−4

.67 .76 .70 .51

.72 .54 .85 .74

.70 .68 .61 .63

.65 .62 .76 .68

.68

Polynomial third degree

10 bins 100 ms

C = 10−5

.65 .72 .66 .51

.69 .52 .84 .71

.67 .68 .61 .62

.63 .61 .75 .64

.66

Linear (dot product)

10 bins 100 ms

C = 0.01

.54 .65 .67 .35

.62 .34 .75 .57

.66 .64 .50 .57

.51 .51 .70 .62

.57

Notes: Total means (across data sets, folds hands, and movement directions) are computed for each kernel. The Spikernel outperforms the rest, and the nonlinear kernels outperform the linear regression in all data sets.

Table 3 shows the best bin sizes, number of bins, and kernel parameters that were found on the first half of data set 1 and the mean correlation coefficients obtained with those parameters on the rest of the data. Some of the learning problems were easier than others (we observed larger differences between data sets than between folds of the same data set). Contrary to our preliminary findings (Shpigelman, Singer, Paz, & Vaadia, 2003), there was no significantly easier axis to learn. The mean scores define a ranking order on the kernels. The Spikernel produced the best mean results, followed by the polynomial second-degree kernel, the exponential kernel, the polynomial third-degree kernel, and finally, the linear regression (dot product kernel). To determine how consistent the ranking of the kernels is, we computed the difference in correlation coefficient scores between each pair of kernels in each fold and determined the 95% confidence interval of these differences. The Spikernel is significantly better than the other kernels and the linear regression (the mean difference is larger than the confidence interval). The other nonlinear kernels (exponential and polynomial second and third degree) are also significantly better than linear regression, but the differences between these kernels are not significant. For all kernels, windows of 1 s were chosen over windows of 0.5 s as best preprocessing (based on the parameter training data). However, within the 1 s window, different bin sizes were optimal for the different kernels. Specifically, for the exponential kernel, 5 bins of size 200 ms, were best and for the rest, 10 bins of 100 ms were best. Any processing of the data (such as time binning or PCA) can only cause loss of information (Cover & Thomas, 1991)

Spikernels

687

Table 4: Mean Correlation Coefficients (Over All Data) for the Spikernel with Different Combinations of Bin Sizes and Number of Bins. Number of Bins

50 ms

100 ms

200 ms

5 bins 10 bins 20 bins

.52 .66 .66

.65 .72

.68

Note: The best fit to the data is still 10 bins of 100 ms.

but may aid an algorithm. The question of what the time resolution of cells is is not answered here, but for the purpose of linear regression and SVR (with the kernels tested) on our MI data, it seems that binning was necessary. In many cases, a poststimulus time histogram, averaged over many trials, is used to get an approximation of firing rate that is accurate in both value and position in time (assuming that one can replicate the same experimental condition many times). In single-trial reconstruction, this approach cannot be used, and the trade-off between time precision versus accurate approximation of rate must be balanced. To better understand what the right bin size is, we retested the Spikernel with bin sizes of 50 ms, 100 ms, and 200 ms and number of bins 5, 10, and 20, leaving out combinations resulting in windows of 2 seconds or more. The mean correlation coefficients (over all the data) are shown in Table 4. The 10 by 100 ms configuration is optimal for the whole data as well as for the parameter fitting step. Note that since the average spike count per second was less then 10, the average spike count in bins of 50 ms to 200 ms bins was 0.5 to 2 spikes. In this respect, the data supplied to the algorithms are on the border between a rate and a spike train (with crude resolution). Run-time issues were not the focus of this study, and little effort was made to make the code implementation as efficient as possible. Run time for the exponential and polynomial kernels (prediction time) was close to real time (the time it took for the monkey to make the movements) on a Pentium 4, 2.4 GHz CPU. Run time for the Spikernel is approximately 100 times longer than the exponential kernel. Thus, real time use is currently impossible. The main contribution of the Spikernel is the allowance of time warping. To better understand the effect of λ on Spikernel prediction quality, we retested the Spikernel on all of the data, using 10 bins of size 100 ms (n = 5, µ = 0.99) and λ ∈ {0.5, 0.7, 0.9}. The mean correlation coefficients were {0.68, 0.72, 0.66}, respectively. We also retested the exponential kernel with γ = 0.005 on all the data with 5 bins of size 100 ms. This would correspond to a Spikernel with its previous parameters (µ = 0.99, n = 5) but no time warping. The correlation coefficient achieved was 0.64. Again, the relevance of time warping to neural activity cannot be directly addressed by our method. However, we can safely conclude that consideration of time-

688

L. Shpigelman, Y. Singer, R. Paz, and E. Vaadia

warped neural activity (in the Spikernel sense) does improve prediction. We believe that it would make sense to assume that this is relevant to cells as well, as mentioned in section 4.1, because the 3D structure of a cell’s dendritic tree can align integration of differently placed inputs from different times. 7 Conclusion In this article, we described an approach based on recent advances in kernelbased learning for predicting response variables from neural activities. On the data we collected, all the kernels we devised outperform the standard scalar product that is used in linear regression. Furthermore, the Spikernel, a biologically motivated kernel, operator consistently outperforms the other kernels. The main contribution of this kernel is the allowance of time warping when comparing two sequences of population spike counts. Our current research is focused in two directions. First, we are investigating the adaptations of the Spikernel to other neural activities such as local field potentials and the development of kernels for spike trains. Our second goal is to devise statistical learning algorithms that use the Spikernel as part of a dynamical system that may incorporate biofeedback. We believe that such extensions are important and necessary steps toward operational neural prostheses. Acknowledgments We thank the anonymous reviewers for their constructive criticism. This study was partly supported by a Center of Excellence grant (8006/00) administrated by the ISF, BMBF-DIP, and by the United States–Israel Binational Science Foundation. L.S. is supported by a Horowitz fellowship. References Brockwell, A. E., Rojas, A. L., & Kass, R. E. (2004). Recursive Bayesian decoding of motor cortical signals by particle filtering. Journal of Neurophysiology, 91, 1899–1907. Brown, E. N., Frank, L. M., Tang, D., Quirk, M. C., & Wilson, M. A. (1998). A statistical paradigm for neural spike train decoding applied to position prediction from ensemble firing patterns of rat hippocampal place cells. Journal of Neuroscience, 18, 7411–7425. Chapin, J. K., Moxon, K. A., Markowitz, R. S., & Nicolelis, M. A. (1999). Realtime control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neuroscience, 2, 664–670. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. Wiley.

Spikernels

689

Donchin, O., Gribova, A., Steinberg, O., Bergman, H., & Vaadia, E. (1998). Primary motor cortex is involved in bimanual coordination. Nature, 395, 274– 278. Fu, Q. G., Flament, D., Coltz, J. D., & Ebner, T. J. (1997). Relationship of cerebellar Purkinje cell simple spike discharge to movement kinematics in the monkey. Journal of Neurophysiology, 78, 478–491. Genton, M. G. (2001). Classes of kernels for machine learning: A statistical perspective. Journal of Machine Learning Research, 2, 299–312. Georgopoulos, A. P., Kalaska, J., & Massey, J. (1983). Spatial coding of movements: A hypothesis concerning the coding of movement direction by motor cortical populations. Experimental Brain Research (Supp.), 7, 327–336. Georgopoulos, A. P., Kettner, R. E., & Schwartz, A. B. (1988). Primate motor cortex and free arm movements to visual targets in three-dimensional space. Journal of Neuroscience, 8, 2913–2947. Georgopoulos, A. P., Schwartz, A. B., & Kettner, R. E. (1986). Neuronal population coding of movement direction. Science, 233, 1416–1419. Haussler, D. (1999). Convolution kernels on discrete structures (Tech. Rep. No. UCSC-CRL-99-10). Santa Cruz, CA: Uniiversity of California. Isaacs, R. E., Weber, D. J., & Schwartz, A. B. (2000). Work toward real-time control of a cortical neural prosthesis. IEEE Trans. Rehabil. Eng., 8, 196–198. Jaakola, T. S., & Haussler, D. (1998). Exploiting generative models in discriminative calssifiers. In M. Kearns, M. Jordan, & S. Solla (Eds.), Advances in neural information processing systems. Cambridge, MA: MIT Press. Laubach, M., Wessberg, J., & Nicolelis, M. A. (2000). Cortical ensemble activity increasingly predicts behavior outcomes during learning of a motor task. Nature, 405(1), 141–154. Laubach, M., Shuler, M., & Nicolelis, M. A. (1999). Independent component analyses for quantifying neuronal ensemble interactions. J. Neurosci Methods, 94, 141–154. Lodhi, H., Shawe-Taylor, J., Cristianini, N., & Watkins, C. J. C. H. (2000). Text classification using string kernels. In S. A. Solla, T. K. Leen, & K.-R. Muller ¨ (Eds.), Advances in nueural information processing systems. Cambridge, MA: MIT Press. Magee, J. (2000). Dendritic integration of excitatory synaptic input. Nat. Rev. Neuroscience, 1, 181–190. Moran, D. W., & Schwartz, A. B. (1999). Motor cortical representation of speed and direction during reaching. Journal of Neurophysiology, 82, 2676–2692. Nicolelis, M. A. (2001). Actions from thoughts. Nature, 409(18), 403–407. Nicolelis, M. A., Ghazanfar, A. A., Faggin, B. M., Votaw, S., & Oliveira, L. M. (1997). Reconstructing the engram: Simultaneous, multi-site, many single neuron recordings. Neuron, 18, 529–537. Paz, R., Boraud, T., Natan, C., Bergman, H., & Vaadia, E. (2003). Preparatory activity in motor cortex reflects learning of local visuomotor skills. Nature Neuroscience, 6(8), 882–890. Pouget, A., & Snyder, L. (2000). Computational approaches to sensorimotor transformations. Nature Neuroscience, 8, 1192–1198.

690

L. Shpigelman, Y. Singer, R. Paz, and E. Vaadia

Reihle, A., Grun, S., Diesmann, M., & Aersten, A. M. H. J. (1997). Spike synchronization and rate modulation differentially involved in motor cortical function. Science, 278, 1950–1952. Schwartz, A. B. (1994). Direct cortical representation of drawing. Science, 265, 540–542. Segev, I., & Rall, W. (1998). Excitable dendrites and spines: Earlier theoretical insights elucidate recent direct observations. TINS, 21, 453–459. Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R., & Donoghue, J. P. (2002). Instant neural control of a movement signal. Nature, 416, 141–142. Shpigelman, L., Singer, Y., Paz, R., & Vaadia, E. (2003). Spikernels: Embedding spiking neurons in inner product spaces. In S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in neural information processing systems, 15. Cambridge, MA: MIT Press. Smola, A., & Scholkopf, ¨ B. (2004). A tutorial on support vector regression. Statistics of Computing, 14, 199-222. Taylor, D. M., Tillery, S. I. H. S. I., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science, 1829–1832. Vaadia, E., Haalman, I., Abeles, M., Bergman, H., Prut, Y., Slovin, H., & Aertsen, A. (1995). Dynamics of neuronal interactions in monkey cortex in relation to behavioral events. Nature, 373, 515–518. Vapnik, V. (1995). The nature of statistical learning theory. New York: Springer. Watkins, C. (1999). Dynamic alignment kernels. In P. Bartlett, B. Scholkopf, ¨ D. Schuurmans, & A. Smola (Eds.), Advances in large main classifiers (pp. 39– 50). Cambridge, MA: MIT Press. Wessberg, J., Stambaugh, C. R., Kralik, J. D., Beck, P. D., Laubach, M., Chapin, J. K., Kim, J., Biggs, J., Srinivasan, M. A., & Nicolelis, M. A. (2000). Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature, 408(16), 361–365. Received August 8, 2003; accepted August 5, 2004.

LETTER

Communicated by Ad Aertsen

Memory Capacity of Balanced Networks Yuval Aviel [email protected] Interdisciplinary Center for Neural Computation, Hebrew University, Jerusalem, Israel

David Horn [email protected] School of Physics and Astronomy, Tel Aviv University, Tel Aviv, Israel

Moshe Abeles [email protected] Interdisciplinary Center for Neural Computation, Hebrew University, Jerusalem, Israel

We study the problem of memory capacity in balanced networks of spiking neurons. Associative memories are represented by either synfire chains (SFC) or Hebbian cell assemblies (HCA). Both can be embedded in these balanced networks by a proper choice of the architecture of the network. The size wE of a pool in an SFC or of an HCA is limited from below √ and from above by dynamical considerations. Proper scaling of wE by K, where K is the total excitatory synaptic connectivity, allows us to obtain a uniform description of our system for any given K. Using combinatorial arguments, we derive an upper limit on memory capacity. The capacity allowed by the dynamics of the system, αc , is measured by simulations. For HCA, we obtain αc of order 0.1, and for SFC, we find values of order 0.065. The capacity can be improved by introducing shadow patterns, inhibitory cell assemblies that are fed by the excitatory assemblies in both memory models. This leads to a doubly balanced network, where, in addition to the usual global balancing of excitation and inhibition, there exists specific balance between the effects of both types of assemblies on the background activity of the network. For each of the memory models and for each network architecture, we obtain an allowed region (phase √ space) for wE / K in which the model is viable. 1 Introduction An interesting property of neural networks is their ability to function as memory devices. In this mode, memories are embedded in the synaptic connections of the network, to be recalled later. Recalling is typically done by external ignition of part of the desired memory. The system, using the Neural Computation 17, 691–713 (2005)

c 2005 Massachusetts Institute of Technology 

692

Y. Aviel, D. Horn, and M. Abeles

given hint, should then settle on an associated memory; hence, it is called an associative memory model. Reading out the network’s activity, it is possible to decide if the input memory exists, and if it does, then what its constituents are. One method of embedding memories is that of attractor neural networks (Amit, 1989), where an initial input may lead to dynamical flow into an attractor, which serves as a Hebbian cell assembly (HCA) (Hebb, 1949) representing the recalled memory through elevated firing rates of its neurons. Another method is that of storing memories in spatiotemporal patterns of activity, such as synfire chains (SFC) (Abeles, 1991). The memories are encoded in the precise firing patterns produced by the given external input. As this method involves fine temporal structure, it is capable of dynamic binding (Bienenstock, 1995). Applying binding (von der Malsburg & Schneider, 1986) to HCAs, one has to resort to other temporal means (Horn, Sagi, & Usher, 1991). A network memory model should allow a neuron to participate in more than one memory, that is, have memories stored in a distributed manner. In this case, a neuron may receive input also when it is not supposed to fire. This input is due to the overlap between memories and should be treated as noise. Moreover, background activity in the absence of any memory retrieval should also be allowed and may be regarded as being due to some other noise source. One should keep in mind that the statistics of an active cortical tissue is one of high variability. An irregular spike train is observed on the individual neuron level, together with an asynchronous activity on the global level (Shadlen & Newsome, 1994). Among the sources of that variability are the 50% inputs from other cortical areas (Braitenberg & Schuz, 1991) and probabilistic synaptic release (Huang & Stevens, 1997). A network model capable of displaying stationary noisy activity of this kind is the balanced network (BN) model. A network is said to be balanced (Shadlen & Newsome, 1994; van Vreeswijk & Sompolinsky, 1998) if each neuron in the network receives equal amounts of excitation and inhibition. Its membrane potential will then fluctuate around some mean value; the firing process is noise driven and therefore irregular (Abeles, 1982; Gerstein & Mandelbrot, 1964). BNs have been shown (Brunel, 2000) to mimic the in vivo firing statistics of cortical tissue, and it is therefore plausible that cortical neurons receive balanced input. BNs have also been shown (Brunel, 2000; van Vreeswijk & Sompolinsky, 1998) to have a stable asynchronous state (AS). Generally, BNs assume sparse and random connectivity. It is possible to embed memories in the connections of a BN, but this would violate the random connectivity assumption. The main consequence of introducing ordered connectivity in an otherwise random connectivity matrix is the appearance of a new critical point beyond which the AS is unstable. This was studied in detail in Aviel, Mehring, Horn, and Abeles (2003).

Memory Capacity of Balanced Networks

693

Having set the background, we will now turn to our memory models of choice. A milestone in the theory of attractor neural network is the Hopfield model (Hopfield, 1982), capable of storing and retrieving binary vectors by a clever construction of the connectivity matrix of the neural network. The models of Hopfield, as well as some predecessors (Willshaw, Buneman, & Longuet-Higgins, 1969) and followers (Tsodyks & Feigelman, 1988), are, however, based on binary rather than spiking neurons (but see Treves, 1990, for a Hopfield model of linear threshold neurons). All attractor neural networks serve as some kind of implementation of the idea of Hebbian cell assemblies (HCA) (Hebb, 1949), that is, they contain neuronal ensembles that represent a memory by elevated firing rates. The cell assembly may be characterized by denser intra-assembly connectivity that facilitates a sustained activity on partial assembly excitation (Braitenberg, 1978). The Hopfield model is a step backward from the biological complexity of HCAs, but it allows detailed analysis. Gerstner (Gerstner & van Hemmen, 1992) showed how the Hopfield model may be incorporated in a network of spiking neurons. A network that reacts to external inputs with reproducible firing patterns (over neurons and time) is said to code its input by spatiotemporal patterns. The model we use for producing spatiotemporal patterns is the synfire chain (SFC) (Abeles, 1982). (For other interesting models, see Izhikevich, Gally, & Edelman, 2004; Levy, Horn, Meilijson, & Ruppin, 2001; and Miller, 1996.) The SFC dictates a well-defined connectivity pattern among neurons in the form of feedforward connections between pools of neurons. Each of the wE neurons in a pool receives L connections from neurons in the previous pool, thus creating a chain of pools. Other additional input connections as well as outputs are allowed. If L is large enough, then a synchronized firing volley of most of the neurons in a pool may form a wave of activity that propagates along the chain (Diesmann, Gewaltig, & Aertsen, 1999). Also if L is large enough and if the igniting volley is synchronized and strong enough, the waves are stable in the presence of background noise (Diesmann et al., 1999). To avoid terminological confusion, the feedforward connectivity schemes are referred to henceforth as chains, and the synchronized volley propagating along a chain as a synfire wave, or simply a wave. A wave can propagate in a synchronized manner along a chain, or it can lose its synchrony, dissolving into the background activity. A wave is said to be stable if it remains as a synchronized volley for more than 100 ms. Having introduced all the ingredients—asynchronous activity at the global level, attractors or spatiotemporal patterns at the assemblies’ level, and irregular spiking at the neuronal level—we may state our mission: we seek a balanced network of spiking neurons that can serve as a high-capacity memory device. To establish the goal, we use BNs of integrate-and-fire (IF)

694

Y. Aviel, D. Horn, and M. Abeles

neurons, similar to the model described in Brunel (2000). As memory models, we use either a HCA or SFC. HCAs embedded in a recurrent network were suggested as a model (Amit & Brunel, 1997) of the persistent activity observed in delayed matchto-sample experiments performed in the inferotemporal and prefrontal cortex area (Miyashita & Chang, 1988). Within this model, the network, in the absence of memory stimulation, exhibits sustained asynchronous activity. After learning, an elevated activity within an HCA is obtained if a familiar stimulus is presented briefly. Wang (1999) found that in order to realize a stable, low-rate, persistent activity coexisting with a stable resting state, recurrent excitation should be primarily mediated by kinetically slow synapses of the NMDA type. Later, Compte, Brunel, Goldman-Rakic, and Wang (2000) examined the synaptic mechanisms of selective persistent activity underlying spatial working memory in the prefrontal cortex. Their model reproduces the phenomenology of the occulomotor delayed-response experiment of Funahashi, Bruce, and Goldman-Rakic (1989). Brunel and Wang (2001) studied the effects of external input and neuromodulation on persistent activity in a working memory model. (Additional reviews on HCA can be found in Sommer and Wennekers, 2000, 2001.) In Aviel, Mehring et al. (2003) we studied the embedding of SFCs in a BN of IF neurons. The main obstacle that we found is the conflict between constraints due to embedded memories, on one hand, and the AS stability, on the other hand. The conflicting demands are not easily resolved. This is in agreement with the study of Mehring, Hehl, Kubo, Diesmann, and Aertsen (2003), where a similar setup was used with added topological, cortex-like connectivity. The authors report a fairly small parameter regime where SFC can be embedded and recalled without destabilizing the AS. None of these publications paid special attention to the question of high capacity. The focus of this article is capacity. We load the system with memory patterns and look for conditions under which we can (1) maximize the load, (2) keep the asynchronous state stable, and (3) recall every memory in a stable manner. Obtaining maximal capacity is, of course, a desirable goal. Many articles have dealt with this kind of question in networks of binary neurons. While far from biological reality, binary neurons lend themselves to analytical examination. The Hopfield model, for example, is useful because it can be analyzed theoretically. In particular, its behavior with respect to memory load is analyzed in Amit, Gutfreund, and Sompolinsky (1985), where it is shown that the maximal number of uncorrelated memory patterns, Pmax , is linear in N, the number of neurons: Pmax = αc N, with αc = 0.14. Applying results obtained in a binary neural network to neural networks of spiking neurons is not straightforward. Spiking neurons involve membrane time constants and nonlinear resetting, which lead to much richer system dynamics. Little is known on the capacity of neural networks of spiking neurons.

Memory Capacity of Balanced Networks

695

Sommers and Wennekers (2001) obtained a high-capacity limit of HCA in a symmetrically coupled network of 100 Pinsky-Rinzel neurons. An inhibitory loop provided negative feedback leading to dynamic threshold control that can improve capacity. Their network operated in the oscillatory regime and obtained high capacity. Here we limit ourselves to the asynchronous regime within a BN, which introduces other types of constraints. The capacity of SFCs has also been investigated. Bienenstock (1995) used an r-winners-take-all model and applied to it signal-to-noise analysis. Hermann, Hertz, and Prugel-Bennett (1995) and Hertz (1999) reduced the IF model to a binary model so statistical mechanics methods can be used. Both arrived at similar conclusions; again Pmax = αc N, but now αc ∼ = 8. Here Pmax is the maximal number of pools. In section 2, our model is presented in detail, in section 3 the scaling of the system is discussed, in section 4 the results are reported, and in section 5 we discuss the results. 2 The Model We employ three levels of modeling in our system. On the microscopic level, a neuronal model is required, for which we use an IF model. On the intermediate level, we model memory patterns through synfire chains and by Hebbian cell assemblies. On the macroscopic level, we use a balanced network and discuss its modification into a doubly balanced network. For simulations we used the SYNOD environment (Diesmann, Gewaltig, & Aertsen, 1995) with the Paranel kernel (Morrison, Mehrin, Geisel, Aertsen, & Diesmann, 2004). The neuronal model was integrated with time steps of 0.1 ms. 2.1 Single Neuron Model. Following Lapique (Tuckwell, 1988), we use an IF model in which the ith neuron’s membrane potential, Vi (t), obeys the equation: τ

dVi (t) = −Vi (t) + RIi (t), dt

(2.1)

where Ii (t) is the synaptic current arriving at the soma and R is the membrane resistance. Spikes as well as postsynaptic currents are modeled by delta functions; hence, the input is written as RIi (t) =



j

f

Jij δ(t − tj − τdelay ),

(2.2)

f tj

where the first sum is over different neurons, and the second sum represents f f their spikes arriving at times t = tj − τdelay . tj is the emission time of the f th spike by neuron j, and τdelay is a transmission delay, which we assume

696

Y. Aviel, D. Horn, and M. Abeles

here to be the same for any pair of neurons. The sum is over all neurons that project their output to neuron i, both local and external afferents. The strength of the synapse that neuron j forms on neuron i is Jij . When Vi (t) reaches the firing threshold θ, an action potential is emitted by neuron i, and after a refractory period τrp , during which the potential is insensitive to stimulation, the depolarization is reset to Vreset . The following parameters were used in all simulations: the transmission delay τdelay = 1.5 ms, the threshold θ = 20 mV, the membrane time constant τ = 10 ms, the refractory period τrp = 2.5 ms, the resetting potential Vreset = 0 mV, and the membrane resistance R = 40 M . The inhibitory and excitatory neurons have identical parameters.

2.2 Memory Model. Two memory models are explored, constructed on the basis of either the HCA or SFC. HCAs form an associative memory model in a biologically plausible network. An assembly is a group of wE randomly chosen excitatory neurons. The assembly is distinguished by a dense interassembly connectivity. A neuron in an assembly receives L connections from other neurons in the assembly. The dense connectivity allows for a sustained high firing rate once an assembly is ignited. A neuron will typically participate in more than one assembly. In that case, it will have high connectivity with all these assemblies. If two neurons happen to participate together in two assemblies, we assign two different synapses to the connectivity induced by the two memories. All excitatory (inhibitory) synaptic connections are assumed to be of equal strength, J (JI ), and a stronger bond between a pair of neurons is brought about by multiple synapses. Each neuron will be assumed to have a fixed synaptic resource, made of K excitatory (inhibitory) “synaptic units” J (JI ). This constraint on synaptic resources will impose an upper bound on the number of memory patterns one can embed in the network. SFCs (Abeles, 1991) are feedforward connections among neuronal pools, with a fixed number of excitatory neurons, wE , in each pool. wE is also referred to as the chain width. To wire an SFC, we randomly pick pools of wE neurons and connect them in a feedforward manner, thus obtaining a chain of pools. Converging and diverging connections form a link between two consecutive pools. Similar to HCAs, a neuron will typically participate in more than one pool. In contrast to HCAs, each neuron in a pool receives Lspecific connections from the previous pool, not from the pool to which it belongs. If wE is large enough, then a synchronized firing volley of most of the neurons in a pool may propagate along the chain, forming a synfire wave, a stable wave of activity (Diesmann et al., 1999). The wave signals an activity of a specific memory object. The wave can be (de-) synchronized with other waves, allowing for a mechanism of dynamically (un-) binding objects (Hayon, 2002).

Memory Capacity of Balanced Networks

697

In these two memory models, we have three parameters: wE , the size of the memory pattern (either an assembly or a pool); L, the interpattern connectivity; and P, the number of memories (assemblies or pools). Ignition of a memory pattern is performed by increasing the external input to all members of the pattern during 5 ms. An HCA pattern will be said to be stable if, after ignition, it manifests a sustained elevated firing rate for at least 100 ms. A synfire wave will be said to be stable if a wave propagates along the chain for at least 100 ms. Both patterns have a minimal value of wE , wmin , below which patterns are unstable. The minimum is due to the condition that the firing of the assembly or pool guarantees the continuing firing of the same assembly or the next pool. In case of SFCs (Aviel, Mehring et al., 2003; Tetzlaff, Buschermohle, ¨ Geisel, & Diesmann, 2003), w has also a maximal value, above which spontaneous emergence of waves occur. This is also likely to be the case for HCAs. Once wiring of all memory patterns is done, more connections are added such that if an excitatory neuron has fewer than K excitatory (KI inhibitory) synapses, random sources are chosen from the network until it reaches exactly K excitatory (KI inhibitory) synapses. The resulting connectivity is a mixture of random connectivity and ordered connectivity. 2.3 Network Model. The excitatory population consists of NE excitatory neurons, and the inhibitory one consists of NI ≡ γ NE inhibitory neurons. A sparse connectivity is required to adhere as closely as possible to biological values and to induce a source of randomness. In addition to the external input (K excitatory afferents), each neuron in the network receives exactly K excitatory and KI = γ K inhibitory afferents from the excitatory and inhibitory populations, respectively. In our simulations, we use K = εNE , with ε = 0.1. Our formalism allows other relations as well, for example, K = Const when NE grows. (This will be elaborated in section 5.) If a neuron of population y (either E or I) innervates a neuron of population x (either E or I), its synaptic strength Jxy is defined as follows: √ J ≡ JxE = J0 / K,

 JI ≡ JxI = −gJ0 / KI ,

(2.3)

√ √ where J0 is a constant. Note that JI = −g/ γ · J; hence, the constant g/ γ is the relative strength of the inhibitory synapses. A Poisson process with rate vext K simulates the external input. vext ≡ v · vthre , where vthre is the minimal rate needed to emit a spike within τ milliseconds (on the average) in a neuron that gets balanced input (Brunel, 2000). The network parameters used in this article are γ = 1/4, ε = 0.1, g = 5, J0 = 10, and v = 1/20. The total input to a neuron is therefore nearly balanced. There is a small bias toward an excess of excitation, which controls the firing rates.

698

Y. Aviel, D. Horn, and M. Abeles

3 Scaling Scaling is a crucial issue in modeling complex systems. It tells us how parameters should be modified (scaled) while changing the number of elements (or the size) of a system. In our case, the system is the network with its connectivity matrix, the external input, and the single-neuron dynamics. The number of elements is the number of neurons or synapses in the network. Complex systems often show complex behavior that depends on the parameters of the system. For proper scaling, the behavior may only weakly depend on the size of the system. In this regard, it makes sense to consider the thermodynamic limit, where the number of elements goes to infinity. In this section, we propose a scaling scheme that allows us to capture the essential behavior of the model for all values of K. We start by scaling the synaptic weight, J. The proportionality factor of the synaptic weight is typically chosen in model calculations as a function of the number of synapses, K, J = J0 /Kβ ,

(3.1)

where β gets values 0, 0.5, or 1. The mean field approach using β = 1 leads to weak coupling in the high K regime. Examples of such studies are the Hopfield model (Hopfield, 1982) and phase-variable analysis (Hansel, Mato, & Meunier, 1995). Constant synaptic strength, β = 0, was also used in the literature. Brunel (2000), for example, used fixed synapses together with the assumption that the constant postsynaptic potential (PSP) contribution is small relative to the distance between threshold and resting potential. Also in this regime, the synaptic coupling is weak, and one can model the barrage of incoming PSPs as gaussian noise. Since the contribution of each PSP is small, the overall change in the membrane potential is smooth, and diffusion approximation can be used. Following van Vreeswijk and Sompolinsky (1998), we propose using β = 0.5. The reason underlying this choice comes from requiring the firing rate of each neuron to be independent of the number of synapses, K. The same logic applies to the inhibitory synaptic strength JI . Choosing β = 0.5, we obtain equation 2.3. Our system operates under balanced conditions; the mean input to a neuron is near zero, and fluctuations drive the spiking process. Let hx be the field generated by synapses of population x (x = E or I) felt by some neuron, and let us assume that the system is in an asynchronous state (AS). Under these conditions, all synaptic inputs can be modeled by a Poisson process with rate v. To simplify calculations, we assume that both populations fire at the same rate v. Using equation 2.3, the mean and variance can be described as follows: √ µ ≡ hE − hI = v · (JK − JI KI ) = J · v · K(1 − g γ )

Memory Capacity of Balanced Networks

= J0 · v ·

√

699

√ K(1 − g γ )

(3.2)

σ 2 = var(hE ) + var(hI ) = v · (J2 K + JI2 KI ) = J2 · v · K(1 + g2 ) = Jo2 v(1 + g2 ).

(3.3)

√ From equation 3.2, we see that g γ controls the balance between the two √ populations. The mean of the field is zero if g γ = 1, and therefore independent of K. But also the variance is independent of K. This would √ not have been the case for β = 0 or 1. On average, therefore, an input of K excitatory PSPs produces a spike. The minimum rate of external input that is needed in order to emit a spike within τ ms (on average) in a neuron that √ does not get other inputs (Brunel, 2000) is defined to be vthre ≡ θ/(τ · J · K) = θ/(τ · J0 ). Again, vthre is independent of K. √ In our simulations, g > 1/ γ , so there is a potential excess of inhibition, but there is also an additional excitatory external input. Even if the total input is not exactly √ balanced, the feedback between the population, which is on the order of K according to equation 3.2, leads to an appropriate change of the firing rates that stabilize the AS (van Vreeswijk & Sompolinsky, 1998). Another hint for setting β to 0.5 comes from our previous study (Aviel, Mehring √ et al., 2003), where we demonstrated that wE has to obey wE = Const· K. In this case, an input from wE synapses, JwE , should lead to firing with √ high probability. It means that JwE ∼ = O(1), that is, J is proportional to 1/ K. √ As a consequence, we set √ the pattern size wE = Cw K, and the intrapattern connectivity L = CL K, where Cw and CL are the scaling prefactors. The scaling of the external rate and √ the synaptic weights is given by vext ≡ v · vthre = v · θ/(τ · J0 ) and J ≡ J0 / K respectively, as discussed in section 2. We use the values in Table 1. Table 1: Parameter Values. Parameter

J0 Cw CL v

Values HCA

SFC

10 3.3 0.75Cw 0.05

10 4 Cw 0.05

The usefulness of the scaling of pattern connectivity and external rate will become evident from the discussion that follows. In the next section we discuss properties of the resulting balanced network.

700

Y. Aviel, D. Horn, and M. Abeles

4 Results 4.1 Balanced Network. In Aviel, Mehring et al. (2003), we studied the embedding of SFCs in the synaptic connectivity matrix of a balanced network. There, as we do here, we enforced two constraints: the AS has to be the stable background mode of the system, and the synfire wave has to propagate on top of it in a stable manner. Based on simplified models and simulations, we concluded that these two constraints could be met if K is large enough (K > Const · w2min ). In this article, we take the network load α ≡ P/NE into consideration, using synaptic scaling. This allows further refinement of our statements. Brunel’s network (2000) serves here as the starting point, in which we embed HCA or SFC patterns. Apart from introducing ordered patterns in its connectivity matrix, we also use a particular scaling procedure. Synaptic √ weights, instead of being constants, are now scaled like 1/ K (van Vreeswijk & Sompolinsky, 1998). This in turn leads to vthre = θ/(τ · J0 ) = Const. We verified by simulations that indeed the mean firing rate (in the AS) is linearly related to vext and is only weakly dependent on NE . The analysis of the appendix in Aviel, Mehring et al. (2003) supports square root scaling. We have shown there that wE is limited by two constraints: wmin < wE < wmax ,

(4.1)

√ where wmax ≡ Cb K. The upper bound is due to requiring stability of the AS, and the lower bound is posed by wave stability demand. We can estimate wmin as the mean number of excitatory PSPs needed to evoke an action potential with high probability. Assuming a normal distribution of the membrane potential, wmin can be approximated by twice the distance between threshold and V in units of J, where V is the mean membrane potential and J is the synaptic unit: wmin = 2

θ − V

. J

(4.2)

Substituting equation 4.2 in equation 4.1, √ we notice that wE is sandwiched between an upper bound that is of order K and a lower bound that is of order Kβ . Unless β ≤ 0.5, wE has no valid √ value in the limit of large K. By choosing β = 0.5 and wmin ≡ Ca K accordingly, equation 4.1 can be rewritten as Ca < Cw < Cb .

(4.3)

Under these conditions, we expect the SFC to be stable in our system regardless of the number of synapses K.

Memory Capacity of Balanced Networks

701

In Figure 1a, we can see a wave propagating along a chain of 750 pools in a BN of 15,000 excitatory neurons. If the chain width exceeds a critical value or if the network load exceeds capacity, global oscillations appear. In Figure 1b, we increase the number of pools embedded in the network. This leads to the emergence of global oscillations after wave ignition. We also embedded HCAs according to the protocol described in section 2. In a lightly loaded system, as in Figure 2a, an evoked assembly sustained its activity for hundreds of milliseconds without provoking strong global oscillations. As in the SFC case, Figure 2b shows that the global oscillations become prominent as the load increases. A surprisingly high capacity is obtained in both cases. In order to quantify the emergence of global oscillations, the values of the population rate, that is, number of spikes of a population in 1 ms, are considered. An average and standard deviation (SD) of these values are taken over a time window of 300 ms in order to compute a population’s coefficient of variance (CV). The CV is the SD divided by the mean, and in our case it measures the amount of synchrony of the population activity. CV near one means an SD that is close to the mean, which is in agreement with an AS. High CV is a result of a high SD, which implies global synchronous (aperiodic) oscillations in our model. Low CV (< 0.5) signifies high firing rates. Low CV will become apparent in the next section, where the mild oscillations in the background activity during wave propagation will be addressed. In Figure 3, the CV of the population rate is plotted as a function of network load, P/NE , for various network sizes. We present statistics for two cases: pre- and postpattern ignition. In the former, the time window is taken prior to pattern ignition (time 200–500 ms), and in the latter, the time window is taken postignition (500–800 ms). All curves exhibit transitions near critical points. We define the critical point of the network load by Pc , and we define the capacity of the network as αc ≡ Pc /NE . The capacity obtained here is 0.1 and 0.065 for HCA and SFC, respectively. One should note an important difference between HCAs and SFCs. Whereas when HCAs are embedded, the pre- and postignition curves are indistinguishable, the SFCs shows a significantly higher network CV after their ignition. 4.2 Doubly Balanced Network. In the background activity of Figure 1, oscillation can be seen during wave propagation. In this section, we provide a mechanism that gets rid of this problem. The solution we arrived at involves architectural modifications to the original SFC structure. Let us attach a pool of randomly chosen inhibitory neurons, a shadow pool, to each excitatory pool in a synfire chain. A neuron in an excitatory pool projects its output not only to the next pool in the chain, but also to all neurons in its shadow pool. A neuron in the shadow pool does not project its output in any ordered manner, but diffuses its output randomly to the rest

702

Y. Aviel, D. Horn, and M. Abeles

A

K=1500 P=750 w=136 d=0.0

Neuron number

2500 2000 1500 1000 500 0 200

300

400

500

600

Time [ms]

700

800

B

K=1500 P=1050 w=136 d=0.0

Neuron number

2500 2000 1500 1000 500 0 200

300

400

500

600

Time [ms]

700

800

Figure 1: Raster plots of an SFC embedded in a BN with K = 1500. (a) α = 0.05, (b) α = 0.07. Neurons that participate in more than one pool may appear more than once on the raster plot, whose y-axis is ordered according to pools and represents every second neuron in each pool. The first 40 pools of the SFC are presented on the y-axis.

Memory Capacity of Balanced Networks

703

A

K=1500 P=750 w=128 d=0.0 2500

Neuron number

2000 1500 1000 500 0 200

300

400

500

600

Time [ms]

700

800

B

K=1500 P=1575 w=128 d=0.0 2500

Neuron number

2000 1500 1000 500 0 200

300

400

500

600

Time [ms]

700

800

Figure 2: Raster plots of HCAs embedded in a BN with K = 1500. (a) α = 0.05. (b) α = 0.105. The twelfth pattern is externally ignited at time t = 500 ms for 5 ms. Neurons that participate in more than one HCA may appear more than once on the raster plot, whose y-axis is ordered according to HCAs, and represents every second neuron in each pattern. Forty HCAs are presented on the y-axis.

704

Y. Aviel, D. Horn, and M. Abeles

A 4

3.5

Population CV

3

2.5

2

1.5

1

0.5 0.05

0.06

0.07

0.08

0.09

0.1

0.11

0.12

0.13

P/NE

B 4

3.5

Population CV

3

2.5

2

1.5

1

0.5 0.05

0.055

0.06

0.065

0.07

0.075

0.08

P/N

E

Figure 3: BN population CV as a function of α for various values of K. (a) HCAs are embedded. (b) An SFC is embedded. Post- and preignition curves are indicated by o and + markers, respectively. In both cases, statistics were gathered from the 300 ms pre- and postignition. Curves for three values of K (500, dotted curve; 1000, dashed curve; and 1500, solid line) are superimposed. The transitions to the oscillatory mode occur near the same α value, αc (0.1 for HCAs, 0.07 for SFCs preignition, and 0.65 for SFC postignition) for all K values.

Memory Capacity of Balanced Networks

705

Pool

Excitatory chain Inhibitory (shadow) chain

Figure 4: A modified synfire chain. Excitatory neurons (open circles) construct the traditional excitatory chain, characterized by the full feedforward connections. Inhibitory neurons (filled circles) form the shadow pools. Each shadow pool receives excitation from its associated excitatory pool. A neuron can participate more than once in the chain. Additional excitatory (solid line) and inhibitory (dashed lines) inputs and outputs are allowed (marked with curved lines).

of the network, as in a completely random network. Similar connectivity, but for different reasons, was suggested in Hayon (2002) and Mehring et al. (2003). A sketch of the connectivity scheme of a modified synfire chain is shown in Figure 4. These inhibitory pools do not carry specific information down the chain, as is the case for the excitatory pools, but rather echo a synchronized activity of their attached excitatory pool. The role of the shadow pools is to guarantee a correct amount of inhibition during the propagation of the wave. The excess of converging connections in the network, due to the embedded chain, induces an excess of correlated excitation, which may lead to global excitation (Aviel, Mehring et al., 2003). The sole purpose of the shadow pools is to cancel that excess of excitation. An analogous shadow pattern is associated with every HCA in an HCA model. ˜ E . This leads to the facThe size of a shadow pattern is defined as wI ≡ dw tor d, representing the relative strength of inhibitory to excitatory currents, due to a pattern or pool, affecting a neuron that is connected to both: √ −JI wI gJ0 Kd˜ gd˜ d≡ = = √ , x ∈ {E, I}. √ JwE γ J0 KI √ In other words wI = d( γ /g)wE . In the simulations reported below, we use d = 2 for HCAs and d = 1 for SFCs, that is, wI = 0.2 · wE and wI = 0.1 · wE , respectively. These values were chosen after a search for optimal d values that lead to the smoothest background behavior after pattern ignition.

706

Y. Aviel, D. Horn, and M. Abeles 3

Population CV

2.5

2

SFC

HCA

1.5

1

0.5 0.05

0.06

0.07

0.08

0.09 P/N

0.1

0.11

0.12

0.13

E

Figure 5: DBN preignition population CV as a function of α, for three values of K (500, dotted curve; 1000, dashed curve; and 1500, solid line). The transition in SFC (+) occurs before that of HCA (the square marks) and is independent of K. Capacity is 0.115 and 0.07 for HCA and SFC, respectively.

We refer to this new type of network as a doubly balanced network (DBN). To avoid confusion, we will refer to the previous case, d = 0, as a singly balanced network (SBN). In Figure 5, we repeat the simulations of Figure 3, only this time we simulate DBNs. Since the pre- and postignition curves overlap in the DBN, we present only the preignition curve. The two cases are plotted on the same scale, so that the difference in capacity can be appreciated. As evident, the capacity of the DBN is superior to that of the SBN in both cases. 4.3 Capacity. In this section, we show that the maximal number of patterns that can be embedded in the network is limited by combinatorial considerations of synaptic resources and is proportional to N. That such a limit has to exist follows from simple counting of all excitatory synapses. On one hand, we know there are KN such synapses available. On the other, we know that each HCA, or pair of consecutive pools in an SFC, use up w2 P of them. Hence, Pw2 < KN, or α = N < wK2 = C12 . w A more careful analysis can take care of the accounting of synapses in the way they are assigned within our model. We divide the K excitatory synapses of each neuron into chunks of L synapses. A neuron of population x (E or I) can participate in at most m ≡ K/L patterns due to synaptic constraints. The total number of chunks, Nx m, sets an upper bound on the

Memory Capacity of Balanced Networks

707

number of patterns, since embedding P patterns requires wx P chunks. Hence we get wx Pmax ≤ m · Nx ,

(4.4)

where Pmax is the maximal number of patterns or pools. Next, defining , we find that αx ≡ PNmax x αx ≤

√ m K/CL K = . wx wx

(4.5)

To leading order in NE , this turns into √  K/CL K (4.6) √ NE = (Cw CL Dx )−1 NE − O( NE ), Dx Cw K √ where Dx ≡ d/(g/ γ ) if x = I, or 1 for x = E. Thus, we conclude that synaptic combinatorial considerations lead to a maximal number of patterns Pmax . If DI ≥ 1, then γ αI < αE , and the inhibitory neurons set the maximum value of P. Otherwise, if DI = 1, the excitatory neurons set Pmax : αx Nx =

Pmax = αmax NE   αmax ≡ min (Cw CL )−1 , (Cw CL DI )−1 .

(4.7)

In our DBN, as well as the SBN where DI < 1, the excitatory neurons determine the limit to be Pmax = (Cw CL )−1 NE . Substituting the parameters of our HCA in equation 4.5, we get Pmax ∼ = 0.12NE . In Figure 6, Pc of HCAs is plotted as a function of NE . A linear relation can be observed up to values of NE as high as 45,000. Furthermore, the proximity of the dynamical capacity to the combinatorial one is evident. Results for cases of SBN are also plotted for comparison. As already noted, their dynamical capacity is lower than that of DBNs. 4.4 Allowed Phase Space. We are now in a position to return to the issues of scaling and inquire about the dependence of Cb of equation 4.3 on network load. We fix α and progressively increase wE in an SBN until global oscillation appears. As in Figures 3 and 5, the curves display a critical point, wc . This criticality √ was discussed in Aviel, Mehring et al. (2003), where we found wc = Cb K. In Figure 7, we repeat these simulations for various values of α and K. Each simulation (Cw , α, K) is repeated five times to give an indication of the trial error. We also plot a crude estimate, based on simulations, of Ca and the combinatorial capacity upper limit, αmax , according to equation 4.7.

708

Y. Aviel, D. Horn, and M. Abeles

Figure 6: Combinatorial and dynamical capacities of HCAs as a function of NE for DBN (o) and SBN (∗). The solid line is αmax · NE .

Figure 7: The allowed region of Cw , the shaded gray area, is limited by Ca from below and Cb from above. The latter, represented by markers with error bars, is estimated from numerical simulations for three different values of K. The markers are slightly shifted with respect to each other—K = 500 (circles) to the left and 1500 (squares) to the right—to allow easy interpretation. These values lie below the constraint Cw CL α < 1 that led us (see equation 4.7) to the combinatorial upper bound αmax .

Memory Capacity of Balanced Networks

709

In the phase portrait illustrated in Figure 7, values of Cw and α that are above the combinatorial upper bound (dashed-dotted line) are forbidden due to the synaptic constraints of our model. Combinations of Cw and α that are below the combinatorial upper bound but outside the shaded area lead to instability of the AS. Below Ca , pattern activities dissolve into the background; hence, they are unstable. The only regime that allows stability of both the AS and the pattern activity is the shaded region in the figure. As the network load increases, Cb and Ca approach each other, shrinking the allowed regime of Cw , in which embedding is possible. In a DBN, the Cb values are higher, closer to the combinatorial upper bound. This enlarges the stability regime of the assemblies in a BN to the extent that it enables realizing the combinatorial upper bound in some cases (Aviel, Horn, & Abeles, 2003a). When embedding SFCs in the BN, the phase portrait stays qualitatively simiv, but Cb values are lower, leading to lower dynamical capacity than in the case of HCAs. 5 Discussion In this letter, we studied embedding of memory patterns in a balanced network of IF neurons. Two types of memories were used: Hebbian cell assemblies (HCAs; Hebb, 1949) and synfire chains (SFCs; Abeles, 1991). We propose a scaling behavior of synaptic weights and other parameters that render our model invariant to changes in K, the synaptic size of the network. This square root scaling allows for high capacity of both HCA and SFC. We emphasized the scaling of variables with K, but it is usually NE that one varies. Using the relation K = εNE , these two variables are linearly related. Note, however, that K may also be assumed to be constant, or vary at a different rate from NE , and our results will still be valid as long as K/NE  1. In particular, the increase of the maximal number of patterns Pmax with NE is valid even if K is constant. We distinguished between the traditional balanced network, which we termed singly balanced network (SBN), and our new doubly balanced network (DBN). In a DBN, memory patterns embedded in the excitatory-toexcitatory part of the connectivity matrix project also onto their shadow patterns. The shadow patterns are merely random pools of inhibitory neurons that receive inputs from their associated excitatory patterns. Counteracting emerged excitatory correlations by inhibitory ones imposes another type of balance. Hence, it is a doubly balanced network. We have shown in Aviel, Horn, and Abeles (2003b) that an optimal choice of d, the ratio of inhibitory to excitatory correlation currents, achieves the desired effect. If d is too small, background oscillations appear, as was the case of the SBN. If d is too large, the induced inhibition kills the synfire waves. The proper value, around

710

Y. Aviel, D. Horn, and M. Abeles

d = 1, allows for optimal performance. The DBNs have the advantage that their background activity during memory recall stays asynchronous even for high memory loads. Another indication of the stabilizing affect of the shadow patterns is given in Aviel et al. (2003b), where it is shown that if a strong, synchronized input is used to ignite a wave in an SBN, rather than a step current, then global oscillations are inevitable. With a DBN, on the other hand, waves are possible on top of asynchronous background activity. Introducing double balance is only one way of stabilizing the AS. Other ways involve more sophisticated neuronal models (Wang, 1999), or introducing variability through nonhomogeneous neuronal population or through different connectivity schemes. For example, models such as Amit and Brunel (1997) and van Vreeswijk and Sompolinsky (1998) used a variable number of synapses on each neuron. This leads to a broad distribution of firing rates across the population, which in turn introduces variability that helps to stabilize the AS. In our work, however, each neuron receives exactly the same number of synapses (Brunel, 2000). Here we suggest a stabilizing mechanism that is directly related to the cause of the instability. Clearly, additional stabilizing mechanisms, such as variable number of synapses, variable transmission delays, or more sophisticated neuronal models, will only help to increase the AS’s stability. In the current-based IF model used here, the synaptic currents are independent of the membrane voltage. While this significantly simplifies the model, it is also different from the more biologically plausible conductancebased IF model. These two models show appreciable differences if operated under balanced conditions (Wang, 1999; Lerchner, Ahmadi, & Hertz, 2004). Under balanced input conditions, the conductance-based IF model has an effective membrane time constant that vanishes for large K. The currentbased IF model does not change its effective membrane time constant. This can be a subject for further investigation. A capacity limit αc = 0.12 of HCAs calls for comparison with analytic bounds obtained for binary models, like αc = 0.14 in the Hopfield model (Amit et al., 1985). The comparison must be done with care, as the two types of neuronal models are qualitatively different. Hertz (1999) has argued that a capacity limit obtained in a network of IF neurons should be multiplied by τ/2 to compare it with a network of binary neurons. Hence, the αc = 0.115 obtained here is equivalent to αc = 0.57 in a binary model. It is not surprising that the last number is higher than 0.14, since our model’s memory patterns are sparse, as, for example, in the Tsodyks and Feigelman (1988) model, where larger capacities were achieved. An example of an IF network with high capacity was demonstrated by Sommer and Wennekers (2001). However, their network was not balanced, and memories were retrieved from oscillatory modes. To the best of our knowledge, the model presented here is the first to obtain such a high capacity in an asynchronous mode of a network of spiking neurons.

Memory Capacity of Balanced Networks

711

Finally, let us touch some of the interesting open issues. The firings of the neurons in the ignited pool are at a much higher rate and higher regularity than those reported by delayed match-to-sample experiments (Miyashita & Chang, 1988). In our model, neurons in the ignited pool receive much more excitatory than inhibitory input and therefore do not operate in the balanced-input regime anymore. Ways to reduce the ignited state’s firing rate have been suggested (Amit & Brunel, 1997; Wang, 1999). It will be interesting to incorporate these studies with the DBN approach in order to better fit experimental data. In this work, we recall only one pattern at a time. We require the ignited pattern to be stable (i.e., sustained high firing rate for HCA or sustained propagation of synfire wave) for at least 100 ms without provoking global oscillations. The next step is to ask how many concurrent patterns can be successfully recalled. This is yet another type of capacity. Preliminary results show that three HCAs can be recalled in a K = 1500 model, one of which will survive the competition and exhibit sustained activity for hundreds of milliseconds. Our model uses binary synapses—a synapse either exists, with the strength of one synaptic unit, or is absent. We strengthen bonds if the two neurons participate together in more memories. This construction leads to the combinatorial maximum capacity. One may speculate that different synaptic arrangements, for example, along the spirit of the Willshaw model (Willshaw et al., 1969), will lead to even higher dynamical capacity. Acknowledgments This work was supported in part by grants from GIF and DIP. References Abeles, M. (1982). Local cortical circuits—An electrophysiological study. Berlin: Springer-Verlag. Abeles, M. (1991). Corticonics. Cambridge: Cambridge University Press. Amit, D.J. (1989). Modeling brain function: The world of attractor neural networks. Cambridge: Cambridge University Press. Amit, D.J., & Brunel, N. (1997). Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. Cerebral Cortex, 7, 237–252. Amit, D.J, Gutfreund, H., & Sompolinsky, H. (1985). Storing infinite numbers of patterns in a spin-glass model of neural networks. Physical Review Letters, 55, 1530–1533. Aviel, Y., Horn, D., & Abeles, M. (2003a). The doubly balanced network of spiking neurons: A memory model with high capacity. In S. Becker, S. Thrun, & K.Obermayer (Eds.), Advances in neural information processing systems, 15. Cambridge, MA: MIT Press.

712

Y. Aviel, D. Horn, and M. Abeles

Aviel, Y., Horn, D., & Abeles, M. (2003b). Synfire waves in small balanced networks. Neurocomputing, 58–60, 123–127. Aviel, Y., Mehring, C., Horn, D., & Abeles, M. (2003). On embedding synfire chains in a balanced network. Neural Computation, 15(6), 1321–1340. Bienenstock, E. (1995). A model of the neocortex. Network: Computation in Neural Systems, 6, 179–224. Braitenberg, V. (1978). Cell assemblies in the cerebral cortex. In R. Heim & G. Palm (Eds.), Theoretical approaches to complex systems. Berlin: Springer. Braitenberg, V., & Schuz, A. (1991). The anatomy of the cortex: Statistics and geometry. Berlin: Springer. Brunel, N. (2000). Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons. J. Comput. Neurosci., 8(3), 183–208. Brunel, N., & Wang, X.-J. (2001). Effects of neuromodulation in a cortical network model of object working memory dominated by recurrent inhibition. Journal of Computational Neuroscience, 11, 63–85. Compte, A., Brunel, N., Goldman-Rakic, P.S., & Wang, X.-J. (2000). Synaptic mechanisms and network dynamics underlying spatial working memory in a cortical network model. Cerebral Cortex, 10, 910–923. Diesmann, M., Gewaltig, M.O., & Aertsen, A. (1995). SYNOD: An environment for neural systems simulations. Rehovot, Israel: Weizmann Institute of Science. Diesmann, M., Gewaltig, M.O., & Aertsen, A. (1999). Stable propagation of synchronous spiking in cortical neural networks. Nature, 402(6761), 529–533. Funahashi, S., Bruce, C.J., & Goldman-Rakic, P.S. (1989). Mnemonic coding of visual space in the monkey’s dorsolateral prefrontal cortex. Journal of Neurophsiology, 61, 331–349. Gerstein, G.L., & Mandelbrot, B. (1964). Random walk models for the spike activity of a single neuron. Biophysical Journal, 4, 41–68. Gerstner, W., & van Hemmen, L. (1992). Associative memory in a network of “spiking” neurons. Network, 3, 139–164. Hansel, D., Mato, G., & Meunier, C. (1995). Synchronization in excitatory neural networks. Neural Computation, 7, 307–337. Hayon, G. (2002). Modeling compositionality in biologival neural networks by dynamic binding of synfire chains. Unpublished doctoral dissertation, Hebrew University, Jerusalem. Hebb, D.O. (1949). The organization of behavior. New York: Wiley. Herrmann, M., Hertz, J.A., & Prugel-Bennett, A. (1995). Analysis of synfire chains. Network: Comput. Neural Syst., 6, 403–414. Hertz, J.A. (1999). Modeling synfire networks. In G. Burdet, P. Combe, & O. Parodi (Eds.), Neuronal information processing—From biological data to modelling and application. Carg`ese, France. Hopfield, J.J. (1982). Neural networks and physical systems with emergent collective computational abilities. PNAS, 79, 2554–2558. Horn, D., Sagi, D., & Usher, M. (1991). Segmentation, binding and illusory conjunctions. Neural Computation, 3, 510–525. Huang, P.E., & Stevens, F.C. (1997). Estimating the distribution of synaptic reliabilities. Journal of Neurophsiology, 78(6), 2870–2880.

Memory Capacity of Balanced Networks

713

Izhikevich, E.M., Gally, J.A., & Edelman, G.M. (2004). Spike-timing dynamics of neuronal groups. Cerebral Cortex, 14, 933–944. Lerchner, A., Ahmadi, M., & Hertz, J. (2004) High conductance states in a mean field cortical network model. Neurocomputing, 58–60, 935–940. Levy, N., Horn, D., Meilijson, I., & Ruppin, E. (2001). Distributed synchrony in cell assembly of spiking neurons. Neural Networks, 14, 815–824. Mehring, C., Hehl, U., Kubo, M., Diesmann, M., & Aertsen, A. (2003). Activity dynamics and propagation of synchronous spiking in loacally connected random networks. Biological Cybernetics, 88, 395–408. Miller, R. (1996). Neural assemblies and laminar interactions in the cerebral cortex. Biological Cybernetics, 75(3), 253–261. Miyashita, Y., & Chang, H.S. (1988). Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature, 331, 68–70. Morrison, A., Mehring, C., Geisel, T., Aertsen A., & Diesmann, M. (2004). Advancing the boundaries of high connectivity network simulation with distributed computing. Manuscript submitted for publication. Shadlen, M.N., & Newsome, W.T. (1994). Noise, neural codes and cortical organization. Curr. Opin. Neurobiol., 4(4), 569–579. Sommer, F.T, & Wennekers, T. (2000). Modeling studies on the computaional function of fast temporal structure in cortical circuit activity. Journal of Physiology Paris, 94(5–6), 473–488. Sommer, F.T, & Wennekers, T. (2001). Associative memory in networks of spiking neurons. Neural Networks, 14(6–7), 825–834. Tetzlaff, T., Buschermohle, ¨ M., Geisel, T., & Diesmann, M. (2003). The spread of rate and correlation in stationary cortical networks. Neurocomputing, 52–54, 949–954. Treves, A. (1990). Graded-response neurons and information encodings in autoassociative memories. Physical Review A, 42(4), 2418. Tsodyks, M.V., & Feigelman, M.V. (1988). The enhanced storage capacity in neural networks with low activity level. Europhysics Letters, 6(2), 101–105. Tuckwell, H.C. (1988). Introduction to theoretical neurobiology. Cambridge: Cambridge University Press. van Vreeswijk, C., & Sompolinsky, H. (1998). Chaotic balanced state in a model of cortical circuits. Neural Comput., 10(6), 1321–1371. von der Malsburg, C., & Schneider, W. (1986). A neural cocktail party processor. Biological Cybernetics, 54, 29–40. Wang, X.-J. (1999). Synaptic basis of cortical persistent activity: The importance of NMDA receptors to working memory. Journal of Neuroscience, 19(21), 9587– 9607. Willshaw, D.J., Buneman, O.P., & Longuet-Higgins, H.C. (1969). Nonholographic associative memory. Nature, 222, 960–962. Received October 6, 2003; accepted August 3, 2004.

LETTER

Communicated by Irwin Sandberg

On the Capabilities of Higher-Order Neurons: A Radial Basis Function Approach Michael Schmitt [email protected] ¨ Mathematik, Ruhr-Universit¨at Lehrstuhl Mathematik und Informatik, Fakult¨at fur Bochum, D–44780 Bochum, Germany

Higher-order neurons with k monomials in n variables are shown to have Vapnik-Chervonenkis (VC) dimension at least nk + 1. This result supersedes the previously known lower bound obtained via k-term monotone disjunctive normal form (DNF) formulas. Moreover, it implies that the VC dimension of higher-order neurons with k monomials is strictly larger than the VC dimension of k-term monotone DNF. The result is achieved by introducing an exponential approach that employs gaussian radial basis function neural networks for obtaining classifications of points in terms of higher-order neurons.

1 Introduction Higher-order neurons are simple but powerful extensions of linear neuron models. They introduce the concept of nonlinearity by incorporating monomials, that is, products of input variables, as an intermediate step— effectively a hidden layer—of computation. A common way of specifying a higher-order neuron is to determine the number of input variables and the number of monomials, but leaving the order of the monomials open. Thus, one obtains a powerful neuron model that is capable of computing with unlimited degrees of nonlinearity. The computational capabilities that arise from this model, however, are far from being completely understood. One of the well-studied theoretical notions used to characterize the computational richness of neural networks is the Vapnik-Chervonenkis (VC) dimension. More generally, the VC dimension of a function class quantifies its classification capabilities (Vapnik & Chervonenkis, 1971). It indicates the cardinality of the largest set for which all possible binary-valued classifications can be obtained using functions from the class. The VC dimension is also well founded as a measure for the sample complexity of learning (see, e.g., Anthony & Bartlett, 1999). It yields estimates for the number of examples needed by a learning algorithm to output functions that have low generalization errors. Neural Computation 17, 715–729 (2005)

c 2005 Massachusetts Institute of Technology 

716

M. Schmitt

We establish here a new lower bound on the VC dimension of higherorder neurons. We show that the higher-order neuron with k monomials in n variables has VC dimension at least nk + 1. The largest lower bound that has been known previously is derived from the lower bound for boolean formulas in k-term monotone disjunctive normal form (DNF), that is, disjunctions of at most k monomials without negations. This bound has been obtained by Littlestone (1988). In particular, Littlestone has shown that the class of k-term monotone l-DNF formulas (i.e., with monomials containing at most l variables)
 has VC dimension at least lklog(n/m), where l ≤ m ≤ n, and k ≤ m1 . Using l = n/4 and m = n/2, for instance, this yields the lower bound nk/4 for the VC dimension of k-term monotone DNF and, hence, of higher-order neurons with k monomials, where k has to satisfy the given constraints. The new bound that we provide here for higher-order neurons supersedes this previous bound in a threefold way. First, it improves the bound from k-term monotone DNF formulas in value. Second, it releases k from the constraints through n in that the new bound holds for every n and k—in particular, for values of k that are larger than the number of monotone monomials. Finally, nk + 1 is even larger than the VC dimension of the class of k-term monotone DNF formulas itself. We show that the difference between both dimensions is larger than k log(k/e) + 1. So far, a considerable number of results and techniques for VC dimension bounds have been provided in the context of real-valued function classes (see, e.g., Bartlett & Maass, 2003). For specific subclasses that can be computed by higher-order neurons, tight bounds have been calculated. Karpinski and Werther (1993) have shown that univariate polynomials with at most k terms have a VC dimension proportional to k.1 Further, the VC dimension of the class of monomials over the reals is equal to n (see Ehrenfeucht, Haussler, Kearns, & Valiant, 1989; Schmitt, 2002c, for lower and upper bound, respectively). There is also a VC dimension result known for n-variate ddegree polynomials (see, e.g., & Lindenbaum, 1998). This class  Ben-David  has VC dimension equal to n+d . However, as the class contains polynomid   n+d als consisting of d terms and the bound k on the number of monomials restricts the number of variables in terms of k, this result entails for higherorder neurons (without a constraint on the degree) a lower bound not better than the bound due to Littlestone (1988). Previous work has established techniques for deriving lower bounds on the VC dimension for quite general types of real-valued function classes.

1 Strictly speaking, Karpinski and Werther (1993) studied a related notion—the socalled pseudo-dimension. Following their methods, it is not hard to obtain this result for the VC dimension (see also Schmitt, 2002a).

On the Capabilities of Higher-Order Neurons

717

Building on results by Lee, Bartlett, & Williamson (1995), Erlich, Chazan, Petrack, & Levy (1997) provide powerful means for obtaining lower bounds for parameterized function classes.2 An essential requirement for using these techniques, however, is that the function class is “smoothly” parameterized, a fact that does not apply to the exponents of polynomials. The lower bound method of Koiran and Sontag (1997) for various types of neural networks, generalized by Bartlett, Maiorov, and Meir (1998) to neural networks with a given number of layers, cannot be employed for higher-order neurons either. This technique is constrained to networks where each neuron computes a function with finite limits at infinity, a property monomials do not have. Further, Koiran and Sontag (1997) designed a lower-bound method for networks consisting of linear and multiplication gates. However, the way these networks are constructed, with layers consisting of products of linear terms,3 does not give rise to higher-order neurons even when the number of layers is restricted. We provide a completely new approach to the derivation of lower bounds on the VC dimension of higher-order neurons. First, we establish the lower bound nk + 1 on the VC dimension of a specific type of radial basis function (RBF) neural networks (see, e.g., Haykin, 1999). The networks considered here have k gaussian units as computational elements and satisfy certain assumptions with respect to the input domain and the values taken by the parameters. The bound for these networks improves a result of Erlich et al. (1997) in combination with Lee et al. (1995) who established the lower bound n(k − 1) for RBF networks4 with restrictions on neither inputs nor parameters. Then we use our result for RBF networks to obtain the lower bound on the VC dimension of higher-order neurons. Thus, RBF networks open a new way to assess the classification capabilities of higherorder neurons. This gaussian RBF approach has also proven to be helpful in a different context dealing with the roots of univariate polynomials (Schmitt, 2004). Higher-order neurons are a special case of a particular type of neural networks, the so-called product unit neural networks (Durbin & Rumelhart, 2 A parameterized function class is given in terms of a function having two types of variables: input variables and parameter variables. The function class is obtained by instantiating the parameter variables with, in general, real numbers. Neural networks are prominent examples for parameterized function classes. l 3 Precisely, such a layer uses products of the form (x − ai ) where it is crucial that i=1 there is no bound on l. 4 These results and the one presented here concern RBF networks with uniform width, that is, where all units have equal width. This constraint is not a fundamental restriction since, as shown by Park and Sandberg (1991), these networks still have universal approximation capabilities. As far as the VC dimension is concerned, however, better lower bounds are known for more general types of RBF networks (Schmitt, 2002b).

718

M. Schmitt

1989). It immediately follows from the bound for higher-order neurons established here that the VC dimension of product unit neural networks with n input nodes and one layer of k hidden nodes (that is, nodes that are neither input nor output nodes) is at least nk + 1. Concerning upper bounds for the VC dimension of higher-order neurons, there are two relevant results. The bound O(n2 k4 ) due to Karpinski and Macintyre (1997) is the smallest bound known for higher-order neurons with unlimited degree (see also Schmitt, 2002c). The higher-order neuron with n variables, k monomials, and degree no larger than d has VC dimension no more than 2nk log(9d) (Schmitt, 2002c). The derivation of the new lower bound not only narrows the gap between upper and lower bounds, but also gives rise to subclasses of degree-restricted higher-order neurons for which the bound is optimal up to the factor 2 log(9d). We introduce definitions and notation in section 2. Section 3 provides geometric constructions that are required for the derivations of the main results presented in section 4. Finally, in section 5, we compare the new bound with the upper bound for k-term monotone DNF formulas and show that the new bound exceeds the VC dimension of this class. 2 Definitions A higher-order neuron (or sigma-pi unit) with k monomials in n variables computes the functions b

b

b

b

a0 + a1 x11,1 · · · xn1,n + · · · + ak x1k,1 · · · xnk,n

(2.1)

with real coefficients a0 , . . . , ak (the output weights of the neuron with bias a0 ) and nonnegative integer exponents b1,1 , . . . , bk,n . The coefficients and the exponents are the adjustable parameters of the neuron. For given n and k, the function class constituted by all functions in equation 2.1 is also known as the class of k-sparse n-variate polynomials. If the exponents b1,1 , . . . , bk,n are allowed to be arbitrary real numbers, each monomial becomes a product unit, and we obtain a product unit neural network with one hidden layer of k product units. We use bold symbols to indicate vectors, such as x and ci ; nonbold symbols are reserved for scalar variables. Let · denote the Euclidean norm. A radial basis function neural network (RBF network, for short) with n input nodes computes functions that map from Rn to R and can be written as     x − c1 2 x − ck 2 + · · · + w , w0 + w1 exp − exp − k σ2 σ2 where k is the number of RBF units. This particular type of network is also known as a gaussian RBF network. Each exponential term corresponds to the function computed by a gaussian RBF unit with center ci ∈ Rn , where

On the Capabilities of Higher-Order Neurons

719

n is the number of variables, and width σ ∈ R \ {0}. The width is a network parameter that we assume here to be equal for all units, that is, we consider RBF networks with uniform width. Further, w0 , . . . , wk are the output weights, and w0 is also referred to as the bias of the network. The Vapnik-Chervonenkis (VC) dimension of a class F of real-valued functions is defined via the notion of shattering. A set S ⊆ Rn is said to be shattered by F if every dichotomy of S is induced by F , that is, if for every pair (S− , S+ ), where S− ∩ S+ = ∅ and S− ∪ S+ = S, there is some function f ∈ F such that sgn ◦ f (S− ) ⊆ {0} and sgn ◦ f (S+ ) ⊆ {1}. Here sgn: R → {0, 1} denotes the sign function, satisfying sgn(x) = 1 if x ≥ 0, and sgn(x) = 0 otherwise. The VC dimension of F is then defined as the cardinality of the largest set shattered by F . (It is said to be infinite if there is no such set.) The VC dimension of a neuron or a neural network is equated with the VC dimension of the class of functions computed by the neuron or the neural network, respectively. Finally, we make use of the geometric notions of a ball and a hypersphere. A ball in Rn is given in terms of a center c ∈ Rn and a radius ρ ∈ R as the set B(c, ρ) = {x ∈ Rn : x − c ≤ ρ}. A hypersphere is the set of points on the surface of a ball, that is, S(c, ρ) = {x ∈ Rn : x − c = ρ}. 3 Ancillary Constructions In the following, we provide the geometric constructions that are the basis for the main result in Section 4. The idea pursued here is to represent classifications of sets using unions of balls, where a point is classified as positive if and only if it is contained in some ball. In order to be shattered, the sets are chosen to satisfy a certain condition of independence with respect to the positions of their elements. The points are required to lie on hyperspheres such that each hypersphere is maximally determined by the set of points. In other words, removing any point increases the set of possible hyperspheres that contain the reduced set. The following definition makes this notion of independence precise. Definition 1. A set Q ⊆ Rn of at most n + 1 points is in general position for hyperspheres if the system of equalities p − c = η,

for all p ∈ Q,

(3.1)

720

M. Schmitt

in the variables c = (c1 , . . . , cn ) and η has a solution and, for every q ∈ Q, the solution set is a proper subset of the solution set of the system p − c = η,

for all p ∈ Q \ {q}.

(3.2)

Given a set of points that satisfies this definition and lies on a hypersphere, we next want to find a ball such that one of the points lies outside the ball while the other points are on its surface. We show that this can be done provided that the set is in general position for hyperspheres. Moreover, the ball can be chosen with the center and the radius as close as possible to the center and the radius of the hypersphere that contains all points. Lemma 1. Suppose that Q ⊆ Rn is a set of at most n + 1 points in general position for hyperspheres, and let q ∈ Q. Further, let c ∈ Rn , η ∈ R be a solution of the system p − c = η,

for all p ∈ Q.

(3.3)

Then, for every ε > 0, there exists a solution c(ε) ∈ Rn , η(ε) ∈ R of the system p − c(ε) = η(ε),

for all p ∈ Q \ {q},

q − c(ε) > η(ε) satisfying c − c(ε) < ε and |η − η(ε)| < ε. Proof. Without loss of generality, we may assume that η > 0. (If η = 0, then we have |Q| = 1, and the statement is trivial.) Since c and η solve the system 3.3, c and ϑ = η2 − c 2 are a solution of the system p 2 − 2pc = ϑ,

for all p ∈ Q.

(3.4)

Because Q is in general position for hyperspheres, the solution set of the system 3.4 is a proper subset of the solution set of the system p 2 − 2pc = ϑ,

for all p ∈ Q \ {q}.

(3.5)

According to facts from linear algebra, there exist a ∈ Rn and α ∈ R such that for every λ = 0, we have with c + λa and ϑ + λα a solution of the system 3.5 that does not solve the system 3.4. For a given ε > 0, choose λ(ε) ∈ R \{0} such that |λ(ε)| is sufficiently small to satisfy   λ(ε)a < ε and | ϑ + c 2 − ϑ + λ(ε)α + c + λ(ε)a 2 | < ε. (3.6)

On the Capabilities of Higher-Order Neurons

721

It is obvious that the second inequality can be met due to the fact that the equation ϑ + c 2 = η holds, which we get from the definition of ϑ and the assumption η > 0. Since c + λ(ε)a and ϑ + λ(ε)α solve equation 3.5 but not 3.4, it follows that q 2 − 2q(c + λ(ε)a) = ϑ + λ(ε)α, which, using q 2 − 2qc = ϑ from equation 3.4, is equivalent to −2λ(ε)qa = λ(ε)α. Due to this inequality, we can choose the (not yet specified) sign of λ(ε) such that −2λ(ε)qa > λ(ε)α. Again with q 2 − 2qc = ϑ, it follows that q 2 − 2q(c + λ(ε)a) > ϑ + λ(ε)α, and, therefore, q − (c + λ(ε)a) 2 > ϑ + λ(ε)α + c + λ(ε)a 2 . Hence, defining c(ε) = c + λ(ε)a and η(ε) =



ϑ + λ(ε)α + c + λ(ε)a 2 ,

we obtain q − c(ε) > η(ε). Furthermore, the inequalities 3.6 imply that c − c(ε) < ε and |η − η(ε)| < ε hold as claimed. We now apply the previous result to show that any dichotomy of a given set of points can be obtained using balls. As the set may generally be a subset of some larger set, we also ensure that the balls do not enclose any additional point. Further, we guarantee that this can be done with all centers remaining positive, a condition that will turn out to be useful in the following section. We say here that a vector is positive if all of its components are larger than zero. Lemma 2. Let Q ⊆ Rn be a set of n points in general position for hyperspheres, and let P ⊆ Rn be a finite set with Q ⊆ P. Assume further that there exists a positive center c ∈ Rn and a radius η ∈ R such that Q ⊆ S(c, η), P ∩ B(c, η) = Q.

722

M. Schmitt

Then for every R ⊆ Q, there exists a positive center d ∈ Rn and a radius ζ ∈ R such that R ⊆ S(d, ζ ), P ∩ B(d, ζ ) = R. Proof. Clearly, it is sufficient to consider sets R that are proper subsets of Q. Without loss of generality, we may assume that |R| = |Q| − 1. The general case then follows inductively. Suppose that q ∈ Q, and let R = Q \ {q}. According to lemma 1, for every ε > 0 there exist c(ε), η(ε) satisfying p − c(ε) = η(ε),

for all p ∈ Q \ {q},

(3.7)

q − c(ε) > η(ε),

(3.8)

c − c(ε) < ε,

(3.9)

|η − η(ε)| < ε.

(3.10)

Obviously, property (3.7) implies that R ⊆ S(c(ε), η(ε)). Property (3.8) states that q ∈ B(c(ε), η(ε)). Since the assumption P ∩ B(c, η) = Q implies that for every p ∈ P \ Q the constraint p − c > η holds, properties (3.9) and (3.10) entail the condition p − c(ε) > η(ε) for all sufficiently small ε. Thus, for any such ε, we get the assertion P ∩ B(c(ε), η(ε)) = R. Further, as c is positive, property (3.9) ensures that c(ε) is positive for some sufficiently small ε. Hence, the claim follows for d = c(ε), ζ = η(ε). 4 VC Dimension Bound for Higher-Order Neurons Before getting to the main result, we derive the lower bound nk + 1 for the VC dimension of a restricted type of RBF network. For more general RBF networks, results of Erlich et al. (1997) and Lee et al. (1995) yield the lower bound n(k − 1). The following theorem is stronger not only in the value of the bound, but also in the assumptions that hold. The points of the shattered set all have the same distance from the origin, the centers of the RBF units are rational numbers, and the width can be chosen arbitrarily small. Theorem 1. Let n ≥ 2, k ≥ 1, and ρ > 0 be given. There exists a set P ⊆ S(0, ρ) ⊆ Rn of nk + 1 points and a real number σ0 > 0 so that P is shattered by the RBF network with k hidden units, positive rational centers, and any width 0 < σ ≤ σ0 .

On the Capabilities of Higher-Order Neurons








723




























































Figure 1: The points of the shattered set are chosen from the intersections of the hypersphere S(0, ρ) with the surfaces of pairwise disjoint balls B(ci , ηi ). All balls have their centers in the positive orthant. There is one additional point s not contained in any of the balls.

Proof. Suppose that B(c1 , η1 ), . . . , B(ck , ηk ) are pairwise disjoint balls with positive centers c1 , . . . , ck ∈ Rn such that for i = 1, . . . , k, the intersection S(ci , ηi ) ∩ S(0, ρ) is nonempty and not a single point. (An example for n = 2 and k = 3 is shown in Figure 1.) For i = 1, . . . , k, let Pi ⊆ S(ci , ηi ) ∩ S(0, ρ) be a set of n points in general position for hyperspheres. (Note that Pi is constrained to lie on two different hyperspheres. This still allows choosing Pi in such a way that it is in general position since Pi contains n (and not n + 1) points, so that the set of possible centers for Pi yields a line.) Further, let s ∈ S(0, ρ) be some point such that s ∈ B(ci , ηi ), for i = 1, . . . , k. We claim that the set P = {s} ∪ P1 ∪ · · · ∪ Pk , which has nk + 1 points, is shattered by the RBF network with the postulated restrictions on the parameters. Assume that (P− , P+ ) is some arbitrary dichotomy of P where s ∈ P− . (We will argue at the end of the proof that the complementary case can be + treated by reversing signs.) Let (P− i , Pi ) denote the dichotomy induced on Pi . By construction, every Pi satisfies

Pi ⊆ S(ci , ηi ) and P ∩ B(ci , ηi ) = Pi .

724

M. Schmitt

Hence by lemma 2, instantiating the set Q with Pi and the set R with P+ i , it follows that there exist positive centers di and radii ζi such that + P+ i ⊆ S(di , ζi ) and P ∩ B(di , ζi ) = Pi ,

for i = 1, . . . , k. Moreover, the centers di can be replaced by rational centers d˜ i that are sufficiently close to di , such that every point of P lying outside the ball B(di , ζi ) is outside the ball B(d˜ i , ζ˜i ) for some ζ˜i ∈ R close to ζi , and every point of P lying on the hypersphere S(di , ζi ) is contained in the ball B(d˜ i , ζ˜i ). Thus, every p ∈ P satisfies p ∈ B(d˜ i , ζ˜i ) if and only if p ∈ P+ i ,

(4.1)

for i = 1, . . . , k. Clearly, since the centers di are positive, the rational centers d˜ i can be chosen to be positive as well. The parameters of the RBF network are specified as follows. The ith unit is associated with the ball B(d˜ i , ζ˜i ). Assigned to it is d˜ i as the center and as 2 output weight the value exp(ζ˜i /σ 2 ) (where σ will determined below) so that the unit contributes the term



 ζ˜i2 x − d˜ i 2 exp exp − σ2 σ2 to the computation of the network. From assertion 4.1, we obtain that every p ∈ P \ P+ i satisfies the constraint p − d˜ i > ζ˜i . Thus, for every sufficiently small σ > 0 and every p ∈ P \ P+ i , we achieve that

 p − d˜ i 2 − ζ˜i2 1 exp − (4.2) < 2 σ k is valid for i = 1, . . . , k. On the other hand, for every p ∈ P+ i , condition 4.1 implies p − d˜ i ≤ ζ˜i , which entails

p − d˜ i 2 − ζ˜i2 exp − σ2

 ≥1

(4.3)

On the Capabilities of Higher-Order Neurons

725

for every σ > 0. Finally, we set the bias term equal to −1. It is now easy to see that the dichotomy (P− , P+ ) is induced by the parameter settings. If p ∈ P− , then, according to inequality 4.2, the weighted output values of the units and the bias sum up to a negative value. In the case p ∈ P+ , we have p ∈ P+ i for some i, and by inequality 4.3, the weighted unit i outputs a value of at least 1, while the other units output positive values, so that the total network output is positive. The construction for the case that classifies s as positive works similarly. We invoke lemma 2 substituting P− i for R and derive the analogous version of assertion 4.1 with P+ replaced by P− i i . Then it is obvious that if the weights defined above are equipped with negative signs and 1 is used as the bias, the network induces the dichotomy as claimed. We observe that σ may have been chosen such that it depends on the particular dichotomy. To complete the proof, we require σ0 to be small enough so that inequality 4.2 holds for σ ≤ σ0 on all points and dichotomies of P. One assumption of the theorem can be slightly weakened. It is not necessary to require that s ∈ S(0, ρ). Instead, every point not contained in any of the balls B(ci , ηi ) can be selected for s. However, the restriction is required for the application of the theorem in the following result, which is the main contribution of this article. For its proof, we recall the definition of a product unit neural network in section 2. Theorem 2. For every n, k ≥ 1, the higher-order neuron with k monomials in n variables has VC dimension at least nk + 1. Proof. We first consider the case n ≥ 2. By theorem 1, for ρ > 0, let P ⊆ Rn , P ⊆ S(0, ρ), be the set of cardinality nk + 1 that is shattered by the RBF network with k hidden units and the stated parameter settings. We show that P can be transformed into a set P that is shattered by the higher-order neuron with k monomials. The weighted output computed by unit i in the RBF network on input p ∈ P can be written as 

p − ci 2 wi · exp − σ2



 p 2 − 2pci + ci 2 = wi · exp − σ2     2pci p 2 + ci 2 exp = wi · exp − σ2 σ2   2 2 ρ + ci = wi · exp − σ2     2p1 ci,1 2pn ci,n · exp · · · exp , σ2 σ2

726

M. Schmitt

where we have used the assumption P ⊆ S(0, ρ) for the last equation and pj , ci,j to denote the jth components of the vectors p, ci , respectively. Consider a product unit network with one hidden layer, where unit i has output weight   ρ 2 + ci 2  wi = wi · exp − σ2 and exponents 2ci,j /σ 2 for j = 1, . . . , n. On inputs from the set P = {(ep1 , . . . , epn ): (p1 , . . . , pn ) ∈ P}, this product unit network computes the same values as the RBF network on P. Moreover, the exponents of the product units are positive rationals. According to theorem 1, for some σ0 , any width 0 < σ ≤ σ0 can be used. Therefore, we may choose σ 2 = 1/ l for some natural number l that is sufficiently large and a common multiple of all denominators occurring in any ci,j , so that the exponents become integers. With these parameter settings, we have a higher-order neuron with k monomials that computes on P the same output values as the RBF network on P. As this can be done for every dichotomy of P, it follows that P is shattered by the higher-order neuron with k monomials. For the case n = 1, we again use the RBF technique and ideas from Schmitt (2002a, 2004). Clearly, the set M = {0, . . . , k} can be shattered by an RBF network with k + 1 hidden units and zero bias. For each i ∈ M, we employ an RBF unit with center i. Given a dichotomy (M− , M+ ), we let the output weight for unit i be −1 if i ∈ M− and 1 if i ∈ M+ . If the width σ is small enough, the output value of the network has the requested sign on every input i ∈ M. Now, let σ be the smallest width sufficient for all dichotomies of M. Then   2  x (x − 1)2 w0 exp − 2 +w1 exp − +··· σ σ2   (x − k)2 · · ·+wk exp − ≥0 σ2 is, by multiplication with exp(x2 /σ 2 ), equivalent to     2x − 1 2kx − k2 exp w0 + w1 exp + · · · + w ≥ 0. k σ2 σ2 The latter can be written as     2x 1 +· · · w0 +w1 exp − 2 exp σ σ2

 2   k 2kx · · ·+wk exp − 2 exp ≥ 0. σ σ2

On the Capabilities of Higher-Order Neurons

727

Substituting y = exp(2x/σ 2 ), this holds if and only if   2  1 k w0 + w1 exp − 2 y + · · ·+wk exp − 2 yk ≥ 0. σ σ Thus, for every dichotomy of M, we obtain a dichotomy of the set M = {e2i/σ : i = 0, . . . , k} 2

induced by a higher-order neuron with k monomials. In other words, M is shattered by this neuron. 5 Comparison with k-Term Monotone DNF A boolean formula that is a disjunction of up to k monomials without negations can be considered as a polynomial restricted to boolean inputs. The previously best-known lower bound for the VC dimension of higher-order neurons with k monomials was the bound for k-term monotone DNF due to Littlestone (1988). By deriving an upper bound for the latter class and applying theorem 2, we show that the VC dimension for higher-order neurons with k monomials is strictly larger than for k-term monotone DNF. We use “log” to denote the logarithm of base 2. Corollary 1. Let n ≥ 1 and 3 ≤ k ≤ 2n . The VC dimension of the higher-order neuron with k monomials in n variables exceeds the VC dimension of the class of k-term monotone DNF in n variables by more than klog(k/e) + 1. Proof. A k-term monotone DNF formula corresponds to a collection of up to k subsets of the set of variables. For n variables, there are no more  n
 than ki=0 2i such collections. The known inequality di=0 mi < (em/d)d , where 1 ≤ d ≤ m, (see, e.g. Anthony & Bartlett, 1999, theorem 3.7) yields k  n

2 i=0

i

<

 e k k

2nk .

By definition, the VC dimension of a finite function class F cannot be larger than log |F |. Hence, the VC dimension for k-term monotone DNF is less than nk − k log(k/e). Theorem 2 implies that this bound falls short of the VC dimension for higher-order neurons with k monomials in n variables by at least k log(k/e) + 1. It is easy to see that in the cases k = 1, 2, which are not covered by corollary 1, the VC dimension of higher-order neurons is larger as well.

728

M. Schmitt

First, as there are no more than 2n boolean monotone monomials, the VC dimension of monotone monomials is at most n. Second, the number of monotone DNF formulas with at most two terms is not larger than 22n + 1, and log(22n + 1) is less than 2n + 1. 6 Conclusion A new lower bound for the VC dimension of higher-order neurons with a given number of monomials has been derived. The bound is stronger and more general than the previous bound established using boolean formulas in monotone DNF. Moreover, the new bound implies that the VC dimension of higher-order neurons with k monomials exceeds the VC dimension of the class of k-term monotone DNF formulas. Therefore, the techniques that use DNF formulas for deriving lower bounds on the VC dimension of higherorder neurons seem to have reached their limits. We have introduced a method whereby higher-order neurons, via gaussian RBF networks, shatter sets. This seems to be paradoxical as, with regard to the domain of the parameters, the gaussian RBF network appears to be more powerful than the higher-order neuron. Each parameter of a gaussian RBF network may assume any real number, whereas the higher-order neuron must have exponents that are nonnegative and integers. Nevertheless, we have shown here that RBF networks can be used to establish lower bounds on the computational capabilities of higher-order neurons. While the previous lower bound via monotone DNF formulas gives rise to monomials with exponents not larger than 1, the approach that uses RBF networks shows that and how large exponents can be employed to shatter sets of a cardinality that is larger than known before. Moreover, the constructions give reason for a completely new interpretation of the exponent vectors of the monomials when higher-order neurons are used for classification tasks. They have been chosen as centers of balls. This perspective might open new ways to the design of learning algorithms for higher-order neurons. With the result presented in this letter, we have narrowed the gap between lower and upper bounds for the VC dimension of higher-order neurons. As the bounds are not yet tight, the hope is that the method introduced here may lead to further insights that eventually yield additional improvements. Acknowledgments This work was supported in part by the IST Programme of the European Community, under the PASCAL Network of Excellence, IST-2002-506778. References Anthony, M., & Bartlett, P. L. (1999). Neural network learning: Theoretical foundations. Cambridge: Cambridge University Press.

On the Capabilities of Higher-Order Neurons

729

Bartlett, P. L., & Maass, W. (2003). Vapnik-Chervonenkis dimension of neural nets. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks, (2nd ed., pp. 1188–1192). Cambridge, MA: MIT Press. Bartlett, P. L., Maiorov, V., & Meir, R. (1998). Almost linear VC-dimension bounds for piecewise polynomial networks. Neural Computation, 10, 2159–2173. Ben-David, S., & Lindenbaum, M. (1998). Localization vs. identification of semialgebraic sets. Machine Learning, 32, 207–224. Durbin, R., & Rumelhart, D. (1989). Product units: A computationally powerful and biologically plausible extension to backpropagation networks. Neural Computation, 1, 133–142. Ehrenfeucht, A., Haussler, D., Kearns, M., & Valiant, L. (1989). A general lower bound on the number of examples needed for learning. Information and Computation, 82, 247–261. Erlich, Y., Chazan, D., Petrack, S., & Levy, A. (1997). Lower bound on VCdimension by local shattering. Neural Computation, 9, 771–776. Haykin, S. (1999). Neural networks: A comprehensive foundation (2nd ed.). Upper Saddle River, NJ: Prentice Hall. Karpinski, M., & Macintyre, A. (1997). Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks. Journal of Computer and System Sciences, 54, 169–176. Karpinski, M., & Werther, T. (1993). VC dimension and uniform learnability of sparse polynomials and rational functions. SIAM Journal on Computing, 22, 1276–1285. Koiran, P., & Sontag, E. D. (1997). Neural networks with quadratic VC dimension. Journal of Computer and System Sciences, 54, 190–198. Lee, W. S., Bartlett, P. L., & Williamson, R. C. (1995). Lower bounds on the VC dimension of smoothly parameterized function classes. Neural Computation, 7, 1040–1053. Littlestone, N. (1988). Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2, 285–318. Park, J., & Sandberg, I. W. (1991). Universal approximation using radial-basisfunction networks. Neural Computation, 3, 246–257. Schmitt, M. (2002a). Descartes’ rule of signs for radial basis function neural networks. Neural Computation, 14, 2997–3011. Schmitt, M. (2002b). Neural networks with local receptive fields and superlinear VC dimension. Neural Computation, 14, 919–956. Schmitt, M. (2002c). On the complexity of computing and learning with multiplicative neural networks. Neural Computation, 14, 241–301. Schmitt, M. (2004). New designs for the Descartes rule of signs. American Mathematical Monthly, 111, 159–164. Vapnik, V. N., & Chervonenkis, A. Y. (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, 16, 264–280. Received March 24, 2004; accepted July 30, 2004.

LETTER

Communicated by Steven Nowlan

Estimating the Posterior Probabilities Using the K-Nearest Neighbor Rule Amir F. Atiya [email protected] Department of Computer Engineering, Cairo University, Giza, Egypt

In many pattern classification problems, an estimate of the posterior probabilities (rather than only a classification) is required. This is usually the case when some confidence measure in the classification is needed. In this article, we propose a new posterior probability estimator. The proposed estimator considers the K-nearest neighbors. It attaches a weight to each neighbor that contributes in an additive fashion to the posterior probability estimate. The weights corresponding to the K-nearest-neighbors (which add to 1) are estimated from the data using a maximum likelihood approach. Simulation studies confirm the effectiveness of the proposed estimator. 1 Introduction The posterior probability is a key variable in any pattern classification problem. For one thing, the optimal classifier (Bayes classifier) is based on classifying the point according to the largest posterior probability. In practice, it is not possible to implement the Bayes classification rule precisely because the posterior probabilities are not known for sure and have to be estimated from the data. An estimate as accurate as possible for the posterior probabilities is therefore very desirable. There are several other reasons for the importance of this estimator. If the classifier has the “reject” option, then it would be desirable to refrain from classifying a pattern based on our confidence in the classification accuracy. This is typically done on the basis of observing the posterior probabilities. Also, for some applications, the user requests not only a classification, but also the probabilities of belonging to each class for any given pattern. An example where this fact is true is the bankruptcy prediction application, whereby one predicts whether a company will default on a loan based on the health of its financial variables (Atiya, 2001). A default/nondefault classification of the companies is desirable but not sufficient. The posterior probabilities will allow banks to compute their expected loss on their loan portfolios. The other issue about the importance of the posterior probabilities is the following. To assess the performance (i.e., the classification accuracy) of a specific classifier, one way is to count the number of training set patterns classified correctly. A much more accurate approach (proposed by FukuNeural Computation 17, 731–740 (2005)

c 2005 Massachusetts Institute of Technology


732

A. Atiya

naga & Kessell, 1971, 1973, and Fukunaga & Hostetler, 1975) is the posterior probability approach, whereby one computes P(correct|x) for each pattern in the training set (using the posterior probabilities or their statistical estimates) and averages all these. This estimator even beats other variations of the counting estimator that eliminate or reduce its bias, such as the holdout estimator, the leave-one-out-estimator, and bootstrap approaches. This is because the posterior probability estimator has a much lower variance than all these other methods (see the reviews and analyses of Glick, 1978; Hand, 1986; Jain, Duin, & Mao, 2000; Kittler & Devijver, 1981; Kroke, 1986; Kulkarni, Lugosi, & Venkatesh, 1998; and Lugosi & Pawlak, 1994). In this article, we propose a new posterior probability estimate based on observing the K-nearest neighbors. The K-nearest-neighbor method has had a long history in the pattern classification and density estimation fields. A well-known estimate of posterior probabilities is the following: the posterior probability for class Cm equals the fraction of its K-nearest-neighbors that belong to class Cm (see Fukunaga & Hostetler, 1975). This estimator treats each neighbor equally. However, the problem is that far-away patterns are less “relevant,” as the densities could have changed significantly. This is more so in posterior probability estimation than in classification, because our simulations indicate that a larger K is needed for posterior probability estimation. This is because it lessens the effect of discretization, and estimating a real quantity needs more points than estimating a binary quantity. In this article, we propose an estimator that weighs the different neighbors differently in the posterior probability computation. Weighted nearest-neighbor classifiers have been considered in the past (see Royall, 1966; Bailey & Jain, 1978; the review by Toussaint, 2002; Devroye, Gyorfi, ¨ & Lugosi, 1996; and Dudani, 1976), but these studies focus on classification, not posterior probability estimation. Also, the approach presented here is different and is based on estimating the weights by a maximum likelihood approach. Bermejo & Cabestany (2000) propose a new classification method whereby they fit a mixture-of-gaussians to the immediate K-nearest-neighbors. A posterior probability estimate is also obtained this way. Non-K-nearest-neighbor methods for estimating posterior probabilities have also been developed (see Guerrero-Curieses, Cid-Sueiro, AlaizRodriguez, & Figueiras-Vadal, 2004). In the approach developed here, we propose a different approach from those discussed. Our method produces different weights for the K-nearest neighbors for each problem, and so the characteristics of each problem as to the smoothness of the densities and the sample size can be reflected in the weighting coefficients. The details of the proposed method are presented proposed in the next section. 2 The Proposed Method Consider a pattern classification problem with M classes Cm , m = 1, . . . , M and assume N training patterns x(n), n = 1, . . . , N are available. The class

Estimating the Posterior Probabilities

733

posterior probability is defined as P(Cm |x). A well-known estimate of the posterior probabilities, based on the K-nearest-neighbor classifier, is the following (see Fukunaga & Hostetler, 1975; Fukunaga & Kessell, 1973; Kanal, 1974). Let Km be the number of patterns among the K nearest neighbors (to point x) that belong to class Cm . Then the estimate is given by ˆ m |x) = Km . P(C K

(2.1)

Rather than treating every pattern among the K nearest neighbors the same, we propose to generalize this estimator by attaching different weights to the different neighbors. Let us illustrate the method through a simple example. Consider the case of using K = 7, three-class case. Assume for a particular point, x neighbors 2, 3, 6, and 7 belong to class 1; neighbors 1 and 4 belong to class 2; and neighbor 5 belongs to class 3. Let the weights attached to the seven neighbors be v1 , . . . , v7 , and assume that these weights are greater
than or equal to zero and sum to 1: 7i=1 vi = 1. Then the posterior probability estimates are given by ˆ 1 |x) = v2 + v3 + v6 + v7 , P(C

ˆ 2 |x) = v1 + v4 , P(C

ˆ 3 |x) = v5 .(2.2) P(C

The optimal weights v1 , . . . , v7 are determined by maximizing the likelihood of the data. Note, of course, that the weights obtained are used for all patterns rather than tuning different weights for every pattern. Here is a general description of the method. To take care of the constraints that the weights are positive and sum to 1, we use a softmax representation. Thus, e wi vi =
K+1 j=1

i = 1, . . . , K + 1,

ewj

(2.3)

where the wj weights are now unconstrained and can take any value. Notice that we have K + 1 weights rather than K weights, as would have been expected. The role and definition of wK+1 will be described shortly. Construct a matrix B, whose first K columns are defined as follows: Bij = 1 if the jth neighbor belongs to class Ci , and zero otherwise. The K + 1st column corresponds to the extra weight wK+1 introduced above, and all its entries equal 1/M (M is the number of classes). For the example presented in the previous paragraph, the B matrix is given by 

0 B = 1 0

1 0 0

1 0 0

0 1 0

0 0 1

1 0 0

1 0 0

1 3 1 . 3 1 3

(2.4)

734

A. Atiya

Note that all columns of B sum to 1. The proposed estimate for the posterior probabilities is ˆ m |x) = P(C


K+1

wi i=1 Bmi e
K+1 wj . j=1 e

(2.5)

Let the B matrix for a specific training pattern x(n) be B(n). Let I(n) ∈ {1, . . . , M} denote the class membership of pattern x(n). The likelihood of the training data set is L=

ˆ N n=1 P



 CI(n) |x(n) = N n=1




K+1 wi i=1 BI(n),i (n)e
K+1 wj j=1 e

.

(2.6)

Note that in reality, the posterior probability estimates for the different patterns are not independent. However, we have assumed independence as a first-order approximation. The dependent case is very hard to analyze. The log likelihood becomes log(L) =

N

n=1


log

K+1 wi i=1 BI(n),i (n)e
K+1 wj j=1 e

.

(2.7)

Now we explain the extra complication of introducing the K + 1st weight wK+1 . It is a regularization parameter that prevents obtaining −∞ log likelihood. If it was not present, then if for any point of a certain class all its neighbors belong to different classes, then we get a zero likelihood no matter what weights we choose. The weight wK+1 will restore some positive probability even if all the neighbors are of a different class. It is as if we add an imaginary K + 1st neighbor whose class identity is not known and therefore is considered to belong to each class with an equal probability. The optimal weighting functions wj are determined by maximizing the log-likelihood function. Because of the simplicity of the optimization task, we have used a simple steepest-ascent algorithm. It is described as follows: 1. Start with some initial choice of weights, say, equal values. 2. Update the weights in the direction of the gradient of the log-likelihood function: wi (new) = wi + η

 N

n=1

BI(n),i (n)ewi


K+1 j=1

BI(n),j (n)ewj

Newi −
K+1 wj j=1 e

,

(2.8)

where η is the step size and the term multiplying η is the gradient.

Estimating the Posterior Probabilities

735

3. Repeat step 2 for many iterations until the change in weights is negligible. The following theorem establishes the global convergence of of the above algorithm: Theorem: Proof:

The above algorithm leads to globally optimal weights.

Consider the related constrained optimization problem:

Maximize log(L) =

N

n=1

 log 

K+1

 BI(n),j (n)vj  ,

(2.9)

j=1

subject to

vj ≥ 0,

K+1

vj = 1.

(2.10)

j=1

This problem represents the maximization of a concave function over a convex region. Hence, it has a unique optimum. Assume the contrary to what we are to prove; that is, assume that the original unconstrained optimization of equation 2.7 has two local maxima, w(1) = (w1 (1), . . . , wK+1 (1))T and w(2) = (w1 (2), . . . , wK+1 (2))T . On the line connecting the two maxima, there is a point w with objective function log(L(w)) < log(L(w(l)), l = 1, 2. Transform this line to v-space through the transformation equation 2.3. We will get equation 2.9 with the constraints 2.10 satisfied. Since for this problem, the maximum is unique, then all the points on the line connecting w(1) and w(2) achieve the same objective function log(L), contradicting the assumption of more than one local maximum. It is to be noted, however, that the solution in w space is not unique, as there are (one-dimensional) curves that achieve the same (global) optimum objective function. The theorem shows that the algorithm is guaranteed to reach the global optimum, that is, the best solution. The standard posterior probability estimator of equation 2.1 is contained in the search space of the proposed algorithm (choose vi = 1/K, i = 1, . . . , K). Hence, the solution achieved by the algorithm is nessarily better than the standard estimator in terms of the likelihood of the given set of points. Since what we care about is performance in expectation rather than finite-sample performance, the above argument is invoked for the large sample case. In the asymptotic case of large N, the sample mean of the likelihood function log(L)/N will get very close to the expectation of the likelihood. Hence, the achieved solution will then be superior to the standard equal-weighted estimator in terms of expected likelihood (for large enough N). (It is also well known that under

736

A. Atiya

certain conditions, maximizing the likelihood function asymptotically leads to an efficient estimator, that is, an estimator with the smallest variance (see Scholkopf ¨ & Smola, 2002), but we will not get into the technical details of the asymptotic performance.) If the procedure is intended for the purpose of estimating a more accurate estimate of P(correct), as detailed in section 1, then we have to note that there will be a positive bias in the estimate. This bias can be reduced to almost zero using a leave-one-out type procedure, where the pattern x(n) for which P(correct|x(n)) is being sought is excluded from the optimization procedure. This is at the expense of having to repeat the optimization N times. But this is not a problem, because it takes typically around 300 to 400 iterations to converge from a random point. In this case, a very close initial guess is available (using a solution whereby a different point is eliminated). So the optimization procedure will be much faster. Most distributions encountered in practice possess smoothness properties that make the far-away neighbors less relevant than nearby ones. It is expected that wj decreases with j. It is, however, a good idea to enforce the constraints: w1 ≥ w2 ≥ w3 ≥ · · · ≥ wK .

(2.11)

We are utilizing this extra knowledge to prevent randomness from leading to an unreasonable solution. The achieved solutions leads to the following interpretation, which gives some insight into the optimal solution. At the optimum, the gradient equals zero, leading to N

n=1


K+1 BI(n),i (n)

j=1

N


K+1 j=1

ewj

BI(n),j (n)ewj

= 1.

(2.12)

Substituting from equation 2.5, we get N BI(n),i (n) 1

=1 ˆ N n=1 P(C(n)|x(n))

(2.13)

N 1 1

= , ˆ Ni n∈S P(C(n)|x(n)) Ni i

(2.14)

or

where Si is the set of training patterns that are of the same class as their ith neighbor and Ni is the size of set Si . One observation is that at the optimum, the harmonic mean of the posterior probabilities over the set Si equals the fraction Ni /N, which represents the fraction by which the ith neighbor in

Estimating the Posterior Probabilities

737

question agrees with the true class identity. This agrees with the main intuition that neighbors that agree a lot in class membership to the pattern in question should have larger weights, and that leads to larger posterior probability estimates. 3 Implementation Examples We implemented the proposed method on two examples. Example 1. We consider a three-dimensional, two-class gaussian problem, with the mean vectors for the two classes being µ1 = (0, 0, 0)T

µ2 = (2, 2, 2)T ,

(3.1)

and the covariance matrices for the two classes are

1 = 2 = I,

(3.2)

where I represents the identity matrix. We generated 1000 training points from both classes. After training with these points, we tested the accuracy of the probability estimates with a test set of size 1000. We considered the cases of K = 50, 100, 150 neighbors. The proposed algorithm is implemented with the constraints on the monotonicity of the weights as described in the previous section. We observed that the algorithm converges quite quickly (typically 400 iterations are sufficient). To assess the merits of the method, we have compared it to the standard estimate in equation 2.1, which is equivalent to taking wj = 1/K, j = 1, . . . , K, wK+1 = 0. Since we know the true posterior probabilities, we can simply compute the distance between the estimated posterior probabilities and the true probability for both the proposed algorithm and the benchmark. We chose the Kullback divergence measure as a distance function. It is defined as  M N

P (Cm |x(n)) ˆ = 1 d(P, P) P (Cm |x(n))) log , N n=1 m=1 Pˆ (Cm |x(n))

(3.3)

where Pˆ denotes the estimate, whether it is the new method or the benchmark. To average out the effect of the randomness of the generated sample, we repeated the experiment five times with different sampling points. Table 1 shows the average results of both distance measures for the proposed method, as well as the benchmark, for the different tested K’s. One can see that the proposed method achieves better performance for all tested K. Figure 1 plots the weights of one of the runs with K = 50 in the v-space (i.e., vi as in equation 2.3).

738

A. Atiya

Nearest neighbor weights (in v−space) as a function of k 0.05

weight vi

0.04

0.03

0.02

0.01

0 0

10

20 30 40 neighbor number (k)

50

60

Figure 1: Final weights in v-space (after exponentiation and normalization, i.e., vi as in equation 2.3).

Example 2. We consider a two-dimensional two-class problem problem with mixture-of-gaussian class conditional densities. Specifically, they are given by  1 1 − 1 (x2 +x2 ) 1 − 1 ((x1 −4)2 +x2 ) 2 p(x|C1 ) = e 2 1 2 + e 2 3 2π 2π 1 − 1 ((x1 −2)2 +(x2 +2)2 ) (3.4) + e 2 2π

p(x|C2 ) =

 1 1 − 1 (x2 +(x2 −4)2 ) 1 − 1 ((x1 −2)2 +(x2 −2)2 ) + e 2 1 e 2 3 2π 2π 1 − 1 ((x1 −4)2 +(x2 −4)2 ) . + e 2 2π

(3.5)

We generated 600 training patterns from both classes and also generated the same number of test patterns. Again, we used the version of the algorithm with monotonicity constraints. We considered the case of 10, 20, 30, and 40 neighbors, and also repeated the experiments five times with different sampling point generations. Table 1 shows the results of average Kullback distance for the proposed method as well as the benchmark. One can also see the outperformance of the proposed method.

Estimating the Posterior Probabilities

739

Table 1: Comparison Between the New Method and the Benchmark EqualWeight Estimate for Examples 1 and 2. Example Number

Equal Weight

New Method

Example 1, K = 50 Example 1, K = 100 Example 1, K = 150 Example 2, K = 10 Example 2, K = 20 Example 2, K = 30 Example 2, K = 40

0.0214 0.0136 0.0172 0.1304 0.0261 0.0302 0.0330

0.0081 0.0102 0.0123 0.0282 0.0163 0.0164 0.0136

For both problems, we chose the range of K as that which gives good performance for both methods. Outside the chosen ranges, performance deteriorates appreciably.

4 Conclusion A method has been proposed that enhances the estimate of the posterior probabilities for pattern classification. It is based on observing the K-nearestneighbors and weighting their contribution to the posterior probabilities differently. The weights are estimated using the maximum likelihood procedure. The simulation results indicate that the method is a more accurate estimator than the benchmark K-nearest-neighbor method. References Atiya, A. F. (2001). Bankruptcy prediction for credit risk using neural networks: A survey and new results. IEEE Transactions on Neural Networks, 12(4), 929– 935. Bailey, T., & Jain, A. (1978). A note on distance-weighted K-nearest neighbor rules. IEEE Trans. Systems, Man, Cybernetics, 8, 311–313. Bermejo, S., & Cabestany, J. (2000). Adaptive soft K-nearest-neighbour classifiers. Pattern Recognition, 33, 1999–2005. Devroye, L., Gyorfi, ¨ L., & Lugosi, G. (1996). A probabilistic theory of pattern recognition. Berlin: Springer-Verlag. Dudani, S. (1976). The distance weighted K-nearest-neighbor rule. IEEE Trans. Systems, Man, Cybernetics, 6, 325–327. Fukunaga, K., & Hostetler, L. (1975). K-nearest-neighbor Bayes-risk estimation. IEEE Trans. Information Theory, 21(3), 285–293. Fukunaga, K., & Kessell, D. (1971). Estimation of classification error. IEEE Trans. Computers, 20, 1521–1527.

740

A. Atiya

Fukunaga, K., & Kessell, D. (1973). Nonparametric Bayes error estimation using unclassified samples. IEEE Trans. Information Theory, 19, 434–439. Glick, N. (1978). Additive estimators for probabilities of correct classification. Pattern Recognition, 10, 211–222. Guerrero-Curieses, A., Cid-Sueiro, J., Alaiz-Rodriguez, R., & Figueiras-Vidal, A. (2004). Local estimation of posterior class probabilities to minimize classification errors. IEEE Trans. Neural Networks, 15(2), 309–317. Hand, D. (1986). Recent advances in error rate estimation. Pattern Recognition Letters, 4, 335–346. Jain, A., Duin, R., & Mao, J. (2000). Statistical pattern recognition: A review. IEEE Trans. Pattern Analysis, Machine Intelligence, 22(1), 4–37. Kanal, L. (1974). Patterns in pattern recognition: 1968–1974. IEEE Trans. Information Theory, 20(6), 697–722. Kittler, J., & Devijver, P. (1981). An effient estimator of pattern recognition system error probability. Pattern Recognition, 13, 245–249. Kroke, J. (1986). The robust estimation of classification error rates. Comput., Math., Appl., 12A, 253–260. Kulkarni, S., Lugosi, G., & Venkatesh, S. (1998). Learning pattern classification— a survey. IEEE Trans. Information Theory, 44(6), 2178–2206. Lugosi, G., & Pawlak, M. (1994). On the posterior-probability estimate of the error rate of nonparametric classification rules. IEEE Trans. Information Theory, 40(2), 475–481. Royall, R. (1966). A class of nonparametric estimators of a smooth regression function. Unpublished doctoral dissertation, Stanford University. Scholkopf, ¨ B., & Smola, A. (2002). Learning with kernels. Cambridge, MA: MIT Press. Toussaint, G. (2002). Proximity graphs for nearest neighbor decision rules: Recent progress. In Proc. 34th Symp. Computing and Statistics (INTERFACE-2002). Montreal, Canada. Available at http://wwwcgrl.cs.mcgill.ca/∼godfried/research/nearest.neighbor.html. Received April 2, 2004; accepted July 14, 2004.

REVIEW

Communicated by Laurence Abbott

Quantifying Stimulus Discriminability: A Comparison of Information Theory and Ideal Observer Analysis Eric E. Thomson [email protected]

William B. Kristan [email protected] University of California, San Diego, La Jolla, CA 92093-0357, U.S.A.

Performance in sensory discrimination tasks is commonly quantified using either information theory or ideal observer analysis. These two quantitative frameworks are often assumed to be equivalent. For example, higher mutual information is said to correspond to improved performance of an ideal observer in a stimulus estimation task. To the contrary, drawing on and extending previous results, we show that five information-theoretic quantities (entropy, response-conditional entropy, specific information, equivocation, and mutual information) violate this assumption. More positively, we show how these information measures can be used to calculate upper and lower bounds on ideal observer performance, and vice versa. The results show that the mathematical resources of ideal observer analysis are preferable to information theory for evaluating performance in a stimulus discrimination task. We also discuss the applicability of information theory to questions that ideal observer analysis cannot address.

1 Introduction Many adaptive behaviors require that an organism respond differently to different stimuli, that is, discriminate among stimuli. Familiar examples include the frog’s ability to discriminate prey location (Lettvin, Maturana, McCulloch, & Pitts, 1959) and the monkey’s ability to indicate the net direction of motion in a field of randomly flickering dots (Britten, Shadlen, Newsome, & Movshon, 1992). What properties of sensory neurons mediate such abilities? Do different stimuli reliably produce spike trains with different temporal patterns, or is variability in spike timing merely noise that should be averaged away to extract the underlying signal? Such questions have generated heated and often productive discussions for at least 50 years (MacKay & McCulluch, 1952; for reviews, see Perkel & Bullock, 1968; Rieke, Warland, de Ruyter van Steveninck, & Bialek, 1997; Victor, 1999). Neural Computation 17, 741–778 (2005)

© 2005 Massachusetts Institute of Technology

742

E. Thomson and W. Kristan

Before evaluating performance in a discrimination task, one must first decide what measure to use. While ideal observer performance has historically been used to measure stimulus discrimination (Green & Swets, 1966; Geisler, 2003), information measures are often used with the same goal (Alkasab et al., 1999; Arabzadeh, Panzeri, & Diamond, 2004; Buracas & Albright, 1999; Li, Piech, & Gilbert, 2004; Paz & Vaadia, 2004; Petersen, Panzeri, & Diamond, 2002; Pola, Thiele, Hoffmann, & Panzeri, 2003; Theunnissen & Miller, 1991). The main goal of this article is to analyze the relationship between these two approaches. The relationship between information theory and ideal observer analysis has been the subject of several studies by statisticians and engineers (Wagner, 1965; Tebbe & Dwyer, 1968; Kovalevsky, 1968; Golic, 1987; Feder & Merhav, 1994), providing an untapped vein of research relevant for neuroscience. We review and extend the results of this previous research, showing that information theory and ideal observer analysis are not equivalent and that ideal observer analysis is more appropriate for quantifying stimulus discriminability. We also describe how to derive upper and lower bounds on ideal observer performance as a function of the information measures. Because one of our goals is to determine when it is more appropriate to use information theory or ideal observer analysis to answer a particular research question, we devote significant attention to the interpretation of the quantities typically encountered in the two approaches. Most of the article assumes only knowledge of basic probability theory (e.g., Yates & Goodman, 1999); we introduce and define all key terms from ideal observer analysis and information theory. We place proofs that would interrupt the flow of the paper in footnotes.

2 Preliminary Assumptions and Definitions Initially, we limit our analysis to experiments in which on each trial an organism is presented with one of M stimuli from a discrete set S = {s1 , . . . , s M }. The probability that a particular stimulus will be presented on a given trial is represented by the probability distribution P(S) = {P(s1 ), . . . , P(s M )}, which we assume does not change with time. We assume that each stimulus evokes a response from an N-element set R = {r1 , . . . , r N }. Typically, N
M. In section 5, we extend the results to the case in which S and R are continuous variables. Also, note that none of our conclusions rests on the assumption that S describes a set of stimuli and R describes a set of responses; the analysis applies to any pair of random variables, whether they describe sensory, neural, behavioral, or other states. The dependence of R on S is represented by a channel matrix P(R |S), which is an M by N matrix in which row i contains P(R |si ) (Ash, 1965):

Quantifying Stimulus Discriminability

743

Figure 1: Examples of the quantities used to analyze performance in a sensory discrimination task. (A) A two-element stimulus distribution P(S) (top) and a two-by-two channel matrix P(R | S) (bottom). (B) The corresponding joint distribution P(S,R), calculated from P(S) and P(R | S) in A using the fact from probability theory that P(si , r j ) = P(r j |si )P(si ).



P(r1 | s1 )  ..  P(R | S) =  . P(r1 | s M )

 · · · P(r N | s1 )  .. .. . . .  · · · P(r N | s M )

(2.1)

We assume that P(R |S) does not change with time.1 Note that P(R | S) is not a probability distribution. Rather, each row of P(R | S) is a conditional probability distribution P(R | si ) that sums to one. See Figure 1A for a numerical example. Given P(S) and P(R |S), the joint distribution P(S,R) can easily be calculated (see Figure 1B). Hence, if P(S) and P(R |S) are given, then it is possible to calculate the values of all functions of P(S,R) such as mutual information and ideal observer performance in a stimulus-estimation task. 3 Ideal Observers: Minimum Error Classifiers 3.1 Defining Ideal Observer Performance: P(c). An ideal observer of the neural response R is a minimum-error classifier. As shown in Figure 2, a classifier is a function C that maps the response variable R into Sˆ , where Sˆ is the M-element set {ˆs1 , . . . , sˆ M } that contains all possible estimates of which stimulus was presented. A classifier is correct on a given trial if sˆ = s, that is, if the estimate is identical to the actual stimulus. Similarly, a classifier is in error on those trials in which sˆ = s. A useful and widespread measure of classifier performance is the probability that it will provide a correct 1 That is, we assume the system implements a discrete memoryless channel (Cover & Thomas, 1991).

744

E. Thomson and W. Kristan

Figure 2: Cartoon representation of the steps in a classification task. First, a stimulus from S is selected that evokes a response from R . Then the classifier C estimates which stimulus produced the given response. Formally, a classifier is a function that, given a response r j as input, returns an estimate (ˆs ) of the stimulus that produced that response.

estimate of S (Duda, Hart, & Stork, 2001). By definition, an ideal observer is a classifier that maximizes the probability of being correct (Liu, Knill, & Kersten, 1995; Geisler, 1989, 2003; Knill & Kersten, 1991).2 This is equivalent to saying that an ideal observer minimizes the probability of error. Ideal observers follow a simple decision rule (Duda et al., 2001): given response r j , choose the estimate sˆi that corresponds to the stimulus with the maximum probability in the conditional distribution P(S | r j ). More formally, Given r j ∈ R, choose sˆi such that for all s ∈ S, P(S = si |r j) ≥ P(S = s|r j). (3.1) We denote the stimulus with the maximum probability, given r j , as

2 Albert Siegert, a physicist in the MIT Radiation Laboratory during World War II, introduced the term ideal observer to refer to minimum error classifiers. The term was first used in publication in a signal detection theory text in which the authors attribute the idea to Siegert (Lawson & Uhlenbeck, 1950).

Quantifying Stimulus Discriminability

745

Figure 3: Example of the steps followed to calculate P(c | r j ) and P(c), a continuation of the example from Figure 1. (A) The conditional distributions P(S | r j ) are calculated using P(R | S) and P(S) from Figure 1 and the fact that P(si | r j ) = P(s j , r j )/P(r j ). (B) The corresponding ideal observer decision scheme, constructed using Rule 3.1. (C) P(c | r j ) (and P(e | r j )) are calculated using equation 3.2 and the fact that P(e | r j ) = 1 − P(c | r j ). (D) P(c) and P(e) are calculated using equation 3.3 and the fact that P(e) = 1 − P(c).

smax( j) . Hence, Rule 3.1 simplifies to “Choose sˆmax( j) ” (see Figures 3A and 3B).3 If multiple elements of P(S | r j ) have the same maximum probability, then the ideal observer makes an arbitrary choice among those elements 1 1 1 1 (Duda et al., 2001). For instance, if P(S | r j ) = { 15 , 15 , 10 , 5 , 10 , 5 }, then the ideal observer can choose sˆ1 , sˆ2 , sˆ4 , or sˆ6 . Before describing how to calculate the probability that an ideal observer will correctly estimate S, we introduce notation for use in the rest of the article. Let C = {c, e}, where c is the event that a classifier is correct and e is the event that a classifier is in error. P(C) is the probability distribution of C (i.e., P(C) = {P(c), P(e) = 1 − P(c)}), and P(C | r j ) is the distribution of C when response r j is observed. The probability that an ideal observer will correctly estimate S if response r j occurs is (Duda et al., 2001): P(c |r j ) = P(S = smax( j) |r j ) =

P(smax( j) , r j ) . P(r j )

(3.2)

3 In the classification literature, ideal observers are usually called maximum a posteriori classifiers or Bayesian classifiers (Duda et al., 2001). The reason for this name is that smax( j) is the element with the maximum value in P(S | r j ), which in a Bayesian context is often called the a posteriori distribution of S. We use the term ideal observer because it is more prevalent in psychophysics and neuroscience.

746

E. Thomson and W. Kristan

The first equality in equation 3.2 asserts that P(c | r j ) is the conditional probability of the stimulus chosen by the ideal observer rule (Duda et al., 2001). See Figure 3C. This is because on trials in which r j occurs, the ideal observer will pick sˆmax( j) as its estimate of S, and by definition smax( j) will be the stimulus that actually evoked r j with frequency P(S = smax( j) |r j ). The second equality in equation 3.2 is an instance of Bayes’ theorem (Yates & Goodman, 1999). Note that P(smax( j) , r j ), the numerator on the right-hand side of equation 3.2, is the maximum element in the column that corresponds to response r j in the joint distribution P(S,R).4 P(c), the probability that an ideal observer will correctly estimate S, is the average (over R) of P(c |r j ) (Duda et al., 2001):

P(c) =

N  j= i

P(r j )P(c |r j ) =

N 

P(smax( j) , r j ).

(3.3)

j= i

The second equality in equation 3.3 follows from equation 3.2. Equation 3.3 shows that P(c) is just the sum of the maximum from each column of the joint distribution P(S,R). Further, since the N maximal elements from the columns of P(S,R) sum to P(c), the N · (M − 1) nonmaximal elements must sum to P(e). See Figure 3D for an example. In the rest of the article, when we use P(c) or P(c | r j ), we are specifically referring to the probability that an ideal observer will correctly estimate S, probabilities that can be calculated using equations 3.2 and 3.3. Similarly, P(e) and P(e | r j ) will be used to refer to the probability that an ideal observer will make an error. It follows from equation 3.3 that for a given value of M, P(c) must lie 1 between M and 1 (inclusive). The upper bound (P(c) = 1) corresponds to the best performance possible by an ideal observer of R, and equation 3.3 implies that this perfect performance can be achieved only when each column of P(S,R) has a single nonzero element. The worst ideal observer performance possible is P(c) = P(smax ), the probability of the most likely

Proof: P(si , r j ) = P(si |r j )P(r j ), and since P(r j ) is the same for all elements in the jth column of P(S,R), the maximum conditional probability P(smax( j) |r j ) must also pick out the row with the maximum joint probability in column j. 4

Quantifying Stimulus Discriminability

747

stimulus (Duda et al., 2001).5 Since the smallest possible value of P(smax ) is 1 6 1 , the lowest possible P(c) for any distribution over S is M . For example, M consider the special case in which R is independent of S (i.e., P(S | r j ) = P(S)). In such a case, the ideal observer’s estimate sˆ is simply the most likely stimulus sˆmax . This stimulus occurs with probabilityP(smax ) = P(c), which can 1 1 1 vary between M when the stimuli are equiprobable (i.e.,P(S) = { M ,..., M }) and 1.0 when only a single stimulus is presented (i.e., P(S) = {0, . . . , 1, . . . , 0}). 3.2 Interpreting P(c). By definition, we say the response variable R is a good discriminator of S if different stimuli tend to lead to different responses. In what sense does P(c) quantify this notion? Figure 4 shows three arbitrary gaussian-shaped conditional distributions P(R | si ), each weighted by the probability of the corresponding stimulus P(si ). Note that each such response ensemble is a row of the joint distribution P(S,R). As discussed in section 3.1, because P(c) is the sum of the maxima from the N columns of P(S,R), it follows that P(e) is the sum of the nonmaximal elements from the columns. In Figure 4, these nonmaximal elements of P(S,R) are shaded in gray, illustrating that P(e) is the area of overlap among the M stimulusdependent ensembles. On the whole, then, ideal observer performance is a useful measure of how well a response R discriminates among different stimuli. If the response ensembles corresponding to different stimuli are so segregated that there is very little overlap among them (i.e., low P(e) and high P(c)), then different stimuli typically lead to different responses, the defining feature of discriminability. Conversely, if the ensembles show a good deal of overlap (i.e., high P(e) and low P(c)), then different stimuli often evoke the same response and R is a bad discriminator of S. 4 Bounding Information-Theoretic Quantities with Ideal Observers In this section, we quantitatively compare ideal observer analysis and information theory. In particular, we address the claim that they are 5 In Duda et al. (2001) this property of classifiers is mentioned but not proven, so we prove it here. Proof: Let row k of P(S,R) be the row whose marginal probability is  P(smax ), that is, P(sk ) = P(smax ) = N j= i P(sk , r j ). For each column j, we know that P(smax( j) , r j ) ≥ P(sk , r j ). Summing the terms in this inequality over R yields:

P(c) =

N  j=1

P(smax( j) , r j ) ≥

N 

P(sk , r j ) = P(smax ).

j=1

This lower bound on P(c) is attained when P(smax( j) , r j ) = P(sk , r j ) for each column of P(S,R). 6 Proof: If the maximum of an M element set were less than 1 , then the elements could M not sum to 1.

748

E. Thomson and W. Kristan

Figure 4: Graphical representation of the meaning of P(c) and P(e). The area of overlap among the weighted stimulus-conditional distributions (the grayshaded bars) is identical to P(e). The area of the unfilled bars sums to P(c).

interchangeable theoretical methods that can be used to quantify stimulus discriminability. While most neuroscientists are interested in mutual information, we begin with entropy because it is useful for building intuitions for the more complicated cases and because most of the results generalize to the other information measures. 4.1 Entropy and the Ideal Observer 4.1.1 Three Interpretations of Entropy. If P(S) is an M-element probability distribution, then the entropy of S is defined as (Cover & Thomas, 1991): H(S) = −

M 

P(si ) log2 P(si ).

(4.1)

i=1

Because we take the logarithm to base two, all information measures are in units of bits, though in the rest of the review, we suppress the subscript. Qualitatively, H(S) is usually described as a measure of our uncertainty about S (Ash, 1965), but how should this be interpreted? We discuss three interpretations of H(S). First, H(S) measures how evenly the probability mass is spread among the M elements of S (Cover & Thomas, 1991). At one extreme, if all the probability mass in P(S) is concentrated on one outcome (P(S) = {0, . . . , 1, . . . , 0}), then H(S) is minimized and is zero bits. At the other extreme, if each outcome

Quantifying Stimulus Discriminability

749

Figure 5: Examples of two possible codebooks that encode the value of S (M = 4). The stimulus value si is in the first column of each table, the corresponding codeword wi is in the second column, and the third column shows
i , the corresponding codeword length. The average codeword length, L , is calculated below each table using equation 4.2 under the assumption of equiprobable stimuli. 1 1 is equally probable (P(S) = { M ,..., M }), then the mass is evenly spread 1 1 (P(S) = { M , . . . , M }) and H(S) is maximized at log(M) bits. A second interpretation of H(S) is that it provides lower bounds on the number of binary digits (bits) required to encode S. This interpretation is based on the source coding theorem (see equation 4.3), a central result from coding theory. Before stating the theorem, we briefly review terminology from coding theory. To encode random variable S is to build a codebook, which assigns to each element si of S a unique codeword wi . Each codeword is a sequence of symbols from a set of elementary symbols called the alphabet. The number of symbols in codeword wi is the codeword length, which we denote
i . The codeword length
i can be considered a measure of the cost incurred by encoding stimulus si . The average codeword length, denoted L, measures the average (over S) cost when a particular codebook is used (Cover & Thomas, 1991): M  L= P(si )
i . (4.2) i=1

L is useful for comparing the cost of using different codebooks. Figure 5 provides examples to illustrate these concepts. It shows two of the infinite number of possible codebooks if S is a set of four stimuli. The alphabet in this example is the set {0, 1} of binary digits, so the codebooks assign to each element si of S a codeword wi consisting of a sequence of binary digits. The corresponding codeword lengths
i are shown in the third column of the tables. If we assume the four stimuli are equiprobable, then LA, the average codeword length for codebook A, is 2 bits, and L B is 3 12 bits. Hence, the average cost of using codebook A is less than the average cost of using codebook B.

750

E. Thomson and W. Kristan

The source coding theorem (also known as the noiseless coding theorem; Ash, 1965) provides the basis for the second interpretation of H(S). The theorem is (Cover & Thomas, 1991) H(S) ≤ L min < H(S) + 1,

(4.3)

where L min is the minimum possible average codeword length required to encode random variable S. The theorem shows that as H(S) increases, more bits are required to encode S. The theorem also provides an absolute lower bound on L that can be used to evaluate any individual codebook. For example, it follows from equation 4.3 that codebook A in Figure 5 is in the set of best possible codebooks for S because it actually reaches the lower bound LA = H(S). Also, because L B > H(S) + 1, it follows that there exists a better codebook than codebook B. Note that in both codebooks, each stimulus is assigned a different codeword, so if there existed a channel in which the response to each stimulus was the corresponding codeword, P(c) would be 1.0. Thus, the source coding theorem tells you the fewest number of bits required to encode a variable so that it can be decoded without error. A third interpretation of H(S), inspired by the description of H(S) as a measure of uncertainty about S, is that it is a measure of the difficulty an ideal observer would have estimating S. The flaws in this interpretation are discussed in the next section. 4.1.2 Comparing Entropy and Ideal Observers. H(S) is a function only of P(S). Hence, to directly compare H(S) and P(c), we consider the behavior of an ideal observer faced with the task of estimating S given only P(S). Given P(S), the ideal observer picks the most likely stimulus sˆmax as its estimate of S, and P(c) =P(smax ) (Duda et al., 2001). H(S) is sometimes interpreted as a measure of the difficulty in estimating S on a single trial: the higher the entropy, the less likely that an ideal observer will correctly estimate S (Alkasab et al., 1999). Formally: H(S1 ) ≥ H(S2 ) ⇔ P(c)1 ≤ P(c)2 ,

(4.4)

where S1 and S2 are two random variables and P(c)i is ideal observer performance in estimating Si on the basis of P(Si ). That equation 4.4 is 65 18 17 incorrect can be shown by counterexample. Let P(S1 ) = { 100 , 100 , 100 } and 50 49 1 P(S2 ) = { 100 , 100 , 100 }. In this case, H(S1 ) = 1.28 > 1.07 = H(S2 ) and P(c)1 = 0.65 > 0.50 = P(c)2 . The rest of this section provides a more general analysis of the relationship between P(c) and H(S). We first review previous results that show how to calculate the range of H(S) values that are consistent with a given value of P(c) (Tebbe & Dwyer, 1968; Kovalevsky, 1968). Toward this end, we define MP(c) as the set of all M-element probability distributions for which ideal observer performance is P(c). Since entropy is invariant under permutations

Quantifying Stimulus Discriminability

751

Figure 6: Some members of the set MP(c) = 40.5 . The empty circle indicates Pmin (40.5 ), the distribution in 40.5 with the minimum entropy (H = 1 bit). The filled circle indicates Pmax (40.5 ), the distribution with the maximum entropy (H = 1.8 bits).

of elements of P(S) we can, without loss of generality, sort the probability values in each distribution in MP(c) in descending order (Feder & Merhav, 1994). That is, by construction, all distributions in MP(c) satisfy P(c) = P(s1 ) ≥ P(s2 ) ≥ · · · ≥ P(s M ).

(4.5)

Figure 6 shows an example of MP(c) for the case in which P(c) = 0.5 and M = 4. Each distribution in MP(c) = 40.5 has the same maximal element P(s1 ) = P(smax ) = 0.5, but the probability mass in the remaining three elements can arbitrarily vary as long as the constraints provided by equation 4.5 are satisfied. H(S) is sensitive to this variability, but P(c) is not. In fact, the distributions in MP(c) take on a range of H(S) values. We denote the distribution in MP(c) that has the minimum entropy Pmin (MP(c) ) and the distribution with the maximum entropy Pmax (MP(c) ). We denote the corresponding entropy values h min (MP(c) ) and h max (MP(c) ), respectively. Previous papers (Tebbe & Dwyer, 1968; Kovalevsky, 1968) show that Pmax (MP(c) ) and h max (MP(c) ) are

P(e) P(e) Pmax (MP(c) ) = P(c), ,..., M−1 M−1

(4.6)

h max (MP(c) ) = H(C) + P(e)log(M − 1),

(4.7)

where C is the random variable with outcomes {c, e} defined in section 3.1. Equation 4.6 holds because once P(c) is fixed, the way to maximize entropy is to distribute the residual P (e ) = [1 − P(c)] probability mass evenly to the remaining M − 1 elements of the distribution. Equation 4.7 follows when equation 4.6 is substituted into equation 4.2. In Figure 6, Pmax (40.5 ) is indicated with a filled circle, and this distribution has entropy h max (40.5 ) = 1.8 bits. In Figure 7A, h max (MP(c) ) is plotted as a function of P(c) for M = 2,

752

E. Thomson and W. Kristan A

(1)

1

A 2,h

H( )/H( |rj) [bits]

0.5

M=2 0 2

k=3

(2)

A4,h

k=2

1

M=4

k=1

0 6

(3) A 64,h

4 2 0

M=64 0

0.4

0.2

0.6

0.8

1

P(c)/P(c|rj) B

H=3 bits

0.4 0

PC

g

(M, H)

0.6

0.2 0

H=0.5 bits 0

2 8

25

50

100

M Figure 7: Comparison of H(S) and P(c). (A) Plot of the upper and lower bounds on H(S) as a function of P(c) for M = 2, M = 4, and M = 64 (panels 1, 2, and 3, respectively). The upper bound, h max (MP(c) ), is indicated by a dashed line, and the lower bound, h min (MP(c) ), is indicated by a solid line. The sets of allowable {P(c), H(S)} points are labeled A M,h . Note that A2,h (panel 1) is a line. In panel 2, the arch-shaped line segments corresponding to different values of k are indicated, as are the points corresponding to h max (40.5 ) and h min (40.5 ) from Figure 6 (filled and empty circles, respectively). (Recall that the integer k describes the minimum number of stimuli that must have probability P(c) when the goal is to minimize H(S); see the text). In panel 3, the lower (0.14) and upper (0.69) bounds on P(c) when H(S) = 3 bits are indicated with filled circles and connected with a dashed line. As discussed in section 4.2, the graphs also plot the relationship between P(c|r j ) and H(S |r j ), so these quantities are included on the x- and y-axes, respectively. (B) Plot of PCrange (M,H) as a function of M shows that PCrange (M,H) increases exponentially with M. The filled circles show PCrange (M,3), and the unfilled circles plot PCrange (M,0.5). The lines are the best saturating exponential fit to the points (see text). The large filled circle on the PCrange (M,3) line indicates PCrange (64,3), the range delineated by the dashed line in panel 3 in Figure 7A.

Quantifying Stimulus Discriminability

753

4, and 64 (dotted lines). It can be seen that at a given value of H(S), the maximum possible P(c) value is specified by h max (MP(c) ), so h max (MP(c) ) provides an upper bound on P(c). Note that for a given P(c) value, h max (MP(c) ) increases with log(M − 1), the only term in equation 4.7 that depends on M. h max (MP(c) ) increases with M because for a larger M, there exist a greater number of elements across which the residual probability mass P(e) can be spread. At the other extreme, to minimize H(S), the residual probability mass P(e) must be concentrated as much as possible while satisfying the constraints in equation 4.5. It is as if there are M cups, each of which can hold P(c) liters of water, and the goal is to distribute a single liter of water to the cups while filling as few of the cups as possible. For a given P(c) value, there exists an integer k that indicates the minimum number of cups that must be completely filled when following such a strategy. For the case in which M = 4 and P(c) = 0.5 (see Figure 6), Pmin (40.5 ) is { 12 , 12 , 0, 0}, so k = 2 and no water is distributed to the third or fourth cups. Consider also the case in which M = 4 and P(c) = 0.4. In this case, Pmin (40.4 ) = {0.4, 0.4, 0.2, 0}, k is again two, but there remain 0.2 liters of water once the second cup is full, and this water must be distributed to the third cup (i.e., cup k + 1). Mathematically, this procedure of concentrating probability mass leads to the following equations for Pmin (MP(c) ) and h min (MP(c) ) (Tebbe & Dwyer, 1 1968; Kovalevsky, 1968). For each P(c) value between M and 1, there exists 1 an integer k between 1 and M − 1 such that k+1 ≤ P(c) ≤ k1 , and Pmin (MP(c) ) = {P(c)1 , . . . , P(c)k , 1 − k P(c), 0, . . . , 0}

(4.8)

h min (MP(c) ) = − [k P(c)log(P(c)) + (1 − k P(c)) log(1 − k P(c))].

(4.9)

As can be seen in equation 4.8, k corresponds to the case in which k out of the M stimuli are assigned a probability of P(c). The remaining [1 − k P(c)] probability mass is assigned to stimulus k + 1, and stimuli k + 2 through M are assigned probabilities of zero. Figure 7A plots h min (MP(c) ) as a function of P(c) for M = 2, 4, and 64 (solid lines). Each arch-shaped line segment in Figure 7A corresponds to a different value of k (in Figure 7A.2 the line segments are labeled with their corresponding k values). It can be seen in the figure that h min (MP(c) ) provides a lower bound on P(c). That is, for a given value of H(S), the lowest possible value of P(c) is given by equation 4.9. Note that h min (MP(c) ) does not depend on M. Increasing M, the number of possible stimuli, does not affect the outcome when the goal is to concentrate the probability mass to those stimuli as much as possible. For instance, if P(c) is 0.5, then no matter how large M is, only the first two stimuli are assigned nonzero probabilities (e.g., P(60.5 ) = { 12 , 12 , 0, 0, 0, 0}). The upper and lower bounds h min (MP(c) ) and h max (MP(c) ) circumscribe the set A M,h of all possible points {P(c), H(S)} (see Figure 7). We note four general features of A M,h . First, its upper and lower bounds, h max (MP(c) ) and

754

E. Thomson and W. Kristan

h min (MP(c) ), are tight (Feder & Merhav, 1994). That is, for a given M and P(c), there exists a distribution P(S) that actually equals h max (MP(c) ) and h min (MP(c) ). These distributions are given by equations 4.6 and 4.8, respectively. Second, for a given value of P(c), the range of possible H(S) values increases with log(M − 1). For the case in which there are only two stimuli (M = 2), H(S) is uniquely specified by P(c). This is because once P(s1 ) is fixed, there is no freedom to vary the remaining probability mass: P(s2 ) must equal 1 − P(s1 ). As M increases, h max (MP(c) ) increases with log(M − 1) without bound. This is because the only term in equations 4.7 and 4.9 that depends on M is log(M − 1), which is in the equation for h max (MP(c) ). Third, while equation 4.4 is incorrect (i.e., higher entropy does not imply a decrement in ideal observer performance), it is possible to infer the range of P(c) values consistent with a given value of H(S). That is, given H(S), it is possible to calculate upper and lower bounds on P(c). This is because both h max (MP(c) ) and h min (MP(c) ) are one-to-one, strictly monotonic functions of P(c) (Feder & Merhav, 1994). Hence, both h max (MP(c) ) and h min (MP(c) ) have inverses that provide upper and lower bounds of P(c), respectively. While closed-form analytical solutions for the inverses do not exist, the bounds can be numerically estimated with accuracy limited only by machine precision.7 For example, numerical methods applied with M = 64 and H(S) = 3 bits show that P(c) can vary between 0.14 and 0.69, bounds marked with filled circles in Figure 7A, panel 3.8 In this case the range of P(c) values, defined as the difference between the maximum and minimum P(c) value, is 0.55. Fourth, the range of P(c) values consistent with a given H(S) is a saturating exponential function of M. If we define the maximum and minimum P(c) values consistent with a given H(S) as PCmax (M, H) and PCmin (M, H), respectively, then the range of P(c) values at a given entropy is given by PCrange (M, H) = PCmax (M, H) − PCmin (M, H). Figure 7B plots PCrange (M, 0.5) and PCrange (M, 3.0) as functions of M (filled and open circles, respectively). As can be seen in the figure, PCrange (M, H) is a saturating exponential function of M that saturates at 1 − PCmin (M, H). We expected this saturating exponential based on the following two considerations. First, the range of H(S) values at a given P(c) increases logarithmically with M (see above), and this is due to the fact that h max (MP(c) ) increases logarithmically

7 Equations 4.7 and 4.9 are of the form y = x + log(x), for which there exists no analytical solution for the inverse. 8 We carried out the calculation of the lower bound on P(c) as follows. A set of h max (MP(c) ) values was obtained over the range P(c) = 14 to 1 using equations 4.7 and 4.9. Then a cubic spline was fit to these numbers using P(c) as the dependent variable (de Boor, 1978). Then, given any value H(S), P(c)max can be estimated using the spline. A similar method was used to calculate the upper bound on P(c).

Quantifying Stimulus Discriminability

755

with M. Because the inverse of the log is an exponential function, and PCmax (M, H) grows with the inverse of h min (MP(c) ), PCrange (M, H) should grow exponentially with M. However, since P(c) can never be greater than 1.0, PCrange (M, H) must also be bounded from above by 1 − PCmin (M, H), so we also expected the curve to saturate at that value. While there do not exist analytical solutions for PCrange (M, H) (see the previous paragraph), the points are indeed well fit by a saturating exponential function constrained to have a maximum of 1 − PCmin (M, H)(see Figure 7B).9 4.1.3 Multiple Question Ideal Observers and Entropy. If an ideal observer makes an error on the first guess and is given another chance to guess S, then it will pick the stimulus with the maximum probability in the conditional distribution P(S |r j , S = smax( j) ). In that case, the analysis from the previous section applies, though with the number of stimuli effectively reduced to M − 1. More generally, after G guesses, the stimulus set is effectively reduced to M − G stimuli, where G can vary between zero and M − 1. In contrast, if the goal is to guess the value of a random variable using the fewest number of yes-or-no questions (i.e., the game of 20 Questions; Cover & Thomas, 1991), then the questions that receive yes and no answers can be represented by ones and zeros, respectively. Then, by the source coding theorem, H(S) describes the best possible performance L min (Cover & Thomas, 1991). In 20 Questions, the optimal question sequence will cut down the range of possible outcomes to quickly specify the stimulus. For instance, if there are eight equiprobable stimuli, then the initial question should be of the form, “Is it above four?” The ideal observer, on the other hand, must always guess a single outcome, that is, the ideal observer must always ask questions of the form, “Was it an 8?” 4.2 Response-Conditional Entropy and the Ideal Observer. The entropy of S, given that r j is observed, is known as the response-conditional entropy H(S |r j ), which is defined as (Cover & Thomas, 1991) H(S |r j ) = −

M 

P(si |r j ) log(P(si |r j )).

(4.10)

i=1

Note that equation 4.10 is simply equation 4.2 with the conditional distribution P(S |r j ) substituted for P(S). Hence, all the properties of H(S) (see section 4.1.1) extend to H(S |r j ). For instance, in addition to measuring how evenly spread the conditional distribution P(S |r j ) is, H(S |r j ) provides bounds on the minimum average number of binary digits required to encode S once r j is known. The lines are the least-squares fits to saturating exponentials of the form (1 − M−T β PCmin (M))(1 − e −( α ) ) + PCra nge (T), where T is the smallest value of M for which the given H(S) is defined and α/β are free parameters. 9

756

E. Thomson and W. Kristan

Since H(S |r j ) is a function of P(S |r j ), we compare H(S |r j ) to the performance of an ideal observer of r j (i.e., P(c |r j )), which is also a function of P(S |r j ). Recall that an ideal observer of r j will pick the most likely stimulus from the conditional distribution P(S |r j ) (Rule 1). Because H(S |r j ) is mathematically equivalent to H(S) (i.e., H(S |r j ) is a function of an ordinary M-element probability distribution, and the ideal observer picks the stimulus with the maximum probability from this distribution), H(S |r j ) has the exact same properties with respect to P(c |r j ) that H(S) has with respect to P(c) (see section 4.1.2). For example, for a given P(c |r j ), there is a range of possible H(S |r j ) values bounded by h max (MP(c | r j ) ) = H(C |r j ) + P(e |r j ) log(M − 1)

(4.7 )

h min (MP(c | r j ) ) = −[k P(c |r j ) log P(c |r j ) + (1 − k P(c |r j )) log(1 − k P(c |r j )).

(4.9 )

We label equations 4.7 and 4.9 because they are simply equations 4.7 and 4.9 with P(c) replaced by P(c |r j ) and MP(c) replaced by MP(c | r j ) (see Figure 7). Also note that the set of possible {P(c |r j ), H(S |r j )} points is identical to the set AM,h of possible points {P(c), H(S)} discussed in section 4.1 (see Figure 7). We use the label AM,h for both sets of points, letting the context make clear whether we are referring to entropy or response-conditional entropy. 4.3 Specific Information and the Ideal Observer. How much information does a specific response r j provide about the stimulus set S? In information theory, this is quantified by the specific information between S and r j (DeWeese & Meister, 1999). Specific information is formally defined as I (S, r j ) = H(S) − H(S |r j ) = −

M 

P(si ) log(P(si ))

i=1

+

M 

P(si |r j ) log(P(si |r j )).

(4.11)

i=1

Since H(S) is the original entropy of S, and H(S |r j ) is the entropy of S remaining after r j is observed, it follows that I (S, r j ) is a measure of how much entropy about S is removed upon observation of response r j . (For examples, see Figure 8A). Interestingly, I (S, r j ) can be negative (DeWeese & Meister, 1999), which occurs when P(S |r j ) is more evenly spread than P(S) (e.g., rows 1 and 2 in Figure 8A). Conversely, I (S, r j ) is positive when P(S |r j ) is more concentrated than P(S) (e.g., rows 3 and 4 in Figure 8A). Interpreted in terms of coding theory (see section 4.1.1), I (S, r j ) indicates how many fewer binary digits, on average, are needed to encode S once r j is known. This number is negative when more digits are required to encode S once r j is observed.

Quantifying Stimulus Discriminability

757

Figure 8: Examples that compare P(c) and specific information. (A) Each row of the table contains a conditional distribution P(S |r j ). Columns 2–5 contain the corresponding ideal observer performance P(c |r j ), response-conditional entropy H(S |r j ), and specific information I (S, r j ), respectively. The calculation of I (S, r j ) assumes that P(S) is as given with an entropy H(S) of 1.25 bits. (B) Scatter plot of the four {P(c |r j ), I (S, r j )} pairs from the rows in the table in A. The point from row j of the table in A is denoted P I j . The absolute upper and lower bounds on I (S, r j ) as a function of P(c |r j ) are superimposed for reference (dashed and solid lines, respectively).

While specific information has not received much attention from neuroscientists (but see DeWeese & Meister, 1999), it is a potentially useful measure that can be used to determine whether certain classes of neuronal responses transmit more information than others. For instance, is I (S, r j ) greater for spike trains with higher firing rates? How is I (S, r j ) related to ideal observer performance? While this question has not been discussed in the literature, we extend the previous results to address it. Since I (S, r j ) measures the amount of information carried by a particular response r j about the stimulus set S, we compare I (S, r j ) to the performance of an ideal observer that has observed a particular response r j , that is, P(c |r j ). Before examining the general relationship between I (S, r j ) and P(c |r j ), we use the examples in Figure 8A to highlight three features of their relationship. First, surprisingly, a response can carry negative information about S and be a better predictor of S than a response that carries positive information about S (e.g., compare rows 1 and 3). Second, P(c |r j ) does not

758

E. Thomson and W. Kristan

uniquely specify I (S, r j ) (compare rows 2 and 3). The converse also holds (rows 3 and 4). Third, associated with each conditional distribution P(S|r j ) is a point {P(c |r j ), I (S, r j )} that indicates the ideal observer performance and specific information for that distribution. Figure 8B plots the points that correspond to the four conditional distributions in Figure 8A. The bounds on the set of allowable such points, AM,i , are superimposed on Figure 8B for comparison. The derivation of AM,i , the set of allowable {P(c |r j ), I (S, r j )} points, is a natural extension of the results from sections 4.1 and 4.2. Assume that we know P(S) (a reasonable assumption, as in most discrimination tasks P(S) is controlled by the experimenter). Then H(S), the first term in equation 4.11 is fixed. Hence, I (S, r j ) varies only with the response-specific entropy H(S |r j ) (see section 4.2), the second term in equation 4.11. As shown in section 4.2 {P(c |r j ), H(S |r j )} must lie in the set AM,h , circumscribed by h max (MP(c | r ) ) and h min (MP(c | r ) ), bounds that are defined in equations 4.7 and 4.9 , respectively. It follows from basic properties of inequalities that for a given P(c |r j ) the lower and upper bounds on specific information are: i min (P(S) P(c | r ) ) = H(S) − h max (MP(c | r ) )

(4.12)

i max (P(S) P(c | r ) ) = H(S) − h min (MP(c | r ) ).

(4.13)

Figure 9 shows examples of i min (P(S) P(c | r ) ) and i max (P(S) P(c | r ) ) for different values of M and H(S). Four consequences of equations 4.12 and 4.13 deserve mention. First, if M (the number of stimuli) is fixed and H(S) varies, the shape of AM,i does not change, but is merely shifted vertically with H(S) (see Figure 9). Second, the greater H(S) is, the fewer possible negative I (S, r j ) values there are. If H(S) is maximized and equals log(M), then I (S, r j ) is always greater than or equal to zero (see Figure 9). This is because in such a case, P(S |r j ) cannot be more evenly spread than P(S), which is the necessary condition for the existence of a negative I (S, r j ). At the other extreme, as H(S) → 0, most distributions P(S |r j ) have greater entropy than the highly concentrated P(S), so there exists a greater number of possible negative values of I (S, r j ). Third, the range of permissible I (S, r j ) values at a particular P(c |r j ) increases with log(M − 1). This is because i max (P(S) P(c | r ) ) does not depend on M, and for a given P(c |r j ), i min (P(S) P(c | r ) ) decreases with log (M − 1). As with H(S), at a given value of H(S |r j ), the range of permissible P(c |r j ) values is a saturating exponential function of M (see section 4.1.2). Fourth, because i min (P(S) P(c | r ) ) and i max (P(S) P(c | r ) ) are invertible functions, it is possible to use numerical methods to calculate the range of P(c |r j )—values consistent with a given I (S, r j ) (see section 4.1.2). 4.4 Equivocation and the Ideal Observer. If we average the responseconditional entropy H(S | r j ) (see section 4.2) over all R, we obtain the

Quantifying Stimulus Discriminability

759

Figure 9: Comparison of I (S, r j ) and P(c). (A) Plot of upper and lower bounds on I (S, r j ) as a function of P(c) (dashed and solid lines, respectively). In this example, M = 2, so the upper and lower bounds are the same and the dashed lines are not visible. The three sets of bounds shown correspond to cases in which stimulus distributions with different H(S) values are chosen, and these entropy values are indicated next to the corresponding three lines. δ stands for an arbitrarily small real number. (B) Same as A, but with M = 4.

equivocation (Cover & Thomas, 1991) H(S | R) =

N  j=1

P(r j )H(S | r j ) = −

M N  

P(si , r j ) log(P(si | r j )).

(4.14)

j=1 i=1

H(S | R) is often described as the average uncertainty remaining in S once R is given (Ash, 1965). It describes the average minimum number of binary digits required to encode the value of S once R is specified. Before describing the general relationship between H(S | R) and P(c), we consider the examples in Figure 10A, which show four different joint

760

E. Thomson and W. Kristan

Figure 10: Examples comparing P(c) with H(S | R) and I (S,R). (A) The second column of the table contains four joint probability distributions. Columns 3–5 contain the corresponding values of P(c) (see equation 3.3), H(S | R) (see equation 4.14), and I (S,R) (see equation 4.17), respectively. (B) Scatter plot of the points {P(c), H(S | R)} corresponding to the rows from the table: the point labeled PHi corresponds to row i. The absolute upper and lower bounds on H(S | R) are overlaid for comparison.

distributions (M = 2, N = 3). For each distribution, P(S) = { 12 , 12 }, so H(S) = 1 bit. Given the joint distribution P(S,R), H(S | R) and P(c) can be calculated by substituting the appropriate terms into equations 4.14 and 3.3, respectively. The examples highlight two features of the relationship between H(S | R) and P(c). First, H(S | R) is not uniquely specified by P(c) (rows 1 and 2). The converse also holds (rows 1 and 4). These examples illustrate that H(S | R) and P(c) measure quite different features of P(S,R). P(c) remains unchanged as long as the sum of the maximal elements in the columns remains the same, and the nonmaximal elements of P(S,R) can arbitrarily vary within this constraint. Equivocation, on the other hand, increases as the entropy of this nonmaximal probability mass is increased. For example, the gray area in

Quantifying Stimulus Discriminability

761

Figure 4 shows the probability mass that contributes to P(e). Provided that the maximal elements remain unchanged, H(S | R) will increase as this gray area is spread out and decrease as the gray area becomes more concentrated. Second, higher equivocation does not imply lower P(c) (rows 1 and 3). That is, just because a variable R1 removes more uncertainty about S than another variable R2 (i.e., H(S | R1 ) < H(S | R2 )), this does not imply that an ideal observer can better estimate S on the basis of R1 . More generally, just as with H(S), there exist upper and lower bounds on H(S | R) as a function of P(c). For a given M and P(c), we denote the upper and lower bounds on H(S | R) as Hmax (MP(c) ) and Hmin (MP(c) ), respectively. Previous papers (Tebbe & Dwyer, 1968; Kovalevsky, 1968) show that Hmax (MP(c) ) = h max (MP(c) ).

(4.15)

Equation 4.15 shows that the upper bound on equivocation is the same as the upper bound on entropy. Figure 11A plots Hmax (MP(c) ) for the cases M = 2, 4, and 64 (dashed lines). Obviously, Hmax (MP(c) ) has the exact same properties as h max (MP(c) ) discussed in section 4.1.2. Equation 4.15 is equivalent to Fano’s inequality (Cover & Thomas, 1991), and provides an upper bound on P(c). That is, it delineates the best P(c) consistent with a given equivocation. On the other hand, it is possible that the actual P(c) of a distribution with a given value of H(S | R) is much lower than the upper bound provided by equation 4.15. Previous papers derive Hmin (MP(c) ) (Tebbe & Dwyer, 1968; Kovalevsky, 1968), which provides the lowest P(c) consistent with a given 1 equivocation. We present their result without proof. For P(c) between M and 1, there exists an integer k (the same k that was introduced in equation 1 4.9) such that k+1 ≤ P(c) ≤ k1 , and   k+1 1−k Hmin (MP(c) ) = log(k) + k(k + 1) log P(e) + . (4.16) k k Just as was the case with h min (MP(c) ), Hmin (MP(c) ) is broken up into M − 1 line segments. Each segment is the straight line that connects the end points of the M − 1 arch-shaped segments that delineate h min (MP(c) ), the lower bounds on entropy (see section 4.1.2). The lower bounds on equivocation are shown in Figure 11A for the M = 2, 4, and 64 cases (solid lines), and we include the lower bounds on entropy [h min (MP(c) )] in the same figure for comparison (light gray lines). Note that Hmin (MP(c) ) is not a function of M, so at a given value of P(c), increasing M does not change the lower bound on equivocation. Hmax (MP(c) ) and Hmin (MP(c) ) circumscribe the set AM,H of allowable {P(c), H(S | R)} points, which are indicated in Figure 11A for M = 2, 4, and 64.10 We highlight four features of AM,H . First, Feder and Merhav (1994) give

10

Formally, AM,H is the convex hull of AM,h (Tebbe & Dwyer, 1968; Kovalevsky, 1968).

762

E. Thomson and W. Kristan

Figure 11: Comparison of P(c) and H(S | R). (A) Plot of the upper and lower bounds on H(S | R) as a function of P(c) for M = 2, M = 4, and M = 64 (panels 1, 2, and 3, respectively). The upper bound (Hmax (MP(c) )) is indicated by a dashed line and the lower bound (Hmin (MP(c) )) by a solid line. The corresponding lower bounds on entropy (h min (MP(c) )) are overlaid in gray for comparison. The sets of allowable {P(c), H(S | R)} pairs are labeled AM,H . (B) Example of how P(S) further constrains what values of P(c) are consistent with a given H(S | R) (M = 3 in this example). In this case, we assume that P(smax ) = 0.7, so P(c) cannot be less than 0.7 (see section 3.1). The bounds on H(S | R) with this additional constraint included are shown in black, and the bounds provided by equations 4.15 and 4.16 alone are shown in gray for comparison.

algorithms for generating joint distributions that actually attain the bounds, so the bounds are tight.11 11 As a caveat, note that for a fixed input distribution P(S), these bounds are not necessarily tight (see section 4.5.2).

Quantifying Stimulus Discriminability

763

Second, just as was the case with H(S), for a given P(c) the range of possible H(S | R) values increases with log(M − 1). This is because the only term from equations 4.15 and 4.16 that varies with M is the log(M − 1) term that equation 4.15 inherits from equation 4.7 . As a corollary, the range of P(c) values consistent with a given H(S | R) increases exponentially with M. Third, although H(S | R) does not uniquely map onto P(c), the fact that Hmax (MP(c )) and Hmin (MP(c) ) are invertible (Feder & Merhav, 1994) implies that it is possible to calculate upper and lower bounds on P(c) for a given H(S | R) (see section 4.1 for details). Also, if P(S) is known, then an additional constraint can be used to further narrow the range of P(c) values consistent with a given equivocation. Namely, since P(c) must be greater than or equal to P(smax ) (see section 3.1), we can eliminate all P(c) values below P(smax ). As illustrated in Figure 11B, this constraint can considerably tighten the range of P(c) values consistent with a given H(S | R). Fourth, AM,H does not depend on the response distribution P(R) or the number of possible responses N. If the goal is to make inferences between H(S | R) and P(c), this is a very useful property, as it allows us to avoid two practical problems. First, it is in general a very difficult problem to obtain unbiased estimates of N and P(R). This is partly because when R is a variable describing a neuronal response, N is usually quite large and most outcomes have a very low probability of occurring. In such cases, estimates of N and P(R) suffer from biases due to undersampling (Paninski, 2004; Orlitsky, Santhanam, & Zhang, 2003). Second, if Hmax (MP(c) ) and Hmin (MP(c) ) depended on N, then for each representation of the neural response, we would have to recalculate the upper and lower bounds on H(S | R) in order to calculate the corresponding bounds on P(c). The independence of AM,H from N and P(R) circumvents both of these problems. 4.5 Mutual Information and the Ideal Observer. Typically, researchers calculate equivocation as an intermediate step in the estimation of mutual information, the information-theoretic quantity most often used by neuroscientists. The mutual information (also called transinformation and transmitted information) between random variables S and R is the average (over R) specific information:

I (S,R) =

N 

P(r j ) I (S,r j ) = H(S) − H(S | R) =

j=1

 × log

M N  

P(si ,r j ) . P(si )P(r j )

P(si ,r j )

j=1 i=1

(4.17)

I (S,R) is the standard measure of how much information a variable R transmits about variable S (Ash, 1965; Cover & Thomas, 1991).

764

E. Thomson and W. Kristan

4.5.1 Three Interpretations of I(S,R). Almost universally, I (S,R) is interpreted as a measure of the average amount of uncertainty removed by R about S (Ash, 1965; Cover & Thomas, 1991). Operationally, what does this mean? In this section, we discuss three interpretations of I (S,R) that often motivate the use of I (S,R) by neuroscientists. First, I (S,R) indicates how many fewer binary digits are required, on average, to encode S once R is specified. This follows from the fact that the specific information I (S, r j ) quantifies how many fewer digits are required to encode S once r j is known (see section 4.3), and I (S,R) is the average of I (S,r j ) over all R . Second, I (S,R) is a direct measure of the degree of statistical dependence between S and R (Schneidman, Bialek, & Berry, 2003). If S and R are independent variables, then for all stimulus-response pairs, P(si ,r j ) = P(si )P(r j ) (Yates & Goodman, 1999). This equality implies that all arguments of the log in equation 4.17 are one, so I (S,R) is zero. Interestingly, if observed frequencies are used to estimate the probabilities in equation 4.7, then I (S,R) is one-half the log-likelihood test statistic (G 2 ) when the null hypothesis is that S and R are independent (Forbes, 1995). A third interpretation of I (S,R) is that it measures stimulus discriminability, or how well the stimulus can be predicted given the neuronal response (Alkasab et al., 1999; Arabzadeh et al., 2004; Buracas & Albright, 1999; Li et al., 2004; Paz & Vaadia, 2004; Petersen et al., 2002; Pola et al., 2003; Theunnissen & Miller, 1991; for exceptions, see Oram, Foldiak, Perret, & Sengpiel, 1998; Treves, 1997). That is, researchers often assume that I (S,R) can be used as a surrogate for ideal observer performance, P(c). For example, one paper claims, “Mutual information quantifies how well an ideal observer of neuronal responses can discriminate between all the different stimuli based on a single trial” (Pola et al., 2003, p. 37). Even in time-series analysis, predictive information is defined as the mutual information between past and future events (Bialek, Nemenman, & Tishby, 2001), again suggesting that higher mutual information implies improved predictability. When examined quantitatively, this third interpretation of I (S,R) is shown to be incorrect. Let us assume P(S) is fixed, R1 and R2 are different representations of the neural response (e.g., PSTHs at different bin widths), and P(c)i is ideal observer performance when observing response variable Ri . The third interpretation is equivalent to I (S,R1 ) ≥ I (S,R2 ) ⇔ P(c)1 ≥ P(c)2 .

(4.18)

That is, if R1 carries more information about S than R2 , then an ideal observer would be better at estimating S on the basis of R1 than R2 . That equation 4.18 is false has been known since 1965 when a single counterexample was published (Wagner, 1965). In the next section we examine the general relationship between I (S,R) and P(c).

Quantifying Stimulus Discriminability

765

Figure 12: Examples comparing I (S,R) to P(c). Each point labeled P Ii is the {P(c), I (S,R)} point corresponding to row i of the table in Figure 10A. The upper and lower bounds on I (S,R) as a function of P(c) are overlaid in gray for comparison. In this example, M = 2.

4.5.2 Comparing I (S,R) and P(c). We highlight certain features of the relationship between I (S,R) and P(c) using previous examples: the mutual information corresponding to the four joint distributions in Figure 10A is shown in column 5 of the table in Figure 10A. Figure 12 illustrates the {P(c), I (S,R)} points associated with each of these joint distributions. We note two features of the relationship between P(c) and I (S,R) from these examples. First, equation 4.18 is incorrect, as can be seen by comparing rows 1 and 3. Second, P(c) is not associated with a unique I (S,R) (rows 1 and 2). The converse is also true (rows 1 and 4). These examples suggest that there is a range of I (S,R) values consistent with a given P(c), and vice versa. To address their relationship more generally, we derive upper and lower bounds on I (S,R) as a function of P(c).12 Assume P(S), and hence H(S), is fixed by the experimenter. For a given P(c), there exist the following upper and lower bounds on I (S,R):

 (4.19) Imin P(S) P(c) = H(S) − Hmin (Mp(c) )

 (4.20) Imax P(S) P(c) = H(S) − Hmax (Mp(c) ), where Hmax (MP(c) ) and Hmin (MP(c) ) are as defined in equations 4.12 and 4.13 (see section 4.4). The proof is as follows. Since the first term in equation 4.17 [H(S)] is constant, I (S,R) varies only with H(S | R). However, as Treves (1997) addresses a similar question under the assumption that M = N, that is, the number of responses equals the number of stimuli. Our results relax that assumption and produce tighter upper bounds. 12

766

E. Thomson and W. Kristan

described in section 4.4, we know that all points {P(c), H(S | R)} must lie in AM,H , the region whose lower and upper bounds are given by equations 4.12 and 4.13, respectively. Equations 4.19 and 4.20 follow from this fact and basic properties of inequalities. Clearly the bounds imposed on I (S,R) by equations 4.19 and 4.20 can be tightened further. For one, I (S,R) is always nonnegative (Cover & Thomas, 1991). Second, P(c) cannot be less than P(smax ) (see section 3.1). Plots of the bounds on I (S,R) with these additional two constraints added are shown in Figure 13 for different stimulus distributions P(S) (solid bold lines). The bounds provided by equations 4.19 and 4.20 alone are shown in light gray. We mention three facts about the relationship between P(c) and I (S,R). First, since Imin (P(S) P(c) ) and Imax (P(S) P(c) ) are one-to-one, monotonically increasing functions of P(c), both functions have inverses that can be estimated using the methods discussed in section 4.1. Hence, given an estimate of mutual information I (S,R), it is possible to calculate the range of P(c) values consistent with that estimate. As with all other quantities discussed so far, the range of P(c) values consistent with a given I (S,R) increases exponentially with M, the number of stimuli. Second, neither Imin (P(S) P(c|r ) ) nor Imax (P(S) P(c|r ) ) depends on P(R) or N. We discussed the benefits of this fact in section 4.4. Finally, using numerical optimization, we have generated many joint distributions P(S,R) that reach the bounds Imin (P(S) P(c) ) and Imax (P(S) P(c) ). Four such distributions are provided in Figure 10A. However, we have not proved in general that the bounds are tight. That is, for an arbitrary P(S) and P(c), it is not guaranteed that there exists a distribution P(R,S) such that I (S,R) = Imin (P(S) P(c) ) or Imax (P(S) P(c) ). We conjecture that the bounds are not generally tight. Aside from numerical optimization procedures we have implemented in which the bounds were not reached (data not shown), this conjecture is based on the following reasoning. Given P(S) and P(c), there must exist channel matrices P(R | S) and response distributions P(R) such that P(R | S)T P(S) = P(R). For this equation to hold and the bounds on I (S,R) to be tight, four constraints must be satisfied: the rows of P(R | S) must sum to 1, as must the elements of P(R), the resultant joint distribution P(R,S) must satisfy equation 3.3, and P(S,R) must reach the bound on I (S,R) given in equation 4.19 or 4.20. Even if N > M > 1, we think it is unlikely that these constraints can be satisfied for an arbitrary P(S), P(c) pair. An interesting area for future research would be to use these constraints to derive even tighter bounds on I (S,R). 4.6 Channel Capacity and the Ideal Observer. The capacity of a channel P(R | S) is defined as (Cover & Thomas, 1991) C(P(R | S)) = max I (S,R). P(S)

(4.21)

Quantifying Stimulus Discriminability

767

Figure 13: Comparison of I (S,R) and P(c). (A) Plots of Imax (P(S) P(c) ) and Imin (P(S) P(c) ) as functions of P(c) (dashed and solid gray lines, respectively). In these examples, M = 3. Panels 1 and 2 plot bounds on mutual information under two different assumptions about the value of the stimulus distribution P(S), and these P(S) values are shown in each panel. The corresponding entropy (H(S)) and maximum stimulus probability (P(smax )) values are also shown in each panel. The black lines show the constraints that are added to the bounds due to the nonnegativity of I (S,R) and the fact that P(c) cannot be less than P(smax ) (see the text). (B) Same as in A, except M = 64.

That is, the capacity is the maximum possible mutual information between S and R using the specified channel, where the maximum is calculated over the set of all possible input distributions. We use PC (S) to denote the stimulus distribution that solves this maximization problem. In general,

768

E. Thomson and W. Kristan

Figure 14: Examples comparing C(P(R | S)) to P(c). As discussed in the text, although C(P2 (R | S)) is greater than C(P1 (R | S)) (i.e., C2 > C1 ), an ideal observer of both channels achieves the same performance (P(c) = 2/3).

calculating the channel capacity is a difficult problem that is often solved using numerical optimization techniques (Cover & Thomas, 1991). As with the other information-theoretic quantities, we would like to know whether the fact that a channel has a higher capacity than another implies that an ideal observer of R would better be able to predict S using that channel. More precisely, it would be interesting to know whether the following is true: C(P1 (R | S)) > C(P2 (R | S)) ⇔ P1 (c) > P2 (c),

(4.22)

where Pi (R | S) is channel i and Pi (c) is ideal observer performance using channel i. Also, we stipulate that the P(c) values are calculated when the input distribution P(S) is set to PCi (S), the stimulus distribution that satisfies equation 4.21 for channel i. To our knowledge, nobody has published analytical results that bear on equation 4.22. Also, one must be cautious in interpreting counterexamples to equation 4.22 because different channels will likely have different input distributions that cause the channel to reach capacity, and it is not clear that a direct comparison of P(c) in such cases is appropriate. The channels shown in Figure 14 provide a counterexample to equation 4.22, and we have constructed the example so that in both channels,

Quantifying Stimulus Discriminability

769

PC (S) is the same. The first channel is a symmetrical channel, so C(P(R | S)) and PC (S) can be calculated with known formulas (Cover & Thomas, 1991).13 By substituting the second channel’s values into equation 4.17, the mutual information of the second channel can be simplified to I (S,R) = 13 H(S), which is maximized when P(S) = {1/2 ,1/2 }. While the results of the previous sections do not suggest an obvious way to calculate upper and lower bounds on C(P(R | S)) as a function of P(c), the example in Figure 14 suggests that such bounds likely exist. 5 Extension to Continuous Distributions 5.1 Background and Definitions. The results in section 4 depend on the assumption that S and R are discrete sets. In practice, this is not a significant limitation because continuous distributions are always effectively binned and discretized since we can never represent them with infinite precision. However, for conceptual completeness, we extend the previous analysis to continuous distributions. This extension requires that we modify both the definitions of the information measures and the measure of estimation error. The upshot of the analysis is that if S is continuous, the information measures provide even fewer constraints on the error measure than when S is discrete. The entropy of a continuous distribution S, also known as the differential entropy, is defined analogously to entropy for the discrete case, with the sum replaced by an integral (Cover & Thomas, 1991):  h(S) = −

∞

−∞

P(s) log(P(s))ds.

(5.1)

The remaining information measures such as equivocation are also defined analogously to the discrete case (Cover & Thomas, 1991). The term differential entropy is used instead of entropy because the differential entropy does not have the same mathematical properties as entropy. For example, the differential entropy changes when S is multiplied by a scaling factor a (i.e., h(a S) = h(S) + log(|a |)) (Shannon & Weaver, 1949). Also, h(S) can be negative (Shannon & Weaver, 1949). For example, the differential entropy of a uniform distribution defined over the interval [−w/2, w/2] is log(w), which is negative for w < 1 (Cover & Thomas, 1991). A second change we make to accommodate continuous variables is in our evaluation of estimation error. This is because when sˆ is a point estimate of the stimulus, it follows from equation 3.2 that P(c|r j ) = 0, so P(c) = 0. Hence,

13

For a symmetrical channel, PC (S) is the uniform distribution and the capacity is log(M) − H(P(R | si)), where H(P(R | si )) is the entropy of an arbitrary row of the channel (Cover & Thomas, 1991).

770

E. Thomson and W. Kristan

P(c) is an inappropriate measure of the success in estimating the value of a continuous variable S and needs to be replaced by an error term that increases with the distance between s and sˆ . The most common such measure is the squared error between s and sˆ , (s − sˆ )2 (Yates & Goodman, 1999). The average (over S) squared error, or mean squared error, is a useful overall measure of the quality of sˆ as an estimate of S (Yates & Goodman, 1999):  MSE(S) =

∞

−∞

P(s)(s − sˆ )2 ds.

(5.2)

When using MSE(S) to evaluate an estimate of S, the goal is to use an estimate sˆ that will minimize MSE(S), or the minimum mean squared error estimator of S. The minimum mean square estimator of any random variable S (discrete or continuous) is the mean of S, which we denote as µS (Yates & Goodman, 1999). As a corollary, the variance of S, σ 2S , is the actual value of the minimum mean squared error, which we denote E(S):  E(S) =

∞

−∞

p(s)(s − µS )2 ds = σ 2S .

(5.3)

The first equality is just equation 5.2 with the minimum mean squared error estimator µS substituted for sˆ , and the second equality is the definition of the variance of S (Yates & Goodman, 1999). While not technically an ideal observer, the minimum mean-squared error estimator is a close relative because it minimizes an error function. In sum, to extend the results from section 4 to continuous distributions, we must first replace the entropy-based information measures with the differential entropy-based measures and then replace P(c) with E(S). In the following section, we briefly show how to calculate the bounds on each information measure as a function of E(S). Note that this idea that such bounds could be derived was suggested but not carried out in Feder & Merhav (1994). 5.2 Comparing Information Measures and Minimum Mean Squared Error. The results from section 4 assume that S and R are discrete sets. The extension to the case where R is continuous and S is discrete is trivial, as none of the results depends on the assumption that R is discrete. However, for reasons described in the previous section, the extension to the case of continuous S requires a significant extension of the analysis, to which we now turn. 5.2.1 Differential Entropy and E(S). For a given value of E(S), there is an upper bound on h(S) given by Cover and Thomas (in press):

Quantifying Stimulus Discriminability

h max (S) =

1 log(2π e E(S)). 2

771

(5.4)

Figure 11A plots this relationship.14 Equation 5.4 places a lower bound on E(S) for a given value of h(S) (Cover & Thomas, in press), as can be seen in Figure 15A. We prove that there exists no upper bound on E(S) for a given value of h(S). Assume that there exists such an upper bound U on E(S) when h(S) takes on some arbitrary value h. To generate a counterexample, let [−w/2, w/2] be the interval of a uniform distribution that is constructed to satisfy h = log(w), as depicted in Figure 15B.1. We then split this uniform distribution into two segments of width w/2, such that there is a distance D between these two segments (see w Figure 15B.2). Because E(S) = 12 + 14 (D2 + Dw), which can be verified by substituting the equation for the split distribution into equation 5.3, there exists a number D such that E(S) > U . By indirect proof, we have shown that there exists no upper bound U on the error for a given entropy. The set B of possible {h(S), E(S)} points for continuous distributions includes the curve delineated by equation 5.4 and all points to the right of the curve (see Figure 15A). It follows that for a given E(S), there is no lower bound on h(S), or: h min (S) = −∞

(5.5)

In sum, h(S) can vary between − ∞ and h max (S) at a given E(S). 5.2.2 Response-Conditional Entropy and E(S | r j ). If we know that response r j occurred and the resulting conditional distribution of S is P(S | r j ), then the same analysis as in section 5.2.1 applies. That is, the minimum MSE estimator of S given r j is µS|r j, the mean of P(S | r j ), and the error E(S | r j ) in this case is σS2|r j , the variance of P(S | r j ) (Yates & Goodman, 1999). In this case, equations 5.4 and 5.5 apply, with S replaced by S|r j . 5.2.3 Specific Information and E(S|r j ). Following a proof strategy exactly analogous to that in section 4.3, it can be shown that I (S,r j )max = ∞

(5.6)

I (S,r j )min = − h max (S),

(5.7)

bounds, which are illustrated in Figure 15C. 14 Note that equation 5.3 is stated in Cover and Thomas (in press) without proof. Proof of equation 5.4: In the set of all continuous distributions with fixed variance σ 2 , the 

gaussian has the maximum differential entropy, which is h gauss (S) = 12 log 2π eσ 2 (Cover 2 & Thomas, 1991). Also, since the minimum error estimator has error E(S) = σ (see section 5.1), this implies that the maximum differential entropy consistent with a certain value of E(S) is given by equation 5.3.

772

E. Thomson and W. Kristan

Figure 15: Comparison of continuous information measures with minimum mean squared error estimators. (A) Plot of the maximum differential entropy (h max (S)) as a function of the minimum mean squared error E(S) (see equation 5.7). The set of points to the right of, and including, the curve is the set of permissible {E(S), h(S)} points. The same equation also describes the maximum response-conditional entropy and equivocation as a function of minimum meansquared error (see the text). (B) Geometrical representation of the proof that there is no upper bound on error for a given entropy. Panel 1 shows a uniform distribution with width w. When this probability mass is split into two sections of width w/2, and these sections are separated by distance D (panel 2), h(S) remains the same but the error increases with D2 (see the text). (C) Plot of the lower bound on specific information (gray line) and mutual information (black line) as a function of E(S |r ) and E(S), respectively. There is no upper bound on the information measures: the points to right of, and including, the lines are the set of permissible {E(S), I (S,R)} points (see the text).

Quantifying Stimulus Discriminability

773

5.2.4 Equivocation and E(S). The average minimum mean squared error E(S) when response variable R is available to help estimate S is the mean (over R) of E(S | r j ): E(S) =

N 

p(r j )E(S | r j ).

(5.8)

j=1

If R is continuous, the sum can be replaced by the appropriate integral. The bounds on h(S | R) are identical to the bounds on h(S). To prove this, we use a mathematical technique borrowed from the papers that derived the bounds on equivocation in the case of discrete S (Tebbe & Dwyer, 1968; & Kovalevsky, 1968).15 First, recall from section 5.2.2 that B, the set of permissible {E(S | r j ), h(S | r j )} points, is fixed by equations 5.4 and 5.5. Second, note that {E(S), h(S | R)} =

N 

P(r j ){E(S | r j ), h(S | r j )},

(5.9)

j=1

where the sum can be replaced by an integral if R is continuous. The right-hand side of equation 5.9 is simply a convex combination of points of the form {E(S | r j ), h(S | r j )}, which from section 5.2.2 we know must be in the set B. Hence, the set B∗ of permissible {E(S), h(S | R)}, pairs must lie in the convex hull of B. However, B is already a convex set, so B = B∗ (Boyd & Vandenberghe, 2004). In other words, the bounds on h(S | R) as a function of E(S) are identical to the bounds on h(S | r j ) as a function of E(S | r j ), that is, h max (S | R) = h max (S)

(5.10)

h min (S | R) = h min (S) = −∞.

(5.11)

Bounds are shown in Figure 15A. 5.2.5 Mutual Information and E(S). Using the results from section 5.2.4, it is possible to use an argument analogous to that in section 4.5 to show Imax (S,R) = h(S) − h min (S) = ∞

(5.12)

Imin (S,R) = h(S) − h max (S).

(5.13)

The fact that I (S,R) ≤ h(S) (Cover & Thomas, 1991) provides an additional constraint and the set of permissible {E(S), I (S,R)} points with all of these constraints included is shown in black in Figure 15C.

15 For an introduction to the concepts from convex analysis used in the following proof, see Boyd & Vandenberghe (2004).

774

E. Thomson and W. Kristan

6 Discussion 6.1 Measuring Stimulus Discriminability. An animal that cannot discriminate successfully on single trials will not survive long in natural conditions. A frog, for example, will not get a second chance to catch a fly if it misses on its first try. As discussed in section 3.2, ideal observer performance, P(c), is a useful and natural measure of single-trial stimulus discriminability. Researchers also often use information measures with the goal of quantifying discrimination performance. In section 4, we showed that this is typically not justified. In particular, rather than there being a one-to-one relationship between information-theoretic quantities and P(c), there is typically a range of permissible P(c) values associated with a given information measure. In section 5 we showed that when the analysis is extended to continuous stimulus distributions, the problems with the information measures are only exacerbated, as they provide no upper bounds on estimation error. If the goal is to make inferences from information measures to P(c), a caveat should be noted: as the number of stimuli (M) increases, the range of P(c) values associated with a given information-theoretic quantity increases exponentially (see section 4). Hence, it would be prudent to pick a stimulus set that is small enough to significantly narrow the range of permissible P(c) values. While the most conservative option is to use only two stimuli (M = 2), in some cases this will not be desirable because the stimulus space will not be adequately sampled. 6.2 Information Theory in Neuroscience. We have shown that to quantify stimulus discriminability, the tools of ideal observer analysis are preferable to those of information theory. Information theory, however, is helpful for answering questions that ideal observer analysis cannot address. We list three. We do not intend the list to be exhaustive, but are simply mentioning those applications that follow most naturally from the results in section 4. First, as we discussed in section 4.5.1, mutual information I (S,R) measures the degree of statistical dependence between random variables S and R. It is a useful measure because it is completely general: it will detect any deviation from independence whether it is due to linear correlation or some nonlinear dependency.16 Ideal observer analysis does not directly quantify such dependencies. But even if two random variables are dependent, the tools of ideal observer analysis are required to determine how well one can be predicted from the other. Second, there clearly exist psychophysical tasks that should be evaluated using information measures rather than P(c). For instance, if the goal is to

16 Of course, there exist other tests for statistical dependence (e.g., the χ 2 test), and it is up to the researcher to decide which statistic is appropriate for his or her data.

Quantifying Stimulus Discriminability

775

evaluate a subject’s performance in the game of 20 Questions, then H(S) indicates the best possible performance L min (see section 4.1.3). Third, natural selection may have discovered a coding strategy that uses the fewest number of elementary symbols required to encode certain classes of stimuli. That is, natural selection may have solved the optimization problem of reaching L min (see section 4.1.1). Such a hypothesis would be challenging to test empirically (e.g., it requires determining the set of elementary symbols used in the neural code, a problem discussed extensively in Brenner, Strong, Koberle, Bialek, & de Ruyter van Steveninck, 2000). Regardless of such practical difficulties, information theory contains the quantitative resources to address such questions about neural coding, resources that ideal observer analysis does not provide. 6.3 Open Questions and Future Directions. We finish by discussing five outstanding questions that are theoretically interesting and, to our knowledge, have not been addressed by previous work. First, it would be interesting to extend the discussion in section 4.6 by generating general bounds on channel capacity as a function of ideal observer performance. Also, an extension of the result to capacity in continuous channels would be useful. Second, we have analyzed quantities that depend on entropy rather than entropy rates (Cover & Thomas, 1991; Strong, Koberle, de Ruyter van Steveninck, & Bialek, 1998). However, Feder and Merhav (1994) show that for stationary processes, the results for H(S) described in section 4.1 also apply to entropy rates, so we expect our results to generalize to information rates (Strong et al., 1998). That is, we expect that a neuron that transmits information at a higher rate than another neuron is not necessarily the better predictor of a time-varying stimulus. However, this idea should be formally developed. Third, since spike trains are nonstationary (Berry & Meister, 1998), it would be interesting to determine how neurally inspired nonstationarities in P(S) and P(R | S) would affect the results in sections 4 and 5. Fourth, by picking as our point of comparison the ideal observer, we have treated all errors equally.17 This would be inappropriate in some instances. For example, if the goal is to estimate the spatial location of the stimulus, an estimate close to the actual location is better than an estimate that is completely off. In such cases, instead of judging an estimate as categorically right or wrong, the estimate should be evaluated by an error term that increases with the distance between sˆ and s, such as the mean squared error (see section 5.1). We implicitly addressed this concern in section 5.2, in which we used the minimum mean squared error as a cost

17 Technically, we have used a 0/1 loss function, also known as the Hamming distance between s and sˆ .

776

E. Thomson and W. Kristan

function. We showed that using the mean squared error as the cost function only amplifies the discrepancies between ideal observers and the information measures. However, a more general consideration of the relationship between arbitrary cost functions (Schneidman et al., 2003) and the information measures deserves analysis. Fifth, one of the virtues often stressed of the information-theoretic approach to neural coding is that the information measures are nonparametric (Paz & Vaadia, 2004; Peterson et al., 2002). Ironically, it is partly because we freed the proofs in sections 4 and 5 from assumptions about P(S) and P(R | S) that it was possible to demonstrate the differences between the information measures and ideal observers. On the other hand, under certain assumptions (e.g., the responses to different stimuli are univariate gaussians with identical variances), the relationship between P(c) and the informationtheoretic quantities is one-to-one. We would like to know the minimal assumptions required about P(S) and P(R | S) for ideal observer performance to be uniquely specified by the information measures. We hope that the work presented here will act as an impetus for such an analysis so that we can know under what conditions it is possible to make unambiguous inferences between encoding and decoding measures in the nervous system. Acknowledgments Thanks to Kevin Briggman, E. J. Chichilnisky, Philip Gill, Dan Kersten, Samar Mehta, Joy Thomas, Jonathan Victor, the Sejnowski lab, Diane Whitmer, and an anonymous reviewer. Thanks especially to Barbara Leary. References Alkasab, T. K., Bozza, T. C., Cleland T. A., Dorries, K. M., Pearce, T. C., White, J., & Kauer, J. S. (1999). Characterizing complex chemosensors: Information-theoretic analysis of olfactory systems. TINS, 22, 102–108. Arabzadeh, E., Panzeri, S., & Diamond M. E. (2004). Whisker vibration information carried by rat barrel cortex neurons. J. Neurosci., 24, 6011–6020. Ash, R. B. (1965). Information theory. New York: Dover. Berry, M. J., & Meister, M. (1998). Refractoriness and neural precision. J. Neurosci., 18, 2200–2211. Bialek, W., Nemenman, I., & Tishby, N. (2001). Predictability, complexity, and learning. Neural Computation, 13, 2409–2463. Boyd, S., & Vandenberghe, L. (2004). Convex optimization. Cambridge: Cambridge University Press. Brenner, N., Strong, S. P., Koberle, R., Bialek, W., & de Ruyter van Steveninck, R. R. (2000). Synergy in a neural code. Neural Comp., 12, 1531–1552. Britten, K., Shadlen, M. N., Newsome, W. T., & Movshon, J. A. (1992). The analysis of visual motion: A comparison of neuronal and psychophysical performance. J. Neurosci., 12, 4745–4767.

Quantifying Stimulus Discriminability

777

Buracas, G. T., & Albright, T. D. (1999). Gauging sensory representations in the brain. TINS, 22, 303–309. Cover, T. J., & Thomas, J. (1991). Elements of information theory. New York: Wiley. Cover T. J., & Thomas J. (In press). Elements of information theory (2nd ed.). New York: Wiley. de Boor, C. (1978). A practical guide to splines. New York: Springer-Verlag. DeWeese, M. R., & Meister, M. (1999). How to measure the information gained from one symbol. Network: Comput. Neural Syst., 10, 325–340. Duda, R. O., Hart, P. E., & Stork, D. G. (2001). Pattern classification (2nd ed.). New York: Wiley. Feder, M., & Merhav, M. (1994). Relations between entropy and error probability. IEEE Transactions on Information Theory, 40, 259–266. Forbes, D. A. (1995). Classification-algorithm evaluation: Five performance measures based on confusion matrices. J. Clin. Monit., 11, 189–206. Geisler, W. S. (1989). Ideal observer theory in psychophysics and physiology. Physica Scripta, 39, 153–160. Geisler, W. S. (2003). Ideal observer analysis. In L. M. Chalupa & J. S. Werner (Eds.), Visual neurosciences. Cambridge, MA: MIT Press. Golic, J. (1987). On the relationship between the information measures and the Bayes probability of error. IEEE Transactions on Information Theory, 33, 681–693. Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley. Knill, D. C., & Kersten, D. (1991). Ideal perceptual observers for computation, psychophysics, and neural networks. In R. J. Watt (Ed.), Pattern recognition by man and machine (pp. 83–97). New York: Macmillan Press. Kovalevsky, V. A. (1968). Character readers and pattern recognition. New York: Spartan Press. Lawson, J. L., & Uhlenbeck, G. E. (1950). Threshold signals. New York: McGraw-Hill. Lettvin, J., Maturana, H., McCulloch, W. S., & Pitts, W. (1959). What the frog’s eye tells the frog’s brain. Proc. IRE, 47, 1940–1951. Li, W., Piech, V., & Gilbert, C. D. (2004). Perceptual learning and top-down influences in primary visual cortex. Nat. Neurosci., 7, 651–657. Liu, Z., Knill, D. C., & Kersten, D. (1995). Object classification for human and ideal observers. Vision Research, 35, 549–568. MacKay, D., & McCulluch, D. S. (1952). The limiting information capacity of a neuronal link. Bull. Math. Biophys., 14, 127–135. Oram, M. W., Foldiak, P., Perret, D. I., & Sengpiel, F. (1998). The “ideal homunculus”: Decoding neural population signals. TINS, 21, 259–265. Orlitsky, A., Santhanam, N. P., & Zhang, J. (2003). Always good Turing: Asymptotically optimal probability estimation. Science, 302, 427–431. Paninski, L. (2004). Estimating entropy on m bins given fewer than m samples. IEEE Transactions on Information Theory, 50, 2200–2203. Paz, R., & Vaadia, E. (2004). Learning-induced improvement in encoding and decoding of specific movement directions by neurons in the primary motor cortex. PLoS Biol, 2(2), e45. Available online: http://www.plosbiology.org/plosonline/? request=get-document&doi=10.1371%2F journal.pbio.0020045. Perkel, D. H., & Bullock, T. H. (1968). Neural coding. Neurosciences Research Bulletin, 6, 221–348.

778

E. Thomson and W. Kristan

Petersen, R. S., Panzeri, S., & Diamond, M. E. (2002). Population coding in somatosensory cortex. Curr. Opin. Neurobiol., 12, 441–447. Pola, G., Thiele, A., Hoffmann, K. P., & Panzeri, S. (2003). An exact method to quantify the information transmitted by different mechanisms of correlational coding. Network: Comput. Neural Syst., 14, 35–60. Rieke, F., Warland, D., de Ruyter van Steveninck, R., & Bialek, W. (1997). Spikes: Exploring the neural code. Cambridge, MA: MIT Press. Schneidman, E., Bialek, W., & Berry, M. J. (2003). Synergy, redundancy, and independence in population codes. J. Neurosci., 23, 11539–11553. Shannon, C. E., & Weaver, W. (1949). The mathematical theory of communication. Urbana: University of Illinois Press. Strong, S. P., Koberle, R., de Ruyter van Steveninck, R. R., & Bialek, W. (1998). Entropy and information in neural spike trains. Phys. Rev. Lett., 80, 197–200. Tebbe, D. L., & Dwyer, S. J. (1968). Uncertainty and the probability of error. IEEE Transactions on Information Theory, 14, 516–518. Theunissen, F. E., & Miller, J. P. (1991). Representation of sensory information in the cricket cercal sensory system. II. Information theoretic calculation of system accuracy and optimal tuning-curve widths of four primary interneurons. J. Neurophys., 66, 1690–1703. Treves, A. (1997). On the perceptual structure of face space. BioSystems, 40, 189–196. Victor, J. D. (1999). Temporal aspects of neural coding in the retina and lateral geniculate: A review. Network, 10, R1–66. Wagner, T. J. (1965). Some remarks concerning uncertainty and the probability of error. IEEE Transactions on Information Theory, 11, 144–145. Yates, R. D., & Goodman, D. J. (1999). Probability and stochastic processes: A friendly introduction for electrical and computer engineers. New York: Wiley.

Received May 13, 2004; accepted September 4, 2004.

REVIEW

Communicated by Mark Plumbley

Nonlinear Complex-Valued Extensions of Hebbian Learning: An Essay Simone Fiori [email protected] Facolt`a di Ingegneria dell’Universit`a di Perugia, Polo Scientifico e Didattico del Ternano, I-05100, Terni, Italy

The Hebbian paradigm is perhaps the best-known unsupervised learning theory in connectionism. It has inspired wide research activity in the artificial neural network field because it embodies some interesting properties such as locality and the capability of being applicable to the basic weight-and-sum structure of neuron models. The plain Hebbian principle, however, also presents some inherent theoretical limitations that make it impractical in most cases. Therefore, modifications of the basic Hebbian learning paradigm have been proposed over the past 20 years in order to design profitable signal and data processing algorithms. Such modifications led to the principal component analysis type class of learning rules along with their nonlinear extensions. The aim of this review is primarily to present part of the existing fragmented material in the field of principal component learning within a unified view and contextually to motivate and present extensions of previous works on Hebbian learning to complex-weighted linear neural networks. This work benefits from previous studies on linear signal decomposition by artificial neural networks, nonquadratic component optimization and reconstruction error definition, neural parameters adaptation by constrained optimization of learning criteria of complex-valued arguments, and orthonormality expression via the insertion of topological elements in the networks or by modifying the network learning criterion. In particular, the learning principles considered here and their analysis concern complex-valued principal/minor component/subspace linear/nonlinear rules for complexweighted neural structures, both feedforward and laterally connected. 1 Introduction The basic assumptions of connectionism on neural learning are reasonably nonrestrictive and rather appealing from an engineering point of view:

r

The locality principle, which states that the modifications in the connections depend on the activities of pre- and postsynaptic neurons and do not depend on the activity levels of the other neurons

Neural Computation 17, 779–838 (2005)

© 2005 Massachusetts Institute of Technology

780

S. Fiori

r r

The principle stating that the modification of synapses is slow compared with the characteristic time of neuron dynamics The forgetting principle, stating that if either the pre- or postsynaptic neurons or both are silent, then no synaptic changes take place except for exponential decay

Classical contributions to this analysis are given in Bechtel and Abrahamsen (1993) and Linsker (1992). Perhaps the most influential work in the history of connectionism is the contribution of the neuropsychologist Donald Hebb. In his famous book, Hebb (1949) presented a theory of behavior based on the physiology of the nervous system. He reduced the types of physiological evidence to two main categories: the existence and properties of continuous cerebral activity and the nature of synaptic transmission in the central nervous system. Hebb combined these principles in order to develop a theory of how learning occurs within an organism. As a matter of fact, Hebbian learning and cell/synapses creation/death are kinds of learning for which there is neural evidence. In Hebbian learning, the weights between neurons are adjusted so that each weight better captures the relationship between the units. The Hebbian learning rule specifies that the weight of a connection between two units should be varied in proportion to the product of their activation. It states that the connections between two neurons might be strengthened if the neurons fire simultaneously. As a result, if a unit fires when presented with a pattern, the weights from the active inputs are strengthened, so that the unit will respond to the same pattern even better in the future. In a network of many units, however, a mechanism is necessary to prevent different neurons from encoding the same features. In particular, a way to encourage sparse representations between units is to promote unit decorrelation through lateral inhibition (Barlow, 1989, 1998, 2001; Foldi´ ¨ ak, 1990). Lateral inhibition is a distinguishing feature of some artificial unsupervised neural networks. It provides a mechanism that makes a network of neurons hierarchical and with which the neurons compete for the right to generate a response to the incoming stimuli. This kind of competition enables the abstract neuronal units to form receptive fields that are sensitive to stimuli coming from different regions of the input space (Spratling & Johnson, 2002). Mutual inhibition between units may be achieved by training the lateral connections via an anti-Hebbian rule: whenever two units in a layer are active simultaneously, the connection between them becomes more inhibitory, so that their correlation is decreased, and their joint activity will be discouraged in the future (Foldi´ ¨ ak, 1990). For the standard weight-and-sum neuron model, Hebbian learning makes the synapses be correlators. In some circumstances, the plain Hebbian learning continually strengthens a unit’s weights without bound (Baldi & Hornik, 1995; Bechtel & Abrahamsen, 1993; Miller & MacKay, 1994). Within networks endowed with the appropriate structure,

Nonlinear Complex-Valued Extensions of Hebbian Learning

781

however, the unboundness problem is not present (consider, for instance, the recurrent error correction model discussed in Harpur, 1997). A number of years ago, Amari (1977) investigated a neural theory of association and concept formation, and Oja (1982) studied a stabilized version of the classical Hebb rule. They observed that under fair conditions, the stabilized learning rule makes the neuron capture the most powerful eigenvector of the input covariance matrix or, in other terms, the rule makes the neuron able to extract the principal component from the input multivariate random signal. Since Amari and Oja’s pioneering work, several new learning algorithms have been proposed for extending the one-unit principal component neural system to complete neural networks. The classical contribution in the field of principal component networks may be traced back to Sanger (1989), who used an online version of the well-known Gram-Schmidt orthogonalization algorithm. Also, Rubner and Tavan (1989) and Diamantaras and Kung (1996) introduced a linear neural network endowed with lateral inhibitory connections for achieving output decorrelation. Their algorithm is often referred to as adaptive principal component extractor (APEX). In more recent years, several authors have introduced different principal component analysis (PCA) rules for generalizing classical ones (Abbas & Fahmy, 1994; Bannour & Azimi-Sadjadi, 1995; Costa & Fiori, 2001; Fiori & Piazza, 2000; Fiori, 2001, 2002). A PCA-related argument is principal subspace analysis (PSA), which concerns the computation of the subspace of the input space spanned by the principal eigenvectors (Oja, 1989). Extended discussions on Hebbian, anti-Hebbian, and modified Hebbian learning paradigms and their properties in the context of component and subspace analysis may be found in Atick and Redlich (1993), Fyfe and MacDonald (2002), Harpur (1997), and Weingessel and Hornik (2000). It is likely that one of the reasons for the success of stable Hebbian learning theory is its usefulness for solving many signal processing problems, as illustrated, for instance, in Costa and Fiori (2001), Kung, Diamantaras, and Taur (1994), Luo and Unbehauen (1997), and Palmieri and Zhu (1995), where extracting the first principal components is shown to be of prime importance. Nevertheless, it has been clearly demonstrated that computing the last principal components of a data sequence—those principal components endowed with the smallest (nonzero) power—may be very useful as well (Gao, Ahmed, & Swamy, 1994; Klemm, 1987; Mathew & Reddy, 1994; Schmidt, 1986; Xu, Oja, & Suen, 1992). The extraction of the last principal components is commonly referred to as minor component analysis (MCA). Nonlinear versions of standard Hebbian or anti-Hebbian learning rules, namely, those rules based on the optimization of nonquadratic cost functions, may make the involved linear neural networks capable of performing the independent component analysis (ICA) of incoming signals (for a review of ICA in the neural network field, see, e.g., Bell & Sejnowski, 1995). An early

782

S. Fiori

and pervasive physiological example of such behavior was presented by Barlow and Foldi´ ¨ ak (1989). They recalled that the information about the basic tastes (sweet, salty, bitter, and sour) is not carried on by separate fibers; instead, each fiber carries a mixed signal with different relative sensitivities to the four basic tastes. The operation of separating the four mixed quantities, performed by the nervous system, could not be explained by simple decorrelation rule. Thus, Barlow and Foldi´ ¨ ak proposed a modified (anti-Hebbian) rule for synaptic strength tuning based on the assumption of nonnegativity of the taste variables.1 Classical contributions on nonlinear extensions to PCA may be found in Karhunen and Joutsensalo (1994, 1995) and Oja (1994). The aim of this review is to describe extensions of previous work to complex-weighted linear neural networks in a formal way. The guideline for the presented research is complex-valued gradient-based optimization of nonquadratic learning criteria. The derivations that are presented subsume some contributions found in the scientific literature: We recall interesting contributions from advanced statistical theory of learning and try to relate sparsely published valuable works on these topics in a unifying view. These works span the following topics:

r

r

r

r

Linear signal decomposition by artificial neural networks. This widely known theory allows expressing a multivariate signal as a linear combination of features or components that enjoy some special statistical property (like incorrelatedness or independency). The decomposition consists in jointly finding a basis for the signal and the corresponding components. The decomposition is then achieved by components optimization or by reconstruction error minimization. Nonquadratic component optimization and reconstruction error definition. Both rely on the proper selection of involved nonquadratic learning criteria via available suitable interpretations of Hebbian learning, such as the ones suggested by Song, Yilong, and Feng (1998) and by Sudjianto and Hassoun (1995). Neural parameters adaptation by constrained optimization of learning criteria of complex-valued arguments. This is the basis for neural learning with physical or structural constraints, or both, when complexvalued input signals or input-output signal pairs are to be dealt with. The constraints considered here are related to the orthonormality of neural connection matrices. Orthonormality expression via the insertion of topological elements in the networks or by modifying the network learning criterion.

1 This is what is currently referred to as nonnegative independent component analysis (Plumbley, 2003).

Nonlinear Complex-Valued Extensions of Hebbian Learning

783

Topological constraints are given, for example, by lateral connections that force the neurons in a network to encode uncorrelated features. Modified learning criteria are created, for example, by the Lagrange multiplier method for equality constraints. The development of the theory of linear complex-weighted neural networks is supported by pervasive engineering and physiological motivations. In the supervised-learning field, for instance, it was reported (see Michaels & Upadhyaya, 1999) that complex-valued backpropagation models train faster than conventional backpropagation networks, are more resistant to local minima, and exhibit better generalization ability. By extending previous work proposed independently by some researchers (Benvenuto & Piazza, 1992; Georgiou & Koutsougeras, 1992), Nitta (1997, 2000, 2004) reported that complex-valued backpropagation architectures exhibit reduced probability of learning standstill and are able to perform transformations of geometric figures in the complex plane, which their realvalued counterparts are unable to effect. Also, following the pioneering work by Widrow (see, e.g., Widrow & Winter, 1988) on merging neural networks and adaptive filters, recently Hanna and Mandic (2003) presented a complex-valued nonlinear gradient-descent learning algorithm for a nonlinear neural adaptive filter with adaptive complex-valued activation function; such structure is mentioned to be beneficial when dealing with signals that have rich dynamical behavior. Like other applications, Miyauchi, Seki, Watanabe, and Miyauchi (1993) proposed an interpretation of optical flow based on complex-weighted neural networks, while Muezzino ¨ glu, ˇ Guzeli¸ ¨ s, and Zurada (2003) recently proposed a design method for complexweighted multistate Hopfield memory, which appears as a generalization of the conventional Hopfield model that can be an efficient tool to process static integral information. The new method was shown to outperform the generalized Hebbian rule, which has yet constituted the only learning rule for this model, in associating phase-modulated integral information. Also, from a biological point of view, the current interest in pulse-coded neural networks (Levine, Brown, & Shirey, 1999) has created a need for a mathematically compact way of dealing with phase as well as magnitude of neural signals, which may be easily found in the complex-valued representations. In the unsupervised field, a first extension to classical Sanger’s PCA learning theory to the complex plane has been presented by De Castro, De Castro, Amaral, and Franco (1998) with application to optimal image compression in the spectral domain. An application of nonlinear MCA extended to the complex domain has been presented recently in Fiori (2003b), with application to robust beam forming. Robust beam forming is a well-known signal processing technique allowing for performing spatial filtering of a signal source in the presence of spatial noise and other disturbing sources by means of an array of electromagnetic antennas or electromechanical

784

S. Fiori

microphones, provided that the direction of arrival of the primary source is known. A beam former may be realized by a complex-weighted neural unit fed with the Fourier transform of the measured signals, hence its complex-valued nature. A bioengineering application of adaptive beam forming is in a hearing aid (Kates, 1993). A way to train the beam-forming neuron is to force it to solve a correlation-matrix minimal-eigenvalue extraction problem, which may be formulated as an MCA problem (Fiori, 2003b). Further applications of complex-valued principal or minor component analysis have been reviewed and discussed in Luo and Unbehauen (1997). The theory of complex-weighted network learning has led to effective independent component analysis algorithms for complex-valued statistically independent signals (see, e.g., Fiori, 2000, 2003a). Typical applications of complex-valued ICA algorithms are to electromagnetic phenomena analysis and modeling (Desodt & Muller, 1990; Fiori, Faba, Albini, Cardelli & Burrascano, 2003). Further examples of applications of complex-valued ICA algorithms are frequency-domain algorithms for separation of sound signals and for biomedical data analysis (Cichocki & Amari, 2002). Also, Michaels and Upadhyaya (1999) provided an architecture for signal and image processing (focused on capacitance and inductance sensors) and for modeling the timing and pattern of neural signals in biological settings. In particular, they introduced a complex-valued associative memory theory and showed that in its iterative form, its learning procedure is a complexvalued Hebbian adapting algorithm that uses association rather than error correction. Section 2 of this review is devoted to the complex-valued Hebbian learning theories stemming from the optimization of generalized network output power, while section 3 presents those extended Hebbian learning theories stemming from generalized reconstruction error minimization. In particular, section 2.1 presents a learning theory for a linear feedforward architecture, where the constraints on symmetry and hierarchy are embodied in the learning criterion, while section 2.2 introduces a theory for Rubner-Tavan’s laterally connected network structure (Rubner & Tavan, 1989). Both sections present a comparison with the real-valued networks and rules counterparts. Section 2.3 presents a possible way of choosing the nonlinearity involved in the nonquadratic learning criteria based on the Sudjianto-Hassoun fruitful maximum-mismatch principle (Sudjianto & Hassoun, 1995). Section 2.4 presents some formal results pertaining to the convergence of learning rules in the one-unit case, while section 2.5 deals with the multiunit case. Section 3.1 illustrates a learning theory stemming from generalized reconstruction error minimization for a symmetrical network, and section 3.2 discusses some possible ways to select the involved nonquadratic error measures, while section 3.3 considers the hierarchical case and the corresponding choices of involved nonlinearities. Section 4 concludes the review.

Nonlinear Complex-Valued Extensions of Hebbian Learning

785

2 Complex-Valued Hebbian Learning by Nonquadratic Output Optimization This section presents a unified view and presents and discusses generalizations of networks’ architectures and learning rules for principal component and subspace analysis of complex-valued signals. Generalization is achieved by extending the classical architectures to complex-weighted neural networks and by extending the classical learning criteria to nonquadratic optimization objectives by the introduction of nonlinear functions. A possible choice of the involved nonquadratic function is also discussed. In this section, the Lagrange multiplier method is used extensively in order to construct suitable criteria for learning under orthonormality constraints. It is worth recalling how the Lagrange multiplier method for equality constraints works in practice (see Bertsekas, 1996, for details of the theory and specific computations). Suppose that J (w) and L i (w), i = 1, . . . , m and w ∈ R p, are differentiable functions that map R p → R, and the goal is to solve opt J (w) such that L j (w) = 0 , j = 1, . . . , m, namely, to find the optimum (maximum or minimum) of the function J (w) that satisfies the equality constraints L j (w) = 0. This is equivalent to solving the following problem,
opt

J (w) −

m 

 λ j L j (w) ,

j=1

for w and λ j , without restrictions. The variables λ j ∈ R are termed Lagrange multipliers. It is not difficult to recast the above optimization problem in terms of matrix-type variables if necessary. opt In practice, the optimal multipliers λ j may be found analytically through a multiplier elimination method (Bertsekas, 1996). Then the optimal parameter vector w may be found by a gradient-based algorithm,   dw ∂ J (w) opt =± , dt ∂w   m  ∂ J (w) opt def ∂ opt ∂ = λj J (w) − L j (w) , ∂w ∂w ∂w j=1

(2.1) (2.2)

where the + sign denotes maximization and the − sign denotes minimization.

786

S. Fiori

The contents of this section may be anticipated:

r r r r r

Learning principal/minor component/subspace by complex-valued gradient-based optimization of nonquadratic criteria Learning in an orthogonally constrained network by complex-valued gradient-based optimization of nonquadratic criteria and output decorrelation by lateral inhibition Choice of the nonlinear criteria via Sudjianto-Hassoun interpretation of Hebbian learning, illustrated by numerical examples Analysis of static and dynamical properties of one-unit complexvalued PCA and MCA neural systems, illustrated by numerical examples Multiunit complex-valued component and subspace networks—notes on dynamical properties and on the design of learning algorithms that agree with the orthonormality constraints

2.1 Complex-Valued Gradient-Based Optimization of Nonquadratic Criteria. The origin of modern Hebbian learning may be traced back to Oja’s work on PCA learning theory. PCA is appropriate, for example, when measures on a number of observed variables have been obtained, and the good is to develop a smaller number of artificial variables (termed principal components) that will account for most of the variance in the observed variables. The principal components may then be used as predictor or criterion variables in subsequent analyses. In particular, PCA deals with the extraction of the principal component that accounts for the largest fraction of total variance. Oja’s first principal component learning rule was based on a single neuron, described by y(t) = wT (t)x(t), where x(t) ∈ R p represents a stationary multivariate zero-mean random process endowed with finite covariance matrix , w(t) ∈ R p is the neuron’s weight vector, and y(t) ∈ R is the neuron’s output signal. The learning task of Oja’s neuron is to maximize the variance of its response signal, namely, of E[y2 ] = wT w, under the constraint wT w = 1. This learning rule reads dw(t) = E[x(t)y(t) − w(t)y2 (t)] . dt

(2.3)

In the above relations, t denotes continuous time and E[·] denotes statistical expectation (ensemble average).2 This expression clearly reveals the 2 More formally, the ensemble average should be denoted by the symbol E [ f |w], x which denotes conditional expectation of function f (x) with respect to the statistics of x subject to the hypothesis w. For conciseness, it will hereafter be simply written in short notation as E[ f ].

Nonlinear Complex-Valued Extensions of Hebbian Learning

787

presence of the Hebbian term x(t)y(t) and of a stabilizing term; thus, it is also referred to as a stabilized Hebbian learning equation. The Oja rule was initially formulated as an approximated normalized Hebbian learning algorithm by using a normalization step that binds the connection vector w to belong to a unit-radius sphere (Oja, 1982): In the discrete-time counterpart of such Hebbian learning, the learning step-size is usually supposed to be very small; therefore, it is possible to expand by Taylor series the normalized Hebbian rule with respect to the learning step size and to truncate the expansion to the first-order term. This procedure led to the discrete-time version of learning rule (2.3). In first PCA, it is known that the extracted weight vector coincides with the eigenvector of the covariance matrix  corresponding to its largest eigenvalue. In general PCA, more than one unit-norm weight vector is sought. As it is known that the eigenvectors of a covariance (hence symmetric) matrix are orthogonal to each other, it is necessary to enforce the orthogonality of network’s weight vectors. Related concepts to PCA are minor component analysis, which consists of extracting the weakest principal components, and principal or minor subspace analysis, which consists in finding (whatever basis of) the linear subspaces spanned by the principal or minor vectors. In order to generalize Oja’s work to the complex domain, a linear neural network may be considered, which is formed by linear complex-weighted units. Let the following learning objective function for the neural net be defined, def

J (wk ) = U(wk ) + L(wk ) , k = 1, ..., m,

(2.4)

where wk ∈ C p represents the weight vector of the k th neuron. The criterion def U(·) contains a nonlinear function of the k th neuron’s output yk = wkH x and is defined as follows, def

U(wk ) = E[ f (wkH x)] ,

(2.5)

where x ∈ C p is the input vector of the network whose values are drawn from a p-dimensional complex-valued input space and the superscript H denotes Hermitian transpose (i.e., transpose plus complex conjugation). The number of neurons of the network is denoted here with m, where m ≤ p. The function f (·) is a real-valued, positive function of complex-valued argument and is of the form + f (ζ ) = g(|ζ |) , ζ ∈ C , g : R+ 0 → R0 . def

(2.6)

The function g(u) is normally required to be differentiable and convex with a minimum in u = 0 in a convenient right-sided neighborhood of the origin.

788

S. Fiori

The choice of the function f (ζ ) as a real function of |ζ | seems reasonable for the following reasons:

r r

r

r

In the classic (Oja’s) case, the objective function contains the neuron’s output variance, which depends on the modulus (i.e., the unsigned value) of the output only. The leading principle in minor, principal component, or subspace analysis is projection power minimization or maximization. In the signal processing literature, the power of a complex-valued signal is defined as the average of its square modulus. This quadratic optimization principle may be extended to robust (nonquadratic) optimization in a natural way, as shown in this and following sections. Ample literature is available on signal processing techniques and algorithms, which are based on transformed-output-modulus optimization principles. A recent review of such techniques in blind signal processing may be found, for instance, in Amari and Cichocki (1998), Chow and Fang (2001), Pandey (2001), and Thirion Moreau and Moreau (2002). Some algorithms described in this article have readily found application to engineering-type signal-processing tasks (e.g., adaptive beam forming, blind separation of digital modulation type signals, and blind localization of electromagnetic sources), which are based on transformed output modulus optimization.

The function L(·) is needed for embodying the necessary constraints of orthonormality of the weight vectors in criterion 2.4, namely: w Hj wk = 0 if j = k , wkH wk = 1 .

(2.7)

Note that each orthogonality condition may be rewritten more conveniently by observing that w Hj wk = 0 if and only if Re{w Hj wk } = 0 and Im{w Hj wk } = 0. Thus, the function L(·) may be expressed as

def

L(wk ) = λkk (wkH wk − 1) +

K (k)  j=1, j=k

Re{λk j wkH w j } .

(2.8)

A set of Lagrange multipliers {λk j } has been introduced in order to take into account the condition ensuring the normality of the weight vectors and the pairs of conditions needed for ensuring the orthogonality principle to be met after learning. In fact, the following identity holds: Re{λk j (wkH w j )} = Re{λk j }Re{wkH w j } − Im{λk j }Im{wkH w j }.

Nonlinear Complex-Valued Extensions of Hebbian Learning

789

Therefore, the expression Re{λk j (wkH w j )} is a compact way to express the equality constraints Re{w Hj wk } = 0 and Im{w Hj wk } = 0 by the Lagrange multiplier method (the minus sign between the above terms does not matter because the multipliers are variables to be determined). Note that the multipliers are complex valued, except for the λkk , which are real valued; therefore, the cost function L(·) is by construction real valued. The indexing function K (·) has been introduced in order to take into account two distinct cases:

r r

The symmetric case, where K (k) = m for each neuron, which leads to a generalized principal subspace analyzer The hierarchical case, where K (k) = k, actually leading to a generalized principal component analyzer opt

In order to look for optimal weights wk maximizing criterion 2.4, a gradient steepest-ascent learning algorithm is employed by following the general scheme shown in the equations 2.1 and 2.2. By definition, the gradient of a real-valued function F (w) with respect to a complex-valued vector w is intended as ∂ F (w) def ∂ F (u, v) ∂ F (u, v) = +i , ∂w ∂u ∂v

(2.9)

def √ where i = −1 and u + iv = w; also, F (u, v) : R p × R p → R denotes the real-valued function of two real-valued arguments derived from F (u + iv). First, it is necessary to evaluate the gradient:

     ∂U(wk ) dg(|yk |) ∂|yk | g (|yk |)  =E =E yk x . ∂wk d|yk | ∂wk |yk |

(2.10)

k| = yk x, where the superIn fact, from definition 2.9, it follows that |yk | ∂|y ∂wk script  stands for complex conjugation. In the same way, the gradient of the function L(·) is to be evaluated. The expression of the gradient of L(wk ) with respect to wk is found to be

K (k)  ∂ L(wk ) = 2λkk wk + λk j w j . ∂wk j=1, j=k

(2.11)

By gathering equations 2.10 and 2.11, we thus obtain    K (k)  g (|yk |)  ∂ J (wk ) =E λk j w j . yk x + 2λkk wk + ∂wk |yk | j=1, j=k

(2.12)

790

S. Fiori

The standard elimination process may be performed in order to get rid of the dependence on the Lagrange multipliers. It consists of solving the following set of equations for λk j , after having fixed the variable k,  wrH

∂ J (wk ) ∂wk

 = 0 , r = 1, . . . , m ,

(2.13)

under the constraint of orthonormality of the vectors wk expressed by equations 2.7. Tedious but straightforward calculations give, after footer renaming, the result 

opt

λk j = −E

g  (|yk |) yk yj |yk |



1 1 − δk j 2

 ,

(2.14)

where δkh denotes Kronecker’s delta. It would be interesting to observe that the multipliers do not enjoy any symmetry properties unless g  (u) = u. In opt opt this special case, in fact, (λk j ) = λ jk would hold. By plugging expressions 2.14 into equation 2.12, the formula for the optimal gradient of J is readily found: 

∂J ∂wk



opt =E


 K (k)  g  (|yk |)  y x− yj w j . |yk | k j=1

(2.15)

Now the optimal gradient may be used in a steepest-ascent algorithm to design a neural optimizing system. In short, by defining the quantities K (k) 

dg(u) 1 du u

(2.16)

dwk = Pk E[G(|yk |)yk x] , k = 1, 2, . . . , m . dt

(2.17)

def

Pk = I p −

def

w j w Hj and G(u) =

j=1

with u ∈ R+ , the new learning rule is

The factor E[G(|yk |)yk x] may be interpreted as a complex-valued extended Hebbian term common to each neuron, while the projector Pk is a deflating term that drives each weight vector wk into a different subspace.3 It is worth noting that by choosing as g(·) the function g(u) = 12 u2 , that means having g  (u) = u, and thus G(u) = 1, the gradient steepest-ascent 3

The term projector is a bit extended here: unless the vectors wk become orthonormal, the operator Pk does not represent a true projection because it is not idempotent, that is, P2k = Pk .

Nonlinear Complex-Valued Extensions of Hebbian Learning

791

learning equations for a continuous-time PSA/PCA neural network with m outputs become


 K (k)  dwk  yj w j = E yk x − , k = 1, 2, . . . , m . dt j=1

(2.18)

When K (k) = k, it coincides with the learning rule proposed by DeCastro et al. (1998), which rewrites compactly as

d W = E xx H − UT(W H xx H W) W , dt where UT(·) returns the upper-triangular part of the matrix contained within. If, furthermore, x ∈ R p , then the DeCastro-DeCastro-Amaral-Franco rule coincides with Sanger’s learning paradigm (Sanger, 1989). In the generalized case, the function g(·) is assumed different from quadratic, usually by the help of robust statistics theory (Karhunen & Joutsensalo, 1995). An alternative choice of this function will be discussed in section 2.3. 2.2 Output Decorrelation by Lateral Inhibition. Kung and Diamantaras realized a Hebbian learning network with Rubner-Tavan’s laterally connected linear neural network (Rubner & Tavan, 1989), endowed with the unsupervised APEX learning rule (Diamantaras & Kung, 1996). The input-output network equations are y = z + HH y , z = WH x ,

(2.19)

where x ∈ C p , y, z ∈ Cm , W ∈ C p×m , and H ∈ Cm×m . Note that H is strictly upper triangular; thus, the network is hierarchical (but not recurrent). It is in fact worth noting that the network output signal yk may be expressed as yk = wkH x + hkH y ,

(2.20)

for k = 1, . . . , m, with wk ∈ C p and hk ∈ Cm . An exemplary laterally connected network of this kind is depicted in Figure 1. As the network is hierarchical, the vectors hk enjoy the property h k j = 0 for j ≥ k; thus, for instance, h1 = [0 0 · · · 0 0]T , h2 = [h 21 0 · · · 0 0]T , h3 = [h 31 h 32 0 · · · 0 0]T , and so forth. This implies that the quantity hkH y does not depend on wk , but only on w j with j < k. Kung and Diamantaras proved that under reasonable conditions, in the real-valued case, the APEX rule makes the column vectors of the direct connection weight matrix W asymptotically converge to the first principal eigenvectors of the covariance matrix of the input signal, and the lateral connection weight matrix H asymptotically vanishes to zero.

792

S. Fiori

Direct connections weights

Lateral connections weights

Figure 1: An exemplary network exhibiting laterally connections.

Later, Chen and Hou (1992) extended the APEX algorithm to perform PCA of complex-valued random signals. They proved experimentally that the complex-valued APEX algorithm actually allows extracting a number of principal components from a complex-valued signal. The aim of this section is to give an alternative explanation of the complex-valued APEX learning rules, based on gradient optimization and to extend this result to Hebbian learning. In order to design a PCA-type learning procedure for the laterally connected network illustrated in Figure 1, let us define the following criterion function: def

Nk (W, H) = E[ f (wkH x)] + hkH E[yy H ]hk .

(2.21)

The first term on the right-hand side contains the generalized power of the transformed signal zk = wkH x, where again f (ζ ) = g(|ζ |), ζ ∈ C, while the second term contains a linear combination of the cross-correlation values of the outputs. (Generalized power refers to the expectation of a nonquadratic function of network’s output.) By definition of PCA, the first term has to be maximized under the constraint wkH wk = 1, while the second term must be zeroed. Generalized power maximization and decorrelation of output signals may be thought of as separate objectives. These targets may be attained through the following objective function: def

P(W, H) =

m   k=1

 Nk (W, H) + λk (wkH wk − 1) + E[ψk ](hkH hk ) ,

(2.22)

Nonlinear Complex-Valued Extensions of Hebbian Learning

793

where the functions λk and E[ψk ] are Lagrange multipliers. The choice of denoting with E[ψk ] the Lagrange multipliers corresponding to the constraints on the lateral connection strengths is motivated by the observation that in the considered learning equations, the optimal values of multipliers appear as the statistical expectation of some functions; thus, the adopted notation inherently embodies such features (Fiori & Piazza, 2000). Also, this choice makes the notation easier.  H It deserves to be noted that the terms m k=1 E[ψk ](hk hk ) add degrees of freedom to the learning system. Such terms ensure zero lateral forcing at convergence (i.e., hkH hk = 0), and the presence of the free functions ψk affects the learning dynamics and can speed up the convergence of the network (Costa & Fiori, 2001; Fiori & Piazza, 2000) without changing the set of stationary points of the algorithm. The direct connection W adaptation aims at maximizing the power of the transformed signal by maximizing the objective function, equation 2.22, with respect to W only. In order to adapt each wk , the gradient steepestascent rule shown in equations 2.1 and 2.2 may be used. By reasoning as in the previous section for computation of multipliers, the optimal gradient of P with respect to wk is found to be 

∂P ∂wk

opt

= E[G(|yk |)(xyk − zk yk wk )] , k = 1, 2, . . . , m ,

(2.23)

with G(·) being defined as in section 2.1. Then the lateral connection weight matrix H is used only in order to minimize the cost function defined in equation 2.22 in order to decorrelate the network’s outputs. By defining the real part and the imaginary part of def (R) def (R) (I ) (I ) yk and hk as yk = yk + i yk and h = hk + ihk , with some mathematical work we find: (R)

∂ yk

(R) ∂hk (R)

∂ yk

(I ) ∂hk

(I )

(R)

= y[k] , (R)

(R) def

(R) (I )

(R) ∂hk

(I )

= y[k] ,

(I )

= y[k] ,

where y[k] = [y1

∂ yk ∂ yk

(I ) ∂hk (R)

y2

(I )

= −y[k] , (R)

(I ) def

(I )

(I )

· · · yk−1 0 · · · 0]T , y[k] = [y1 y2

(I )

· · · yk−1 0 · · · 0]T ,

(R) def

with k > 1 and y[1] = y[1] = [0 · · · 0]T . Hence, we find

∂ Nk = 2E y[k] yk , ∂hk def

(2.24) def

where y[k] = [y1 y2 · · · yk−1 0 · · · 0]T with k > 1 and y[1] = [0 0 · · · 0 0]T . The function P may be iteratively minimized, with respect to the variable

794

S. Fiori

matrix H, by means of a gradient steepest-descent rule, where the gradient of P with respect to hk assumes the expression

∂P = 2E y[k] yk + ψk hk , k = 1, 2, . . . , m . ∂hk It is interesting to note that in view of optimization, there are no theoretical reasons to force functions ψk to assume any particular value (Costa & Fiori, 2001; Fiori & Piazza, 2000). In other terms, unless further constraints are established in the optimization of function 2.22, it is impossible to find optimal values for the multipliers E[ψk ]. On the basis of the previous calculations, the obtained nonlinear complex-valued learning rules for output decorrelation by lateral inhibition read:  dwk   = E[G(|yk |)(xyk − zk yk wk )] , k = 1, 2, . . . , m ,  dt

 dh   k = −E y[k] yk − ψk hk , k = 1, 2, . . . , m . dt

(2.25)

It could be interesting to discuss two possible choices of the functions ψ(·)’s and to relate the obtained learning rules to learning paradigms known from the scientific literature. The first one consists in assuming null the free functions ψk . In order to find a correspondence in the scientific literature with a related learning rule, we consider the classic (quadratic optimization) case, in which the function g(u) is assumed equal to 12 u2 and real-valued signals are dealt with. Then, in the hypothesis that for t large enough, the lateral forcing has vanished (hk ≈ 0), because the neurons have learned to encode uncorrelated features, from the first of equations 2.19, we see that zk ≈ yk . Consequently, the learning rules 2.25 can be approximated by  dwk   = E[yk (x − yk wk )] , k = 1, 2, . . . , m ,  dt  dh   k = −E[y[k] yk ] , k = 1, 2, . . . , m , dt

(2.26)

known as Rubner-Tavan’s learning equations (Rubner & Tavan, 1989). It is easy to recognize a principal component rule for the direct connections and an anti-Hebbian rule for the lateral connections, in the spirit of Hebbian and anti-Hebbian learning recalled in section 1. The second choice consists in assuming ψk = |yk |2 . Again in the quadratic optimization hypothesis and under the asymptotic assumption zk ≈ yk , the

Nonlinear Complex-Valued Extensions of Hebbian Learning

795

model 2.25 closely resembles the learning algorithm by Chen and Hou (1992) (which coincides with the Kung-Diamantaras learning rule in presence of real-valued signals):  dwk   = E[yk (x − yk wk )] , k = 1, 2, . . . , m ,  dt   dhk = −E[y (y + y h )] , k = 1, 2, . . . , m .  k k k [k] dt

(2.27)

It deserves to be noted that the learning equations for wk in equation 2.25 differ from the corresponding Chen-Hou equations because of the presence of the term zk yk instead of yk yk = |yk |2 . This difference may be nonnegligible. The quantity zk yk is a complex number; thus, it embodies a phase factor that |yk |2 does not possess. This difference disappears when m = 1. In this case, the equations coincide with Oja’s first principal component analyzer in the complex domain. 2.3 The Sudjianto-Hassoun Principle and Its Extension to the Complex-Valued Case. In sections 2.1 and 2.2, nonquadratic criteria optimization was dealt with in order to define extended Hebbian learning rules. Here we aim at briefly recalling the Sudjianto-Hassoun interpretation of Hebbian learning and extending this theory to the complex-valued case, because it naturally leads to a possible choice of nonquadratic criterion for learning. Also, an exemplary application to complex-valued independent component analysis will help in clarifying the usefulness of the SudjiantoHassoun principle in extended Hebbian learning. Sudjianto and Hassoun (1995) considered the problem of maximizing def a criterion J (w) = E[S2 (wT x)] subject to the restriction wT w = 1, where T y = w x is the output of a single-unit neural network and S(·) is a generic saturating sigmoidal function, for instance, such that S(y) ∈ [−1, +1]. They noted that maximizing the variance of a saturating function of y leads the model neuron to prefer configurations w that correspond to having the values of S(y) concentrated around the extremes −1 and +1. If the quantity def v = S(y) is perceived as a new random variable with probability density function q V (v|w), q V (v) in short notation, this corresponds to having this distribution U shaped (Sudjianto & Hassoun, 1995). The gradient steepestascent learning rule for this abstract neuronal system is dw = (I − wwT )E[(y)x] , dt def

(2.28)

¯ the probability denwhere (u) = 2S (u)S(u). Let us denote now by q Y (y|w) ¯ and with sity function of the random variable y due to a configuration w

796

S. Fiori

¯ its cumulative distribution function, namely: QY (y|w) def



¯ = QY (y|w)

y −∞

¯ q Y (u|w)du .

¯ ¯ − 1. In this case, it is well known Let us also assume S(y) = 2QY (y|w) (Sudjianto & Hassoun, 1995) that the variable v will be uniformly distributed within [−1, +1]. The central idea developed by Sudjianto and Hassoun is that the learning rule, equation 2.28, will converge to a weight vector surely ¯ since the rule seeks a U-shaped distribution of v, that is, a different from w, distribution that deviates from a uniform one. In order to extend the Sudjianto-Hassoun principle to the complexvalued case, let us consider the cost function def

U(w) = E[S2 (|y|)] ,

(2.29)

for a complex-weighted neuron with output y = w H x, with x and w belonging to C p . Its gradient steepest-ascent maximization under the constraint w H w = 1 yields the learning rule   dw y H = (I − ww )E (|y|) x , dt |y|

(2.30)

which closely recalls equation 2.17 for m = 1. The learning rule 2.30 drives function S(|y|) to take on values around its extremes, for instance, +1; therefore, it seeks the configurations w, making the probability density function q V (v) different from a uniform one. Now we may consider for S(|y|) a function like def

Q|Y| (|y|) =

 0

|y|

q |Y| (u)du .

By equating equation 2.30 to 2.17, with m = 1, the relationship between q |Y| and g  is found to be g  (|y|) = 2q |Y| (|y|)



|y| 0

q |Y| (u)du .

(2.31)

The concept of Sudjianto-Hassoun interpretation of Hebbian learning has been just touched on here. An extensive analysis of this interesting and fruitful theory has been recently proposed in Fiori (2003c) in the context of complex-valued independent component analysis. In summary, independent component analysis allows extracting statistically independent source signals from their linear combinations where the

Nonlinear Complex-Valued Extensions of Hebbian Learning

797

mixing operator is unknown and the temporal dynamic of the source signals is also unknown. Another useful interpretation of ICA techniques concerns their feature extraction ability, which offers a mathematically sound way of decomposing signals that exhibit involved dynamics into independent basis signals (Cichocki & Amari, 2002; Hyv¨arinen, Karhunen, & Oja, 2001). Formally, the simpler source observation model is written as x = As, where s ∈ C p denotes the vector of source signals, x ∈ C p denotes the vector of observed signals, and A ∈ C p× p denotes the matrix of mixing coefficients. In the above (noiseless, instantaneous) case, the matrix A may be assumed orthonormal without loss of generality. In fact, it is known that if A is not orthonormal, it is possible to preprocess the observed data via a so-called whitening algorithm that makes it possible to separate out the independent components via a rotation only (Cichocki & Amari, 2002; Hyv¨arinen et al., 2001). In this case, a linear artificial neural network described by y = W H x with W ∈ C p× p orthonormal may be effective in separating the independent source signals out from the observed mixtures. In the case that the sources are typical base-band digital signals used in telecommunications, such as QAM or PSK (QAM stands for quadrature amplitude modulation and PSK for phase shift keying; (Haykin, 1989), we may give a quite appealing interpretation of the way the SudjiantoHassoun principle works. In fact, in the above-mentioned cases, the probability density functions of the generic source signal sk modulus are written as q |sk | (|sk |) =



q |sk |,r δ(|sk | − s¯k,r ) ,

(2.32)

r

where the constant s¯k,r denotes the r :th possible discrete value assumed by the quantity |sk |, the symbol δ(·) denotes Dirac’s delta, and the quantities q |sk |,r denote the probability that |sk | = s¯k,r . Now, the probability density function of each observation is different from peaked (i.e., more similar to a uniform one), so the aim of the separating network is to make the output distributions of the artificial neural network as peaked as possible. This behavior is readily recognized as extending the “make the output distribution be as U shaped as possible” to “make the same distribution as peaked as possible.” These informal considerations offer further justification for the choice of learning rules based on the optimization of criterion functions that depend on the network output moduli only. The above informal considerations have been formall investigated in Bingham and Hyv¨arinen (2000) and Fiori (2000, 2003c). Some contributions on the use of nonclassical principal component techniques to blind source separation have been recently summarized in Fiori (2003c) and the choice of the involved nonlinear functions has been

798

S. Fiori

discussed, for instance, in Hyv¨arinen (1999) and Mathis and Douglas (2002). The main weaknesses of these approaches seem to be:

r r r

The nonlinear functions are generally chosen on the basis of existing contributions from robust statistics and on heuristic observations and findings. Their choice generally relies on the fact that using nonlinear functions adds high-order statistical features to the system but gives little insight into how these features are related to the separation problem. Because of the strong nonlinearity of the learning equations, it is difficult to give any analytical proof of convergence or detailed studies about the features of the employed learning criteria.

In order to illustrate the usefulness of Sudjianto-Hassoun principle in extended Hebbian learning for independent component analysis and to clarify the above calculi with an example, let us suppose that input x contains a complex-valued orthogonal mixture of statistically independent signals (Desodt & Muller, 1990; Laheld & Cardoso, 1994) and that one of these signals is a gaussian noise of the form ν = ν (R) + iν (I ) , where ν (R) and ν (I ) are zero-mean gaussian random variables with variance σ 2 . Then it is known that the modulus |ν| follows the Rayleigh distribution,   u u2 q R (u) = 2 exp − 2 (u) , σ 2σ where (u) is the unit step function. Then by formula (2.31), we find  (u) = gR

     u2 u2 2u exp − − exp − , u≥0. σ2 2σ 2 σ2 

Figure 2 depicts the Rayleigh warping function g u(u) for a unit power noise. In this case, it is possible to express the cumulative distribution function as 

u2 QR (u) = 1 − exp − 2 2σ

 ,

whose shape is illustrated in Figure 2. It is interesting to note that the obtained shape for the neuron model’s nonlinear function QR (u) formally differs from the classical hyperbolic-tangent one, even if its shape resembles the usual saturating sigmoid. Also, in this case, the sigmoid’s shape parameter σ has a clear physical meaning. In order to illustrate how the above theory works in practice, we consider a simple 4 × 4 complex-valued independent component analysis case. Figure 3 shows the source signals, s1 , s2 , s3 , and s4 —namely, three QAM16

Nonlinear Complex-Valued Extensions of Hebbian Learning

799

1

0.9

0.8

0.6

R

g’(u)/u , Q (u)

0.7

0.5

0.4

0.3

0.2

0.1

0

0

0.5

1

1.5

2

2.5 u

3

3.5

4

4.5

5



Figure 2: Rayleigh warping function g u(u) (solid line) and the corresponding cumulative function QR (u) (dashed line) for σ = 1.

signals and a complex gaussian noise of standard deviation σ = 0.5—and their four linear superpositions x1 , x2 , x3 , and x4 . The same figure also shows the histograms of the moduli |sk | and |xk |. It is clear that as the QAM16 symbols are discrete by definition, their histograms are peaked. This simple rule does not apply, of course, to the gaussian noise. Conversely, the histograms of the observations xk are distributed. Figure 4 shows the whitened signals, denoted here by v1 , v2 , v3 , and v4 , and the estimated source signals obtained through the neural learning algorithm obtained by the extended Hebbian learning rule described in this section, denoted by y1 , y2 , y3 , and y4 . The histograms of the moduli of the complex-valued signals y1 , y2 , y3 look close to peaks, as predicted by the theory. The main contribution of the Sudjianto-Hassoun principle-based learning theory to ICA is to allow designing the proper structure of the nonlinear part of neurons provided that some information is known about the statistical structure of signals that the source signals differ from. This principle may prove very useful in blind signal processing, where the statistical structure of the involved signals is not known in advance, but the statistical structure

800

S. Fiori

Modulus histogram

Signal s

Signal s

40

Imaginary

20

100 0

0

1

0

Signal s

2

3

1

1

1

1

0

0

0

0

−1

−1 −1

Imaginary

4

200

1

0

−1 −1

1

0

0

10

10

0

0

0

0

10

−20 −20

0

20

−10 −10

0

10

−10 −10

200

200

200

200

100

100

100

100

0

0

10 Signal x1

20

0

0

10 Signal x2

20

0

1

−1

1

20

0

0

2

−1 −1

1

10

−10 −10 Modulus histogram

Signal s

2

300

0

10 Signal x3

20

0

0

0

1

0

10

10 Signal x3

20

Figure 3: (Second row) Source signals s1 , s2 , s3 , and s4 (three QAM16 signals and a complex gaussian noise). (Third row) Their four linear superpositions x1 , x2 , x3 , and x4 . (First row) Histograms of the moduli |sk |. (Fourth row) Histograms of the moduli |xk |.

of signals that differ from the useful ones, such as noises, might be easily known. 2.4 Analysis of the One-Unit Complex-Valued PCA/MCA. The learning rule in equation 2.17 for a single neuron in the quadratic complex-valued case particularizes to the adapting law: dw = E[y x − |y|2 w] , dt

(2.33)

with x and w belonging to C p . By defining the Hermitian covariance matrix def  = E[xx H ], this differential equation rewrites as dw = w − (w H w)w . dt

(2.34)

Imaginary

Nonlinear Complex-Valued Extensions of Hebbian Learning 2

2

2

2

1

1

1

1

0

0

0

0

−1

−1

−1

−1

Imaginary

−2 −2

0 Re

2

−2 −2

0 Re

2

−2 −2

0 Re

2

−2 −2

2

2

2

2

1

1

1

1

0

0

0

0

−1

−1

−1

−1

−2 −2

Histogram

801

0 Re

2

−2 −2

600

600

400

400

0 Re

2

−2 −2

0 Re

2

800

−2 −2

0 Re

2

0 Re

2

2 |y4|

4

300

600

200

400 200 0

200

0

1 |y1|

2

0

100

200 0

1 |y2|

2

0

0

1 |y3|

2

0

0

Figure 4: (Top row) Whitened observations v1 , v2 , v3 , and v4 . (Middle row) Estimated source signals y1 , y2 , y3 , and y4 obtained through the extended Hebbian learning rule enriched by the Sudjianto-Hassoun principle. (Bottom row) Histograms of the moduli |yk |.

This system has a set of stationary points defined by def

E∗ = {w ∈ C p : w − σ w = 0} .

(2.35)

With the exception of the trivial solution w = 0, the elements w ∈ E∗ coincide to the eigenvectors of . The following proposition states the convergence of the system 2.34 to the eigenvector in E∗ corresponding to the largest eigenvalue σ . The proof of this theorem follows from the arguments successfully used in the realvalued case by other authors (Oja, 1992; Sanger, 1989). Theorem 1. Let us suppose  Hermitian in equation 2.34 with eigenpairs (σ 1 , q 1 ), (σ 2 , q 2 ), . . . , (σ p , q p ). Suppose then that eigenvalues are distinct and

802

S. Fiori

arranged in descending order, and eigenvectors are normalized so that qkH qk = 1 and w(0) H q 1 = 0. Then there holds lim w(t) = q 1 e iα ,

t→+∞

where α is arbitrary in [0, 2π]. Proof. Let us expand vector w(t) by means of the system’s eigenbasis (Haykin, 1998; Sanger, 1989), which means writing: w(t) = θ1 (t)q1 + θ2 (t)q2 + · · · + θ p (t)q p ,

(2.36)

where scalar functions θk (t) ∈ C are termed principal modes. Plugging equation 2.36 in equation 2.34 yields p  dθh (t) h=1

dt

 p p p p     H θh (t)qh − [θk (t)qk ]  [θk (t)qk ] θh (t)qh . qh = h=1

k=1

k=1

h=1

By recalling property qk = σk qk , we have
p  p p p m     dθh (t)  H θh (t)qh σh − θk (t) θr (t)σr qk qr θh (t)qh . qh = dt h=1 h=1 k=1 r =1 h=1 Now from the fact that qkH q = δk , it follows that the differential equations for the h th principal modes, with h ≥ 2, read θ˙ h (t) = θh (t)σh −

p 

|θk (t)|2 σk θh (t) , h = 2, . . . , p ,

(2.37)

k=1

while it is particularly useful to write in a separate equation the differential law pertaining to the first principal mode θ1 (t), that is, θ˙ 1 (t) = θ1 (t)σ1 − |θ1 (t)|2 θ1 (t)σ1 −

p 

|θk (t)|2 σk θ1 (t) .

(2.38)

k=2

Principal modes θh (t) are complex-valued quantities, and their real and (R) (I ) imaginary parts are denoted here as θh (t) and θh (t), respectively. Our aim is now to solve differential system 2.37 + 2.38, which is a coupled nonlinear system of differential equations. In order to decouple the nonlinear differential subsystem 2.37, let us define the new functions: (R)

def

φh (t) =

(R)

θh (t) (R)

θ1 (t)

(I )

def

, φh (t) =

(I )

θh (t) (I )

θ1 (t)

, h = 2, . . . , p .

Nonlinear Complex-Valued Extensions of Hebbian Learning

803

For the real part of the h:th principal mode, there holds (R) (R) (R) (R) θ˙ (t)θ1 (t) − θh (t)θ˙ 1 (t) d (R) , φh (t) = h (R) dt θ (t)2

(2.39)

1

and the system 2.37+2.38, particularized for the real parts, reads: (R) (R) θ˙ h (t) = θh (t)σh −

p 

(R)

(2.40)

(R)

(2.41)

|θk (t)|2 σk θh (t) , h = 2, . . . , p .

k=1 (R) (R) θ˙ 1 (t) = θ1 (t)σ1 −

p 

|θk (t)|2 σk θ1 (t) .

k=1

By using equations 2.40 and 2.41 within equation 2.39, direct calculations (R) show that the dynamics of the state variables φh (t) looks like (R) (R) φ˙ h (t) = (σh − σ1 )φh (t) , h = 2, . . . , p . (I )

A similar formula holds for φh (t). Thus, we have proven that θh (t) = φh (0)e (σh −σ1 )t θ1 (t) ,

(2.42)

θh (t) = φh (0)e (σh −σ1 )t θ1 (t) .

(2.43)

(R)

(I )

(R)

(R)

(I )

(I )

The above formulas show that subsystem 2.37 has been successfully decoupled, since the dynamics of each principal mode no longer depends on the other modes, apart from the first one. It is useful to note that from equations 2.42 and 2.43, the expression below follows: (R)

(I )

|θk (t)|2 = θk (t)2 + θk (t)2 = e 2(σk −σ1 )t [φk (0)2 θ1 (t)2 + φk (0)2 θ1 (t)2 ] . (R)

(R)

(I )

(I )

Now the aim is to solve the differential equation 2.38 for θ1 (t). It is worth noting that it slightly simplifies if the following variable change is performed: θ1 (t) = c(t)e σ1 t , c(t) ∈ C . After this change, we have |θ1 (t)|2 = |c(t)|2 e 2σ1 t , (R)

(I )

|θh (t)|2 = e 2σk t [φh (0)2 c (R) (t)2 + φh (0)2 c (I ) (t)2 ] ,

804

S. Fiori

where c (R) (t) and c (I ) (t) stand, respectively, for the real part and the imaginary part of the function c(t). Then differential equation 2.38 becomes c(t) ˙ = − |c(t)|2 e 2σ1 t σ1 c(t) −

p 

(R)

(I )

e 2σk t [φk (0)2 c (R) (t)2 + φk (0)2 c (I ) (t)2 ]σk c(t) .

(2.44)

k=2

By defining the auxiliary quantities, def

η(R) (t) = −

p 

(R)

(2.45)

(I )

(2.46)

e 2σk t σk φk (0)2 − σ1 e 2σ1 t ,

k=2 def

η(I ) (t) = −

p 

e 2σk t σk φk (0)2 − σ1 e 2σ1 t ,

k=2

the differential equations for the real and the imaginary parts of c(t) rewrite compactly: 

c˙ (R) (t) = η(R) (t)c (R) (t)3 + η(I ) (t)c (I ) (t)2 c (R) (t) , c˙ (I ) (t) = η(I ) (t)c (I ) (t)3 + η(R) (t)c (I ) (t)c (R) (t)2 .

(2.47)

Since η(R) (t) < 0, η(I ) (t) < 0 and at least c(0) = w(0) H q1 = 0, two cases should be considered. CASE 1: Only one of the terms c (R) (0) and c (I ) (0) differs from zero. In this case, system 2.47 simplifies into c(t) ˙ = η(t)c(t)3 , where η(t) stands for η(R) (t) or η(I ) (t). The above differential equation may be solved by 

c(t)

c(0)

dc = c3



t

1 1 = −2 2 c(t) c(0)2

η(τ )dτ ⇒

0



t

η(τ )dτ .

0

This readily leads to 1 e −2σ1 t = − 2e −2σ1 t 2 θ1 (t) θ1 (0)2



t

η(τ )dτ .

0

/ From definitions 2.45 and 2.46, it can be seen that under condition σ1 ∈ {σ2 , . . . , σm }, there holds lim 2e −2σ1 t

t→+∞

 0

t

η(τ )dτ = −1 .

(2.48)

Nonlinear Complex-Valued Extensions of Hebbian Learning

805

This implies the conclusion lim

t→+∞

1 =1. θ1 (t)2

(2.49)

CASE 2: c (R) (0) = 0 and c (I ) (0) = 0. The hypotheses imply c (R) (t) = 0 and c (I ) (t) = 0. Thus, it is possible to multiply both sides of equations 2.47 by a factor 2/c (R) (t) and 2/c (I ) (t), respectively. This way we have: d log c (R) (t)2 d log c (I ) (t)2 = = 2[η(R) (t)c (R) (t)2 + η(I ) (t)c (I ) (t)2 ] . (2.50) dt dt The first equality in the chain tells us that log c (R) (t)2 = log c (I ) (t)2 − log κ, where κ is a positive constant determined by the initial conditions. Equivalently, we have c (I ) (t)2 = κc (R) (t)2 . This means that equation 2.50 can be easily solved, in that there holds d log c (R) (t)2 = 2[η(R) (t) + κη(I ) (t)]c (R) (t)2 . dt The latter equation is equivalent to c˙ (R) (t) = [η(R) (t) + κη(I ) (t)]c(t)(R) (t)3 . Again we have: 1 (R)

θ1 (t)2

=

e −2σ1 t (R)

θ1 (0)2

− 2e −2σ1 t



t

[η(R) (τ ) + κη(I ) (τ )]dτ ,

(2.51)

0 (I )

θ1 (0)2 1 1 1 = , κ = . (R) |θ1 (t)|2 1 + κ θ (R) (t)2 θ1 (0)2 1

(2.52)

By using twice the result 2.48, it may be shown that lim 2e −2σ1 t

t→+∞



t

[η(R) (τ ) + κη(I ) (τ )]dτ = −(1 + κ).

0

Thus, we arrive at the conclusion lim

t→+∞

1 =1. |θ1 (t)|2

The results 2.49 and 2.53 also imply lim |θk (t)| = 0 , k = 2, 3, . . . , p .

t→+∞

(2.53)

806

S. Fiori

It is worth mentioning that the assumption that all eigenvalues are distinct, while reasonable, for example, in signal processing tasks, is not crucial. The crucial assumption is that the first eigenvalue differs from the others, as suggested by the fact that the result, equation 2.49, holds under condition σ1 ∈ / {σ2 , . . . , σm }. In this case, a proof of a similar consistency result may be based on the convergence proof for the real-valued case provided in theorem 3.3 of Helmke and Moore (1993). When the function g(·) defined in equaiton 2.6 is not quadratic, it is likely that the neural unit just studied behaves in a different way from Oja’s neuron. A study on what happens in the general nonquadratic case for a real-weighted neuron has been carried out by Oja (1994). The analysis is extended here to the complex case. To begin, let us define the following quantities, 

g  (|y|)  L(w) = E y x = E G(|w H x|)(xx H ) w , |y| def



def

G(w) = E[g  (|y|)|y|] = E[G(|w H x|)|w H x|2 ] ,

(2.54) (2.55)

with G(u) defined again as in section 2.1. Note that L(w) ∈ C p , while G(w) ∈ C. Then learning rule 2.17 rewrites as dw = L(w) − wG(w) , dt

(2.56)

as in Oja (1994). The stationary points of system 2.56 are among the elements of the set: def

W∗ = {w ∈ C p : L(w) − wG(w) = 0} .

(2.57)

Let us suppose g  (u) has null even derivatives at the origin, as in Oja (1994). Thus, the MacLaurin expansion of G(u) looks like 1 G(u) = g (2) (0) + u2 g (4) (0) + higher-order terms (h.o.t.). 6 Replacing the expanded function G(u) into definitions 2.54 and 2.55 gives 1 L(w) = wg (2) (0) + w H E[(xx H )w(xx H )]wg (4) (0) + h.o.t. , 6 1 G(w) = w H wg (2) (0) + E[|w H x|4 ]g (4) (0) + h.o.t. , 6 where again  denotes the network input covariance matrix.

Nonlinear Complex-Valued Extensions of Hebbian Learning

807

Therefore, a neuron model with a sigmoidal activation function G(·) instead of a linear one introduces higher-order statistics in learning equations. This can be seen by looking at the set of stationary points W∗ that only in a first-order approximation coincides with E∗ , provided that the |g (n) (0)| are small enough for n > 2. Therefore, in general, the vectors in W∗ deviate from the principal directions in E∗ and thus also depend on higher-order statistical structures than covariance. A widely known problem in the scientific literature is that extending PCA theory to minor component analysis theory is not a straightforward task. This is true even in the complex domain case. In particular, it is not possible to replace maximization of criterion 2.4 J (w) = U(w) + L(w) with respect to w, with its minimization, in that the resulting learning rule, dw ∂J =− = E[−y x + |y|2 w] , dt ∂w

(2.58)

is unstable. The proof of the following proposition follows from the proof of the corresponding real-valued case presented in Fiori (2003b) and Oja (1992). Theorem 2. Let  ∈ C p× p be a Hermitian matrix with eigenpairs (σ 1 , q 1 ), (σ 2 , q 2 ), . . . , (σ p , q p ). Let us suppose eigenvalues are distinct and arranged in descending order, and eigenvectors are normalized so that qkH qk = 1 . Then ˙ = −w + (w H w)w with w(0) H q1 = 0 becomes unstable in finite system w time. Proof. The proof essentially follows that of theorem 1. Let us consider the principal modes dynamics: θ˙ h (t) = −θh (t)σh +

p 

|θk (t)|2 σk θh (t) , h = 1, . . . , p − 1 ,

k=1

θ˙ p (t) = −θ p (t)σ p +

p−1 

|θk (t)|2 σk θ p (t) + |θ p (t)|2 θ p (t)σ p .

k=1

Let us now define the pseudo-modes, (R)

def

φh (t) =

(R)

θh (t) (R)

θ p (t)

(I )

def

, φh (t) =

(I )

θh (t) (I )

θ p (t)

, h = 1, . . . , p − 1 ,

where superscripts (R) and (I ) denote, as usual, the real part and the imaginary part of a complex-valued variable. By means of the pseudo-modes, it

808

S. Fiori

is possible to show that θh (t) = φh (0)e −(σh −σ p )t θ p(R) (t) , (R)

(R)

θh (t) = φh (0)e −(σh −σ p )t θ p(I ) (t) . (I )

(I )

Introducing the complex-valued auxiliary function c(t) so that θ p (t) = c(t)e−σ p t , it is possible to find the resolving differential equation: c(t) ˙ = |c(t)|2 e −2σ p t σ p c(t) −

p−1 

e −2σk t [φk (0)2 c (R) (t)2 + φk (0)2 c (I ) (t)2 ]σk c(t) . (R)

(I )

k=1

Again, it is useful to define functions: def

η(R) (t) =

p−1 

e −2σk t σk φk (0)2 + σ p e −2σ p t , (R)

k=1 def

η(I ) (t) =

p−1 

e −2σk t σk φk (0)2 + σ p e −2σ p t . (I )

k=1

By means of these functions, it is straightforward to show that the real part and the imaginary part of c(t) satisfy system 2.47. Thus, the solution has the form 2.52. Now the time integrals of η(R) and η(I ) are explicitly required. The first one is  2e

2σ p t

t

η(R) (τ )dτ = e 2σ p t

0

p−1 

φk (0)2 (e −2σk t − 1) + 1 − e 2σ p t . (R)

k=1

The second integral has a similar form; thus, we may ultimately write 1 = B − A(t)e 2σ p t , |θ p (t)|2 where B is a positive constant and A(t) is an increasing function of the time. Therefore, depending on initial conditions, a finite time t¯ exists such that: 1 = 0. |θ p (t¯)|2 Oja (1994) proposed an algorithm allowing for the extraction of generalized minor component analysis from real-valued signals that overcomes the stability problem. The aim of this part is to formalize the stabilization method within the optimization framework and to extend the theory to the complex case.

Nonlinear Complex-Valued Extensions of Hebbian Learning

809

Let us consider, once again, the problem of minimizing the criterion def

C(w) = E[g(|w H x|)] + λ(w H w − 1) ,

(2.59)

with respect to the weight vector w. The expression for its gradient is ∂C = E[G(|y|)y x] + λw. ∂w

(2.60)

∂C to zero and Thus, the optimal multiplier may be found by setting w H ∂w by solving the resulting equation under the constraint w H w = 1, that is, by solving

 wH

∂C ∂w

opt = E[G(|y|)|y|] + 2λopt (w H w) = 0 .

The central point is now that as optimality requires w H w − 1 = 0, the latter condition is equivalent to E[G(|y|)|y|] + 2λopt − σ¯ (w H w − 1)w = 0 , with σ¯ being an arbitrary constant. By solving the previous equation for λopt  ∂C opt and replacing the found function in the expression of ∂w , we obtain the stabilizable learning rule: dw = −E[G(|y|)(y x − |y|2 w)] − σ¯ (w H w − 1) . dt

(2.61)

When G(u) = 1, it is possible to prove that the corresponding minor component analysis learning algorithm converges to the expected solution. def

Theorem 3. Let  = E[xx H ] be the covariance matrix of the random process x(t) with eigenpairs (σ 1 , q 1 ), . . . , (σ p , q p ) and G(u) = 1 in equation 2.61. Suppose eigenvalues are distinct and arranged in descending order, and eigenvectors are normalized so that qkH qk = 1. If σ¯ > σ 1 then the state vector w of system 2.61 with w(0 ) H q p = 0 asymptotically converges toward q p up to a phase factor. ˙ = −( − σ¯ I)w + Proof. System 2.61 with G(u) = 1 can be rewritten as w ¯ def ¯ are σ¯ − σ p > w H ( − σ¯ I)ww. Define  = −( − σ¯ I). The eigenvalues of  σ¯ − σ p−1 > · · · > σ¯ − σ1 > 0, while its eigenvectors coincide to eigenvectors ¯ + (w H w)w, ¯ ˙ = w of . Thus, theorem 2.4 applies to system w allowing the conclusion that w asymptotically converges to the last eigenvector q p up to a phase factor.

810

S. Fiori 4

1.5

4

3

3 1

2

2 0.5

0

3

Im{x }

1 Im{x2}

Im{x1}

1

0

−1

0

−1 −0.5

−2

−2 −1

−3

−4 −2

−3

0

2 Re{x1}

4

−1.5 −5

0

5 Re{x2}

10

−4 −5

0 Re{x3}

5

Figure 5: Input data for the first principal and minor component analysis experiment.

It is important to note that this result gives only a sufficient condition for the stability of rule 2.61. This rule embodies a modification that is known as origin shift in linear algebra and that in this context has been shown to arise from the Lagrange multiplier method in a natural way. As a numerical example, let us consider an input random process x ∈ C3 whose covariance matrix has eigenvalues σ1 = 3, σ2 = 2, and σ3 = 1. Such a signal is illustrated in Figure 5. Running the learning rule 2.33, which allows extracting the first principal component from the data, one expects that the first principal mode (θ1 ) tends to one (up to a phase factor), while the second and third principal modes tend to zero, where the principal modes have been defined in theorem 2. These results are confirmed by the left-hand panel of Figure 6. Furthermore, we tried to run the learning rule 2.61 on the same data set in order to extract the first minor component. We tried first with σ¯ = 0, which means using the nonstabilized last minor component analyzer: Simulations show that the rule quickly becomes unstable. Then we tried with σ¯ = 1, as in Oja (1994). Even in this case, the rules look unstable. Finally, we tried to use the sufficient condition provided by theorem 3. Since the largest eigenvalue is 3, we chose σ¯ = 3.5. Simulation results are shown in

Nonlinear Complex-Valued Extensions of Hebbian Learning First principal component analysis

First minor component analysis 1

1

|θ |

|θ1|

1.2

1

0.8

0

2000

4000

0.5

0

6000

0.6

0.6

0

2000

4000

6000

0

2000

4000

6000

0

2000 4000 Iterations

6000

2

|θ |

0.8

|θ2|

0.8

0.4 0.2 0

811

0.4 0.2

0

2000

4000

0

6000

1.5

1.3 1.2 3

|θ |

|θ3|

1 0.5 0

1.1 1

0

2000 4000 Iterations

6000

0.9

Figure 6: Principal modes modules evolution. (Left) First principal component analysis case. (Right) First minor component analysis case.

the right-hand panel of Figure 6. As expected, the first two principal modes converge to zero, while the third mode approaches the value 1. 2.5 Notes on Multiunit Complex-Valued Component and Subspace Networks. By using the fundamental result found by Oja (1982) about the convergence of the neural algorithm, allowing the extraction of the first principal component from a real-valued random process x(t), Sanger (1989) was able to prove the convergence of the generalized Hebbian algorithm (GHA). In particular, Sanger proved that the differential system d W = W − UT(WT W)W , dt def

where  = E[xxT ] and W ∈ R p×m , extracts the first m eigenvectors of the covariance matrix  up to a sign factor. In the same way, on the basis of the theorem 1, it would not be difficult to show that the multiunit learning rule 2.18, in the hierarchical version allows extracting the first m complex eigenvectors up to a phase factor.

812

S. Fiori

Rule 2.17 was derived via the standard Lagrange multipliers method that is used to enforce the constraints of orthogonality on the weight vectors. However, the multipliers guarantee the constraints to be satisfied only at equilibrium, but in general not during system evolution. (A detailed analysis of the drawbacks inherent in the use of Lagrange multiplier methods for orthogonality constraints has been proposed recently in Douglas, Amari, & Kung, 2000.) In order to illustrate this effect more clearly in the case of nonlinear complex-valued, component or subspace learning, let us reformulate the orthogonality constraints by borrowing some concepts from differential geometry. (A good reference book is, Helmke & Moore, 1993.) Let us define the complex-valued compact Stiefel manifold as def

St( p, m, C) = {A ∈ C p×m |A H A = Im } . Now it is clear that the requirement that the connection pattern W at learning equilibrium is an orthonormal matrix may be equivalently formulated as W ∈ St( p, m, C). As the learning rules of interest here are expressed in k terms of differential equations of the kind dw = Fk , with Fk being a proper dt learning vector field for every abstract neuron, it is convenient to express the orthogonality constraints in terms of differential properties. This may be done by invoking the concept of tangent space of the Stiefel manifold at a point, namely: TW St( p, m, C) = {V ∈ C p×m |V H W + WV H = 0m } . In terms of single weight vectors, in order for the connection pattern to belong to the Stiefel manifold, they should satisfy 

dwk dt



H wh + wkH

dwh dt

 =0

(2.62)

for every pair (k, h) ∈ [1, ..., m]2 . In order to verify if this property is satisfied, it is sufficient to replace the expressions 2.17 in equations 2.62. This would lead to the following equations: E[G(|yk |)yk x H ]PkH wh + wkH Ph E[G(|yh |)yh x] = 0 . It is now straightforward to find that in the symmetric network case, the projector operator is such that PkH wh = 0 for every pair of indices (k, h), while in the hierarchical case, it holds that PkH wh = 0 for k ≥ h, while PkH wh = wh for k < h. As a consequence, when k < h, the left-hand side of equation 2.62 becomes E[G(|yk |)yk yh + G(|yh |)yh yk ] = 0 .

(2.63)

Nonlinear Complex-Valued Extensions of Hebbian Learning

813

From this result, it is necessary to conclude that in the symmetrical (generalized principal or minor subspace analysis) case, it is true that W(t) ∈ St( p, m, C) at any time, while in the hierarchical (generalized principal or minor component analysis) case, in general it happens that W(t) ∈ / St( p, m, C) at any time only if the algorithm is out of equilibrium. It is worth noting, however, that in any case, it is guaranteed that the optimization of the weight vectors wk happens on a unitary radius hyper-sphere. In fact, in either the symmetrical or the hierarchical case, it is obtained from k conditions 2.62 that wkH dw = 0, which implies that wk 2 is constant (at one, dt if the weight vectors are initialized to have unitary norm). Ultimately, this means that vector orthogonality is to be progressively achieved by learning and is not guaranteed at any time. This is a drawback of the Lagrange multiplier method. On the basis of general knowledge and the results granted from the preceding sections (especially section 2.4), we may summarize these drawbacks:

r

r

r

r

Generally, the method of multipliers consists in embedding the constraints under which the optimization of a criterion function should be performed in the criterion function itself by adding a properly constructed function to the original criterion. From a geometrical point of view, it seems unnatural to modify the objective function instead of restricting it to the set of feasible configurations or variables values. As a consequence of this construction, the constraints should be thought of as a sort of subtarget to be achieved in parallel to the optimization of the criterion function. Therefore, provided the employed unconstrained optimization algorithm is able to perform the optimization of the augmented criterion, the constraints are fulfilled only when the optimization is completed and not—in general—at any time. The modification of the criterion function may cause the appearance of undesired effects, such as the appearance of local extremes that might make the optimization algorithm unable to solve the learning task or might make the learning progress slower than expected. The general idea behind constrained optimization by the Lagrange multiplier method is to replace the constrained optimization flow with an unconstrained optimization flow (that takes places, for example, in the Euclidean space R p ) by properly embedding the constraints into an augmented learning criterion. It is worth noting that the space that the optimum search takes place in plays an important role in determining the behavior of the gradient-based learning system. As evidenced in section 2.4, for instance, a learning differential equation on R p may be unstable even if it was derived from strong constraints such as the normality of the weight vectors. Defining learning differential equations

814

S. Fiori

on compact manifolds (such as the Stiefel one) instead may help to overcome this serious difficulty. A detailed treatment of the Lagrange multiplier method may be found in Bertsekas (1996). In the nonquadratic case, moreover, it is difficult to give general results on learning capabilities. Some recent results for the real-valued case based on the design of appropriate Lyapunov functions for the system equivalent to equation 2.56 have been introduced in Vegas and Zufiria (2004). These results are valid in the case of homogeneous nonlinearities L(w). It is worth mentioning that for the case that nonlinear complex-valued PCA is used for blind source separation of telecommunication-type source signals with a Sudjianto-Hassoun nonlinearity, a convergence theorem was given in Fiori (2003c). Following the line of the preceding text on multiunit artificial networks, it also appears appropriate to discuss briefly the important topic of the stability of learning equations in view of computer-based implementation. It is worth recalling the distinction between the related concepts of learning differential equation and learning algorithm. A learning differential equation arises when a learning trajectory is considered in the weight space and the learnable variables of an artificial neural networks are parameterized through a free variable (normally the time) that locates a network configuration over a learning trajectory. In the general gradient-based learning setting, for instance, the learning trajectory cannot be written in explicit form, and it is specified through a differential equation (often of the first order) whose intuitive meaning is to specify the velocity field associated with learning a specific task. The solution (integration) of such a differential equation (provided an initial configuration is specified) gives rise to the network’s learning trajectory, formally termed gradient flow. A learning differential equation may be solved in closed form very rarely. Normally it is necessary to approximate the gradient flow numerically through an iterative algorithm. This corresponds to sampling the time variable and trying to obtain a close approximation of the exact learning trajectory. Then the obtained discrete-time learning equation is what it is normally referred to as a learning algorithm. The passage from a generic learning differential equation to an associated learning algorithm is not unique, as it may be performed in several different ways, and it is not consequence free, because, depending on the chosen time discretization method, the salient features of the differential equations may be retained or lost. In particular, it is worth discussing here the effect of time discretization on the two salient features of orthogonality: preservation and learning stability:

r

Euler discretization. The simplest discretization method consists in replacing the derivatives with respect to the time with incremental

Nonlinear Complex-Valued Extensions of Hebbian Learning

815

ratios. This method is widely know as the Euler integration technique. For instance, the derivative in equation 2.3 may be approximated as dw(t) w(t + T) − w(t) ≈ , dt T

r

where T denotes the width of the time interval that the derivative is approximated within. It is clear that the Euler algorithm may be well suited on a linear space, as the Euclidean manifolds C p or C p×m , while the Stiefel manifold St( p, m, C) is a curved space, and it is impossible to move from a network configuration to another configuration through vector or matrix addition. This means that the Euler method does not preserve the orthonormality of a network’s connection matrix, and it is not guaranteed that the learning system eventually converges to a configuration belonging to St( p, m, C) or that the network connection stays in the vicinity of such a manifold for a sufficiently long time, which may ultimately affect the stability of the learning algorithm. These problems have been widely investigated by Yan, Helmke, and Moore (1994) and Zufiria (2002). Fiori and Piazza (2000) showed that by properly selecting the learning step size, it could be possible to obtain a convergent learning system. Chen, Amari, and Lin (1998) and Chen and Amari (2001) also investigated the problem of the existence of a Stiefel-manifold attractor for minor component or subspace analysis algorithms. It is worth recalling that, as already noted in the section 2.4, the minor component and subspace analysis algorithms appear to be the most problematic ones with regard to dynamical stability. Geometric integration. A good discretization method in the time domain should allow converting the learning differential equations into discrete-time algorithms so as to retain (up to reasonable precision) the geometrical properties that characterize the developed learning rules. From a numerical point of view, the solution of matrix-type differential equations on curved manifolds has been widely investigated in the context of geometrical integration (see, e.g., Hairer, Lubich & Wanner, 2002, and Iserles, Munthe-Kaas, Nørsett, & Zanna, 2000), which is a recent branch of numerical analysis. The traditional effort of numerical analysis and computational mathematics has been to render physical phenomena into algorithms that can produce sufficiently affordable, precise, and robust numerical approximations. Geometrical integration is also concerned with producing numerical approximations that preserve the qualitative attributes of the solution to the extent possible. In the context of neural learning under orthogonality constraints, some efforts have been devoted to develop learning algorithms over the group of orthogonal matrices and over the Stiefel manifold. These consist of Riemannian gradient-based and non-gradient-based learning algorithms that strongly bind the connection patterns to the Stiefel

816

S. Fiori

r

manifold. (For recent reviews, refer to Celledoni & Fiori, 2004, and Fiori, 2001, 2002.) One of the advantages offered by geometrical integration over compact manifolds (such as the orthogonal group and the Stiefel manifold) is the intrinsic stability: whether the algorithm converges to the expected solution or not, the network connection matrix always keeps a bounded norm. Fixed-point algorithms. Recently, preliminary results on fast batchtype fixed-point iteration for nonlinear Hebbian learning with orthonormality constraints have been illustrated in Fiori (2003d, 2004, in press). The extension of this class of algorithms to the complex domain as well as some fundamental issues such as convergence and computational-complexity reduction are still open problems.

3 Complex-Valued Hebbian Learning by Nonquadratic Reconstruction Error Minimization A second philosophy known in the literature allowing formulating Hebbian learning involves the concept of reconstruction error minimization. Let a stationary multivariate random signal x(t) of size p, whose covariance matrix has distinct eigenvalues, be expanded by means of a basis of size m < p given by {w1 , w2 , . . . , wm }. This means that a new random signal xˆ (t) may be defined so that xˆ = y1 w1 + y2 w2 + · · · + ym wm . This is termed the econstruction formula, where the coefficients yk are defined as the projections yk = wkH x in the presence of complex-valued signals; projection expressions are termed analysis formulas. It is clear that as m < p, in general xˆ (t) = x(t), and the difference between the original and the reconstructed version of x may be conceived as the reconstruction error (Karhunen & Joutsensalo, 1995; Xu, 1992). In the following sections, we consider nonquadratic complex-valued reconstruction error minimization for a symmetrical network, which leads to a generalized principal subspace analysis theory, and the component-wise nonquadratic reconstruction error minimization principle for a hierarchical network leads to a generalized principal component analysis learning theory. We also discuss the problem of nonlinearity selection by recalling interesting statistical interpretations of nonquadratic learning criteria. The rationale for considering nonquadratic reconstruction error optimization was given by O’Leary (1990) and Liano (1996) in the artificial neural field. Liano’s contribution was limited to supervised neural learning for modeling purposes, but it remains valid for the present unsupervised (linear) data modeling treatment. In particular, Liano observed that most neural networks are trained by minimizing the mean squared error pertaining

Nonlinear Complex-Valued Extensions of Hebbian Learning

817

to the given training set. In the presence of outliers, the resulting neural model can differ significantly from the system that generated the data or from the intended abstract model. The purpose of Liano’s research was to introduce a robust approach that can minimize the influence of gross errors on the accuracy of the neural model. Two different approaches were used in Liano (1996) to study the mechanism by which outliers affect the resulting models: the influence function approach and the maximum likelihood approach. The same approaches were followed by Song et al. (1998) and are also partially followed here (in fact, the maximum likelihood approach is replaced by the maximum entropy method here). The contents of this section may be anticipated as:

r r r r

Learning principal and minor subspace by minimization of nonquadratic complex-valued reconstruction error Choosing the nonlinear criteria via the Song-Yilong-Feng interpretation of Hebbian learning, which is largely based on Liano’s research in the context of supervised multilayer perceptron learning Learning principal and minor component by component-wise nonquadratic reconstruction error minimization Illustrative numerical simulation results in the robustness of the considered learning theories

3.1 Symmetrical Nonquadratic Complex-Valued Reconstruction Error Minimization. In the symmetrical network case, the reconstruction error may be formally defined as def

e = x − xˆ = (I − WW H )x .

(3.1)

It may be shown that in the real-valued case as well as for complex-valued signals, minimizing the mean square reconstruction error leads to principal subspace analysis. The real-valued case has been investigated by Xu (1992). In the complex-valued case of interest here, this property has been analytically shown by Yang (1995), who proved the following two results: def

Theorem 4. W is a stationary point of Y(W) = E[ x − WW H x 2 ] if and only def if W = Um Q where Um ∈ C p×m contains any m distinct eigenvectors of  = H m×m E[xx ] and Q ∈ C is a unitary matrix. Theorem 5. All stationary points of Y(W) are saddle points except when Um contains the m dominant eigenvectors of . In this case, Y(W) attains the global minimum. Note the presence of the arbitrary unitary matrix Q, meaning that the network’s connection pattern W that minimizes the reconstruction error

818

S. Fiori

corresponding to coefficients y = Q H UmH x but in general E[yk y j ] = 0, which means the network-transformed signals are still correlated. One of the most interesting features of this approach is that the optimization problem of searching for a weight matrix W that minimizes the mean square reconstruction error is unconstrained, in that no constraints have to be added to the error cost function in order for the minimization problem to be consistent. This appears as a noticeable difference with respect to the material presented in section 2. In order to reduce the effects of noise, disturbances, and outliers affecting the data sets, generalized versions of the mean square reconstruction error minimization-based PSA algorithms have been developed for analyzing real-valued signals. In what follows, we extend this enhanced theory to the complex domain. Instead of minimizing the quantity E[ e 2 ], here we derive a generalized PSA algorithm by minimizing, through the use of a gradient-based search procedure, a more general cost defined as def

Z(wk ) = E[φ( e 2 )] ,

(3.2)

thought of as a function of any single vector wk for easy notation, where k = 1, 2, . . . , m. The function φ(u) with u ≥ 0 should be differentiable and convex and should possess a unique minimum in u = 0. The problem about its choice is crucial for the behavior of the PSA algorithm and will be discussed. In order to minimize function Z(·) with respect to each wk by means of a gradient steepest-descent recursive algorithm, its gradients are needed. Therefore, we want to evaluate   ∂ Z(wk ) dφ( e 2 ) ∂ e 2 . =E ∂wk d( e 2 ) ∂wk

(3.3)

Note that from definition 3.1, there holds e 2 = x H x − 2x H WW H x + x H WW H WW H x . The expression of the gradient of e 2 is ∂ e 2 = 2(Wy − 2x)yk + 2y H W H wk x . ∂wk By recalling the identity Wy = −e + x, the above expression simplifies into 1 ∂ e 2 = −eyk − xe H wk . 2 ∂wk

(3.4)

Nonlinear Complex-Valued Extensions of Hebbian Learning

819

def

Let us define (u) = dφ(u) . This is what Liano (1996) refers to as the indu fluence function. Then the gradient steepest-descent learning equations for generalized principal subspace extraction for complex-valued signals read ∂ Z(wk ) dwk = E[( e 2 )(eyk + xe H wk )] , k = 1, 2, . . . , m , (3.5) =− dt ∂wk with e being defined by equation 3.1. Choosing the nonlinear function (·) is not generally an easy task. Song et al. (1998) proposed a theory allowing linking the choice of (·) to the statistical features of the reconstruction error for real-valued signals. The aim of the following section is to show how these results may be extended to the complex-valued case. 3.2 The Song-Yilong-Feng Principle and Its Extension to the ComplexValued Case. The Song-Yilong-Feng theory for real-valued ergodic inputsignal x is based on the definition of the random variable: def

ε(x, W) = x − WWT x 2 .

(3.6)

Let us suppose that several random input vectors {xn }n=1,N have been collected. For each input sample, we may compute a reconstruction error value εn = ε(xn , W). The likelihood of W is then defined as the probability def

L(E|W) = q (ε1 , ε2 , . . . , ε N |W) ,

(3.7)

def

where E = [ε1 ε2 · · · ε N ] and q (·, . . . , ·|·) represents the joint probability density function of the εn subjected to the hypothesis W. Under the hypothesis that the xn are identically distributed and statistically independent (i.i.d.), the likelihood writes: L(E|W) =

N  n=1

q ε (εn |W) =

N 

q (εn ) ,

(3.8)

n=1

in short notation.4 Having defined the likelihood of a network connection matrix under the observed reconstruction error values, we may invoke the maximum likelihood estimation theory as a learning tool for the neural network. In order to make this learning principle be profitable, we should hypothesize a distribution for the reconstruction error values.

4 It might be worth recalling that successive x , but not their components, are supposed n to be i.i.d.

820

S. Fiori

In the following, we formulate the maximum-likelihood theory in a version suitable for treating an infinite number of available samples and discuss some possible choices of the error distributions. We also relate the Song-Yilong-Feng theory to the conceptualization of robust classification proposed by Xu and Yuille (1995). Let us suppose that the probability density function of the square reconstruction error ε has the negative exponential (one-side Laplacean) form def

q E (u) = κe −κu (u) ,

(3.9)

where κ > 0 denotes the Laplacean dispersion parameter and (u) again denotes the unit-step function. Then the log likelihood takes on the expression log L(E|W) = N log κ − κ

N 

xn − WWT xn 2 .

n=1

The meaning of this expression is that the maximum likelihood principle and the minimum average square error principle coincide if the conditioned probability density function q ε (ε|W) is assumed as in equation 3.9. Note that the above statement is based on the hypothesis that N is finite; otherwise, the log likelihood could be unbounded. On the basis of this observation, Song et al. (1998) proposed extending the idea to a generic probability density function, replacing the averagesquare-error principle with the maximum likelihood estimation of the weight matrix W. It consists in looking for a weight matrix maximizing the quantity log L(E|W) =

N 

log q (εn ) .

(3.10)

n=1

We aim at extending this optimization criterion in two ways. First, the maximum likelihood theory may be made rigorous for an infinite-size data set (i.e., for density-function-based descriptions instead of sample based). The key step for this extension relies on the use of differential entropy associated with a probability distribution. Second, we want to extend the Song-Yilong-Feng principle to the complex domain. As it shall shortly be clear, this extension leads us to face again the problem of nonlinear function selection. This problem may be solved directly through the Song-YilongFeng principle (an especially interesting formulation comes by choosing the Cauchy-Lorenz distribution for the reconstruction error model) and also from a classification theory by Xu and Yuille (1995) based on statistical physics.

Nonlinear Complex-Valued Extensions of Hebbian Learning

821

In the presence of infinitely many observations, instead of using the likelihood, the differential conditional entropy may be adopted. To this aim, let us define the function def

H(W) = −E x [log q ε (ε|W)|W] = −E[log q (ε)]

(3.11)

in short. The quantity H(W) plays the role of a “minus-mean-loglikelihood”. If we assume again the distribution 3.9 for the square error ε, we find H(W) = − log κ + κ E[ e 2 ] . If the reconstruction error is defined for complex-valued signals as in equation 2.64, minimizing the entropy H with respect to W also means minimizing the mean square error—in symbols: min {H(W)} ⇐⇒

W∈C p×m

min {E[ e 2 ]} .

W∈C p×m

Following the idea in Song et al. (1998), the minimum entropy principle may be extended by assuming different probability density functions q (·). In order to find the matrix W minimizing function 3.11, the gradient steepestdescent approach may be considered. Its use requires evaluating the gradient:    ∂ H(wk ) q ( e 2 ) ∂ e 2 . = −E ∂wk q ( e 2 ) ∂wk def



(u) Defining the statistical score function (u) = − qq (u) , the gradient steepestdescent algorithm pertaining to the entropy 3.11 minimization reads

dwk = E[( e 2 )(eyk + xe H wk )] . dt

(3.12)

Note that equations 2.68 and 3.12 are identical except for the form of functions (·) and (·). The choice of (·) is driven by the hypotheses about the statistics of the scalar quadratic error e 2 . In order to illustrate the usefulness of the differential-entropy-based approach, let us consider in detail the probability density functions q E defined by equation 3.9 and the modified (one-sided) Cauchy-Lorentz distribution recommended in Liano (1996) and Song et al. (1998): def

q C (u) =

2 θ (u) , θ > 0 . 2 π θ + u2

822

S. Fiori

The most interesting difference between the Laplacean and Cauchy probability density functions is that the first distribution is characterized by a mean value and a variance, while the second distribution has neither a mean value nor a variance.5 This special feature of Cauchy distribution deserves some comment:

r

r

r

If we assume a Cauchy distribution for the quadratic error e 2 instead of the Laplacean one, we do not need to embody any a priori hypotheses about the variability of the error in the algorithm. In other words, the risk of biasing the error distribution with inaccurate information is less serious than with the Laplacean distribution. Another interesting implication of the inexistence of moments of the Cauchy distribution is that the effects of Cauchy-based modeling are less tied to the number of available samples. In fact, the practical meaning of the inexistence of moments is that collecting several data points gives no more accurate an estimate of the moments than a single data point does. However, the Cauchy density does impose scale limitation; the choice of the width parameter θ does affect the width of the receptive fields of the neurons in the component and subspace analysis network.

The score function corresponding to the Cauchy distribution is C (u) =

θ2

2u (u) . + u2

Several expressions for the function φ(·) in equation 3.2 have been proposed in the literature (Karhunen & Joutsensalo, 1994, 1995; Xu & Yuille, 1995), usually justified by invoking the theory of robust statistics. These different choices may be discussed using the Song-Yilong-Feng interpretation. In order to accomplish this, it is useful to equate functions (u) and (u). It is worth recalling that (u) =

dφ(u) 1 dq (u) , (u) = − , du q (u) du

where function φ(u) was introduced in formula 3.2 and the probability density function q (u) takes part in formula 3.8. From these relationships, the following formula is readily found: φ(u) = − log q (u) + c ,

(3.13)

5 In fact, the Cauchy distribution does not admit any statistical moment because its characteristic function is not differentiable at the origin.

Nonlinear Complex-Valued Extensions of Hebbian Learning

823

where c ∈ R is a constant. The above formula may be used in two directions. First, if the probability distribution model for the reconstruction noise q (·) has been selected, then the corresponding warping function φ(·) may be computed by formula 3.13. Care should be taken that the resulting function φ(·) fulfills regularity as well as convexity requirements, by definition; otherwise, the pair (q , φ) must be rejected as it is not valid in the present framework. If the pair (q , φ) is acceptable, the constant c, whose value does not influence the shape of the warping function φ(·) but only its vertical placement, may be chosen arbitrarily (e.g., in order to simplify the expression of function φ(·)). Alternatively, formula 3.13 may help in discussing some choices of φ(·) in terms of probability distributions for the square reconstruction error. In this case, care should be taken that the resulting function q (·) be a probability density function; that is, it is a positive integrable function with unitary area. In this case, the constant c may be determined  +∞ by the formula c = − log 0 e −φ(τ ) dτ . As two examples of the first case, we may find the functions φ(·) corresponding to the Laplacean and Cauchy-Lorentz distributions. Straightforward computations show that they write φE (u) = − log κ + c + κu ,   2θ + c + log(θ 2 + u2 ) , φC (u) = − log π respectively. In these cases, the constant c may be chosen   to be zero or, for example, for the Cauchy distribution case, as c = log 2θ , which simplifies π the expression of φC (u). The obtained functions may be easily verified to fulfill the required restrictions such as regularity and convexity, at least in a properly chosen right-sided neighborhood of the origin. As an example of def the second case, let us consider the warping function φK (u) = log cosh(βu) found in Karhunen and Joutsensalo (1994, 1995). The corresponding q (·) function is computed to be q K (u) =

2 2β (u) , π e −βu + e βu

which may easily be proven to be a density distribution and shall hereafter be referred to as K density model. The shape of the above distributions q E , q C , and q K , along with the corresponding warping functions φE , φC , and φK , are depicted in the Figure 7 for arbitrary parameters values. In both the negative exponential distribution model and the K model, as the square reconstruction error ε increases, the density functions value decreases rapidly. This means that in those distributions, a large reconstruction error (usually associated to disturbances) has a small probability of occurring; then the model does not take into account properly the presence of outliers. Conversely,

824

S. Fiori

0.7

10 Laplacean Cauchy-Lorentz K-model

Laplacean Cauchy-Lorentz K-model

9

0.6 8 0.5

7

6 0.4 5 0.3 4

3

0.2

2 0.1 1

0

0

2

4

6

8

0

10

0

2

4

6

8

10

Figure 7: (Left) Reconstruction error distribution functions q E (κ = 0.5; Song et al., 1998), q C (θ = 5/π ; (Song et al., 1998) and q K (β = 1; Karhunen & Joutsensalo, 1995). (Right) Warping functions φE , φC , and φK .

the use of the modified Cauchy distribution improves the robustness of the PSA algorithm since it decreases more slowly than the former two. To end, it is interesting to discuss the relationships among the SongYilong-Feng theory and the conceptualization proposed by Xu and Yuille (1995) based on statistical physics. Let us consider the reconstruction error ε(x, w) due to a one-unit artificial neural network: def

ε(x, w) = x − (w H x)w 2 , x ∈ {xn }n=1,N . Xu and Yuille (1995) considered a neuron’s energy function to be minimized for statistical learning, def

K (f, w) =

N  n=1

f n ε(xn , w) + η

N  (1 − f n ) ,

(3.14)

n=1

where η is a constant and the f k ’s are random binary variables acting as decision flags indicating whether xk is an outlier ( f k = 0) or not ( f k = 1) def and f = [ f 1 f 2 · · · f N ]T . The aim is to minimize K (f, w) with respect to f

Nonlinear Complex-Valued Extensions of Hebbian Learning

825

and w simultaneously. To solve this difficult problem, Xu and Yuille defined a Gibbs distribution, def

ρ(f, w) =

1 −β K (f,w) , e 

where  is a normalization constant and 1/β stands for a “temperature” (for a discussion on the meaning of parameters η and β, see Xu & Yuille, 1995). Then they proposed to approximate ρ(f, w) by computing the marginal probability distribution ρm (w) obtained averaging ρ(f, w) over all possible configurations f (which may be thought of as a kind of mean-field approximation). This gives (Xu & Yuille, 1995) ρm (w) = 1m e −β K eff (w) , with def m = e Nβη and def

−K eff (w) =

N 1 log{1 + e −β[ε(xn ,w)−η] } . β n=1

(3.15)

The optimal connection configuration wopt is found by maximizing−K eff (w). It is now interesting to note that −K eff (w) is very similar to log L(E|w) in equation 3.10; thus, we may define a probability density function, 1

def

q X (u) = A[1 + e −β(u−η) ] β (1 − (u − )) ,

(3.16)

where Ais a normalization constant and the factor 1 − (u − ) makes q X (u) vanish outside a compact Then the normalization constant  1 domain [0, ]. 1 def def can be found by A1 = β1 D u−1 (1 + Cu) β du, where D = exp(−β) and C = exp(βη). Note that  may be arbitrarily large, albeit bounded. A distribution q X is shown in Figure 8. The distribution q X (u) represents an equivalent reconstruction error distribution that would make it possible to select a proper nonlinear warping function in reconstruction-error-based learning via the Xu-Yuille statistical physics–based theory. 3.3 Component-Wise Nonquadratic Reconstruction Error Minimization. The learning theory discussed in section 3.1 allows extracting a principal subspace only from a set of complex-valued random signals. A neural network trained by means of that algorithm is not able to effect PCA substantially because of its inherent symmetry. With the aim of breaking the symmetry and making the network hierarchical, it is possible to define a reconstruction error vector for each neuron (Karhunen & Joutsensalo, 1995) and to design a learning rule for each neuron so that it tries to minimize a local error. Again, only linear reconstruction is dealt with. A possible definition is ek = x −

K (k)  j=1

yj w j ,

(3.17)

826

S. Fiori 0.9

Recon. error probability density distribution

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0

0.5

1

1.5 Reconstruction error

2

2.5

3

Figure 8: Reconstruction error distribution functions q X (β = 0.5, η = 4/π as in Xu & Yuille, 1995, and  = 3).

where k ranges from 1 to m, the number of the neurons. The indexing function K (k) determines whether the network is symmetrical. If K (k) = m, we come back to the  symmetrical case; all the neurons learn to correct the total error e = x − mj=1 y j w j , while K (k) = k makes the network hierarchical.6 Following Karhunen and Joutsensalo (1995), here we discuss the optimization of the cost function def

R(ek ) =

p 

E[ f (e kr )] ,

(3.18)

r =1

where, contrary to the case discussed in section 3.1, the r th entry of the k th error vector ek , namely, e kr , warps by the function f (·). Again, f (ζ ) = g(|ζ |) for ζ ∈ C, with g(u) being differentiable, convex, with a unique minimum in u = 0 in a convenient right-sided neighborhood of the origin. Evaluating the gradient of R(ek ) with respect to the weight vector wk involves rather 6 Note that in the hierarchical case, the reconstruction error may be defined recursively as e1 = x − y1 w1 , while for k ≥ 2: ek = ek−1 − yk wk .

Nonlinear Complex-Valued Extensions of Hebbian Learning

827

difficult operations that may be considered interesting from a methodological point of view; hence, we provide some details about its computation. Particularly, we need the Hadamard product operator, defined as 

     a1 c1 a 1 c1 a 2   c2        def  a 2 c 2  a◦c= .  ◦  .  =  .  ,  ..   ..   ..  ap cp a pc p with a, c being p-dimensional real- or complex-valued vectors. Note that the Hadamard product is commutative and associative. Furthermore, we need a set of vectors br : each br ∈ R p is defined so that its entries are 0 except for the r th one equal to 1 (i.e., the {br } is the canonical basis of R p ). R(ek ) , let us consider the quantity |e kr |2 , To begin with the calculation of ∂ dw k which has the structure |e kr |2 = xr xr − 2P(e kr ) + Q(e kr ) ,

(3.19)

where the following quantities have been defined: def

P(e kr ) =

K (k) 

Re{y j xr w jr } ,

j=1 def

Q(e kr ) =

K (k) K (k)  

Re{yj ys w jr wsr } .

j=1 s=1

The gradient of P(e kr ) computed with respect to the complex-valued weight vector wk is found to be ∂ P(e kr ) = yk x ◦ br + (xr wkr )x . ∂wk

(3.20)

About the gradient of Q(e kr ), after some mathematical work, we find  1 ∂ Q(e kr )     = wkr x . yk yh w j ◦ br + yj whr 2 ∂wk j=1 K (k)

(3.21)

Now the effects of the nonlinearity f (·) have to be taken into account. For notational compactness, let us define functions: def

b(u) =

dg(u) def b(|ζ |) , u ≥ 0 and B(ζ ) = , ζ ∈C. d(u2 ) |ζ |

(3.22)

828

S. Fiori

Let us evaluate the gradient of R(ek ) with respect to the weight vector wk :   p 1 ∂ R(ek ) b(|e kr |) ∂|e kr |2 . = E ∂wk 2 r =1 |e kr | ∂wk

(3.23)

From equations 3.19 to 3.23, rearranging terms, recalling the use of the Hadamard operator ◦, and allowing function B(·) to operate componentwise, we have:  K (k)  1 ∂ R(ek )  H =E (y j w j )[B(ek ) ◦ wk ]x 2 ∂wk j=1 

+E

yk B(ek )

◦

K (k) 



yj w j

j=1

  − E yk x ◦ B(ek ) + x H [B(ek ) ◦ wk ]x .  K (k) By replacing the term j=1 y j w j with x − ek , we obtain the gradient steepest-descent learning algorithm:   dwk 1 ∂ R(ek ) = E ekH [B(ek ) ◦ wk ]x + yk [B(ek ) ◦ ek ] . =− dt 2 ∂wk

(3.24)

def

Let us consider the nonquadratic warping function g(u) = 2u3 for u ∈ This choice yields B(z) = 1—hence, the learning algorithm:

R+ 0.

dwk = E[(ekH wk )x + yk ek ] , k = 1, 2, . . . , m , dt

(3.25)

which coincides with the learning rule 3.5 when (·) = 1 and K (k) = m. Also, in the presence of real-valued input signals, the quantities involved in equation 3.24 are real valued; thus, it may be rewritten as: −

  1 ∂ R(ek ) = E ekT [B(ek ) ◦ wk ]x + yk [B(ek ) ◦ ek ] . 2 ∂wk

(3.26)

Now, since B(u) = b(u)/u for u ∈ R, if we allow b(·) to operate componentwise, the following identities hold: B(ek ) ◦ ek = b(ek ) and ekT [B(ek ) ◦ wk ] = wkT b(ek ). Therefore, the learning rule corresponding to the gradient 3.26 for a realvalued PCA neural network reads dwk = E[wkT b(ek )x + wkT xb(ek )] , k = 1, 2, . . . , m . dt

(3.27)

Nonlinear Complex-Valued Extensions of Hebbian Learning

829

This learning rule coincides with the nonclassic PCA extraction rule discussed, for instance, in Karhunen and Joutsensalo (1995). With reference to the choice of the nonlinear function g(·), it seems reasonable to extend the entropy version of the Song-Yilong-Feng theory in order to model the probability density function of any single reconstruction error  p ek . Formally, we can then particularize the cost function 3.18 as − j=1 E[log q (|e k j |2 )], which ultimately means identifying g(u) = − log q (u2 ). By choosing as the conditional probability density function the one-sided Cauchy distribution, plugging this g(u) into the definition of B(·) leads to BC (z) =

2|z| . θ 2 + |z|4

(3.28)

This shows that the approach in Song et al. (1998) may be successfully extended to the component-wise reconstruction error minimization method. The extended Hebbian learning theory described in this section relates to a series of contributions. One of them is Xu’s interpretation of extended Hebbian learning known as maximum uncertainty theory (a detailed analysis recently has been presented in Fiori, 2001). Other interesting contributions to this topic are the learning theories by Corchado and Fyfe (cited in Fiori, 2003b), Miao and Hua (1998), Plumbley (1992), Shirazi, Peper, and Sawai (1999), and Higuchi and Eguchi (2004). 3.4 Illustrative Numerical Experiments. In order to illustrate the concept of principal component and subspace analysis in the complex domain by nonquadratic reconstruction error minimization via examples, the results of some numerical simulations are reported below. The experiments aimed at illustrating the convergence of the learning algorithms 3.5 and 3.24, and comparing the performances of the robust and nonrobust versions in the presence of complex-valued data corrupted by gross outliers. Robustness should emerge by the proper choice of the nonlinear warping functions (·) and (·). The experiments performed retrace the ones suggested in Song et al. (1998) and Xu and Yuille (1995). A network with two inputs x1 and x2 and one output is trained with a set of uncorrupted samples and with the same data set corrupted by 3% of outliers. The two data sets are shown in Figure 9. Note that for a single-unit neural network, the distinction between PCA and PSA disappears. The network weight vector is denoted here by w. Furthermore, the quantity q1 denotes the first normalized eigenvector of the covariance matrix of the uncorrupted data (i.e., the true eigenvector). As the component and subspace analysis performance index, we consider the principal angle γ , here defined in the following way: def

γ = cos−1

|w H q1 | . wH w

830

S. Fiori 2

4 2

2

Im{x }

Im{x1}

1

0

−1 −2 −10

0

−2 −4

−5

0 Re{x1}

5

−6 −2

10

10

−1

0 Re{x2}

1

2

10

8 5 2

Im{x }

Im{x1}

6 4 2

0

0 −2 −10

−5

0 Re{x1}

5

10

−5 −5

0

5

10 Re{x2}

15

20

Figure 9: Experiments on reconstruction error minimization-based learning: Uncorrupted data set (top) and data set with outliers (bottom).

The outliers are such that they produce a deviation between the true first eigenvector and the noisy first eigenvector of about 30 degrees. First, algorithms 3.5 and 3.24 have been trained on the uncorrupted data, and both have been able to detect the principal eigenvector corresponding to a principal angle value close to zero. Second, algorithm 3.5 with (·) = 1 was trained on the noisy data; the results are shown in Figure 10. The same algorithm, trained on the same data set, is made robust by introducing the Cauchy-Lorentz nonlinearity (·) = C (·), and has given instead the result shown in Figure 11, which pertains to the value of the Cauchy parameter θ = 5. Figures 10 and 11 show the behavior of the principal angle γ as well as the values of the norm of the reconstruction error e 2 during learning. The net result of the introduction of the Cauchy nonlinearity is apparent, as it allows almost completely neglecting the effects of outliers on learning. The results in both robust and nonrobust cases pertaining to algorithm 3.24 are completely equivalent and are therefore not shown.

Nonlinear Complex-Valued Extensions of Hebbian Learning

831

Principal angle γ (Deg)

100 80 60 40 20 0

0

500

1000

1500

2000

2500 Samples

3000

3500

4000

4500

5000

0

500

1000

1500

2000

2500 Samples

3000

3500

4000

4500

5000

Reconstruction error ||e||

14 12 10 8 6 4 2 0

Figure 10: Numerical result obtained with algorithm 3.5, in the nonrobust version ((·) = (·) = 1).

4 Conclusion We initially conceived of this review around 1997 and have been researching it since then. It has led to a great deal of research, and new material has been added through the years as soon as it became available in the scientific literature. As the principal component analysis literature is very large and fragmented, this contribution aimed at bringing together a number of mathematical results for principal component analysis and its generalizations such as principal subspace and minor component analysis in the complex domain. The emphasis of the article was on extensions to complexvalued neural networks and on relating these to a number of previous results known from scientific literature. In particular, extensions of previous research work and relationships were sought on Hebbian learning for linear, complex-weighted, feedforward, and laterally connected neural networks. Learning principles have been presented and analyses carried out on complex-valued principal, minor component, subspace quadratic, and nonquadratic rules.

832

S. Fiori

Principal angle γ (Deg)

100 80 60 40 20 0

0

500

1000

1500

2000

2500 Samples

3000

3500

4000

4500

5000

0

500

1000

1500

2000

2500 Samples

3000

3500

4000

4500

5000

Reconstruction error ||e||

20

15

10

5

0

Figure 11: Numerical result obtained with algorithm 3.5, in the robust version ((·) = (·) = C (·)).

The starting point of the analysis was an optimization approach for complex-valued networks that aimed at producing a purposeful contribution. Many existing results could then be subsumed into this framework (although some may not). Neural networks architectures for principal subspace analysis were considered in section 2. After reviewing previous work that considered the complex-valued linear and real-valued nonlinear case, the equations for the complex-valued nonlinear case were derived using gradient optimization with Lagrange multipliers. Then neural principal component analyzers for the real-valued linear and complex-valued linear cases were reviewed and extended to the complex-valued nonlinear case, again using gradient-based optimization. The problem of the choice of the nonlinearity was mentioned, and an attempt was made to generalize observations of Sudjianto and Hassoun to the complex case. The analysis proceeded by extending some results from real-valued (linear) minor component analysis algorithms to the complex linear case. Minimization of reconstruction error was considered in section 3. Existing principal subspace and principal component approaches for the

Nonlinear Complex-Valued Extensions of Hebbian Learning

833

real-valued nonlinear and complex-valued linear cases were reviewed and gradient equations for the complex-valued nonlinear case were derived. The problem of choosing the nonlinearity was discussed based on a maximum likelihood approach by Song et al. (1998) for the real valued non-linear case. As a crucial part of the presented learning rules is the selection of a cost function on which the derivation of the involved nonlinearity is based, we made some choices concerning this selection. We made the choice to consider only cost functions that take into account the magnitude of their complex argument. While this choice could seem natural, it was discussed and motivated with respect to its implicit assumptions through citations of relevant literature and explanation of its usefulness in applied fields related to signal processing. The treatment of the shape of the nonlinearity seemed to provide insight that would go beyond the results known for nonlinearities in the real-valued case. The discussion of the choice of the nonlinearity and related developments in independent component analysis is an example of this finding. We believe this review will be useful to other researchers because it presents an overview of existing approaches to complex-valued linear and real-valued nonlinear versions of Hebbian learning algorithms and provides derivations of equations that may be useful for the complex-valued nonlinear case. It may be of interest to readers who are interested in the extension of Hebbian learning to the complex and nonlinear case. Acknowledgments I gratefully thank the anonymous reviewers, the action editor, Mark Plumbley, and the editor, Terrence Sejnowski, for the careful and accurate suggestions that helped to improve the clarity and the overall organization of this article, and Leslie-Anne Chaden for her constant constructive support in handling the manuscript. The final version of this work was completed when I was a short-term visitor of the Mathematical Neuroscience Laboratory of the Brain Science Institute at the Institute of Physical and Chemical Research RIKEN (Japan), from July to September 2004. I express my gratitude to the BSI director, Shunichi Amari, and to the laboratory members for the kindest and warmest hospitality. References Abbas, H. M., & Fahmy, M. M. (1994). Neural model for Karhunen-Lo´eve transform with application to adaptive image compression. IEE Proceedings I, Communications, Speech and Vision, 140(2), 135–143. Amari, S. (1977). Neural theory of association and concept formation. Biological Cybernetics, 26, 175–185.

834

S. Fiori

Amari, S. & Cichocki, A. (1998). Adaptive blind signal processing: Neural network approaches. Proceedings of the IEEE, 86(10), 2026–2048. Atick, J. J., & Redlich, A. N. (1993). Convergent algorithm for sensory receptive field development. Neural Computation, 5(1), 45–60. Baldi, P. F., & Hornik, K. (1995). Learning in linear neural networks: A survey. IEEE Transactions on Neural Networks, 6(4), 837–858. Bannour, S., & Azimi-Sadjadi, M. R. (1995). Principal component extraction using recursive least squares learning. IEEE Transactions on Neural Networks, 6(2), 457– 469. Barlow, H. B. (1989). Unsupervised learning. Neural Computation, 1, 295–311. Barlow, H. B. (1998). Guest editorial: Cerebral predictions. Perception, 27(8), 885–888. Barlow, H. B. (2001). Redundancy reduction revisited. Network: Computation in Neural Systems, 12(3), 241–253. Barlow, H. B., & Foldi´ ¨ ak, P. (1989). Adaptation and decorrelation in the cortex. In C. Miall, R. M. Durbin, & G. J. Mitchison (Eds.), The computing neuron, (pp. 54–72). Reading, MA: Addison-Wesley. Bechtel, W., & Abrahamsen, A. (1993). Connectionism and the mind. Oxford: Blackwell. Bell, A. J., & Sejnowski, T. J. (1995). An information maximization approach to blind separation and blind deconvolution. Neural Computation, 7(6), 1129–1159. Benvenuto, N., & Piazza, F. (1992). On the complex back-propagation algorithm. IEEE Transactions on Signal Processing, 40(4), 967–969. Bertsekas, D. P. (1996). Constrained optimization and Lagrange multiplier methods. Bedford, MA: Athena Scientific. Bingham, E., & Hyv¨arinen, A. (2000). A fast fixed-point algorithm for independent component analysis of complex valued signals. International Journal of Neural Systems, 10(1), 1–8. Celledoni, E., & Fiori, S. (2004). Neural learning by geometric integration of reduced “rigid-body” equations. Journal of Computational and Applied Mathematics, 172 (2), 247–269. Chen, T.-P., & Amari, S.. (2001). Unified stabilization approach to principal and minor components extraction algorithms. Neural Networks, 14(10), 1377–1387. Chen, T.-P., Amari, S., & Lin, Q. (1998). A unified algorithm for principal and minor components extraction. Neural Networks, 11(3), 385–390. Chen, Y., & Hou, C. (1992). High resolution adaptive bearing estimation using a complex-weighted neural network. Proc. of International Conference on Acoustics, Speech and Signal Processing, 2, 317–320. Chow, T. W., & Fang, Y. (2001). Neural blind deconvolution of MIMO noisy channels. IEEE Transactions on Circuits and Systems—Part I, 48(1), 116–120. Cichocki, A., & Amari, S. (2002). Adaptive blind signal and image processing. New York: Wiley. Costa, S., & Fiori, S. (2001). Image compression using principal component neural networks. Image and Vision Computing Journal [Special issue], 19(9–10), 649–668. De Castro, M. C. F., De Castro, F. C. C., Amaral, J. N., & Franco, P. R. G. (1998). A complex valued Hebbian learning algorithm. Proc. of International Joint Conference on Neural Networks (pp. 1235–1238) Piscataway, NJ: IEEE. Desodt, G., & Muller, D. (1990). Complex ICA applied to the separation of radar signals. Proc. of Signal Processing V: Theories and Applications (EUSIPCO), 1 , 665–668.

Nonlinear Complex-Valued Extensions of Hebbian Learning

835

Diamantaras, K. I., & Kung, S. Y. (1996). Principal component neural networks: Theory and applications. New York: Wiley. Douglas, S. C., Amari, S. & Kung, S. Y. (2000). On gradient adaptation with unit-norm constraints. IEEE Transactions on Signal Processing, 48(6), 1843–1847. Fiori S. (2000). Blind separation of circularly distributed source signals by the neural extended APEX algorithm. Neurocomputing, 34(1–4), 239–252. Fiori S. (2001). A theory for learning by weight flow on Stiefel-Grassman manifold. Neural Computation, 13(7), 1625–1647. Fiori S. (2002). A theory for learning based on rigid bodies dynamics. IEEE Transactions on Neural Networks, 13(3), 521–531. Fiori S. (2003a). Extended Hebbian learning for blind separation of complex-valued sources. IEEE Transactions on Circuits and Systems—Part II, 50(4), 195–202. Fiori S. (2003b). A neural minor component analysis approach to robust constrained beamforming. IEE Proceedings—Vision, Image and Signal Processing, 150(4), 205– 218. Fiori S. (2003c). Neural ICA by “maximum-mismatch” learning principle. Neural Networks, 16(8), 1201–1221. Fiori S. (2003d). Fully-multiplicative orthogonal-group ICA neural algorithm. Electronics Letters, 39(24), 1737–1738. Fiori S. (2004). A fast fixed-point neural blind deconvolution algorithm. IEEE Transactions on Neural Networks, 15(2), 455–459. Fiori S. (in press). Fixed-point neural independent component analysis algorithms on the orthogonal group. Journal of Future Generation Computer Systems. Fiori, S., Faba, A., Albini, L., Cardelli, E., & Burrascano, P. (2003). Numerical modeling for the localization and the assessment of electromagnetic field sources. IEEE Transactions on Magnetics, 39(3), 1638–1641. Fiori, S., & Piazza, F. (2000). A general class of ψ–APEX PCA neural algorithms. IEEE Transactions on Circuits and Systems—Part I, 47(9), 1394–1398. Foldi´ ¨ ak P. (1990). Forming sparse representations by local anti-Hebbian learning. Biological Cybernetics, 64, 165–170. Fyfe, C., & MacDonald, D. (2002). ε-insensitive Hebbian learning. Neurocomputing, 47(1–4), 35–57. Gao, K., Ahmed, M. O., & Swamy, M. N. S. (1994). A constrained anti-Hebbian learning algorithm for total least-squares estimation with applications to adaptive FIR and IIR filtering. IEEE Transactions on Circuits and Systems–Part II, 41(11), 718– 729. Georgiou, G. M., & Koutsougeras, C. (1992). Complex-domain backpropagation. IEEE Transactions on Circuits and Systems—Part II, 39(5), 330–334. Hairer, E., Lubich, C., & Wanner (2002). Geometric numerical integration: Structure preserving algorithms for ordinary differential equations, Berlin: Springer-Verlag. Hanna, A. I., & Mandic, D. P. (2003). A complex-valued nonlinear neural adaptive filter with a gradient adaptive amplitude of the activation function. Neural Networks, 16(2), 155–159. Harpur, G. F. (1997). Low entropy coding with unsupervised neural networks, Unpublished doctoral disseratation, University of Cambridge. Haykin, S. (1989). An introduction to analog and digital communications. New York: Wiley.

836

S. Fiori

Haykin, S. (1998). Neural networks: A comprehensive foundation, (2nd eds). Upper Saddle River, NJ: Prentice Hall. Hebb, D. (1949). The organization of behaviour. New York: Wiley. Helmke, U., & Moore, J. B. (1993). Optimization and dynamical systems. Berlin: Springer-Verlag. Higuchi, I., & Eguchi, S. (2004). Robust principal component analysis with adaptive selection for tuning parameters. Journal of Machine Learning Research, 5, 453–471. Hyv¨arinen, A. (1999). Fast and robust fixed-point algorithms for independent component analysis. IEEE Transactions on Neural Networks, 10(3), 626–634. Hyv¨arinen, A., Karhunen, J., & Oja, E. (2001). Independent component analysis. New York: Wiley. Iserles, A., Munthe-Kaas, H. Z., Nørsett, S. P., & Zanna, A. (2000). Lie-group methods. Acta Numerica, 9, 215–365. Karhunen, J., & Joutsensalo, J. (1994). Representation and separation of signals using nonlinear PCA type learning. Neural Networks, 7(1), 113–127. Karhunen, J., & Joutsensalo, J. (1995). Generalizations of PCA, optimization problems, and neural networks. Neural Networks, 8(4), 549–562. Kates, J. M. (1993). Superdirective arrays for hearing aids. Journal of Acoustics Society of America, 94(4), 1930–1933. Klemm, R. (1987). Adaptive airborne MTI: An auxiliary channel approach. IEE Proceedings F, 134, 269–276. Kung, S.-I., Diamantaras, K. I., & Taur J. S. (1994). Adaptive principal component extraction (APEX) and applications. IEEE Transactions on Signal Processing, 42(5), 1202–1217. Laheld, B., & Cardoso, J. F. (1994). Adaptive source separation with uniform performance. Proc. of Signal Processing VII: Theories and Applications (EUSIPCO), 1, 183–186. Levine, D. S., Brown, V. R., & Shirey, T. V. (1999). Oscillations in neural systems. Mahwah, NJ: Erlbaum. Liano, K. (1996). Robust error measure for supervised neural network learning with outliers. IEEE Transactions on Neural Networks, 7(1), 246–250. Linsker, R. (1992). Local synaptic rules suffice to maximize mutual information in a linear network. Neural Computation, 4, 691–702. Luo, F.-L., & Unbehauen R. (1997). Applied neural networks for signal processing. Cambridge: Cambridge University Press. Mathew, G., & Reddy, V. U. (1994). Development and analysis of a neural network approach to Pisarenko’s harmonic retrieval method. IEEE Transactions on Signal Processing, 42(3), 663–667. Mathis, H., & Douglas, S. C. (2002). On the existence of universal non-linearities for blind source separation. IEEE Transactions on Signal Processing, 50(5), 1007–1016. Miao, Y., & Hua, Y. (1998). Fast subspace tracking and neural network learning by a novel information criterion. IEEE Transactions on Signal Processing, 46(7), 1967– 1979. Michaels, R. B., & Upadhyaya, B. R. (1999). A complex valued neural network with local learning laws. In C. H. Dagli et al. (Eds.), Intelligent engineering systems through artificial neural networks (Vol. 9, pp. 101–109). New York: ASME Press.

Nonlinear Complex-Valued Extensions of Hebbian Learning

837

Miller, K. D., & MacKay, D. J. C. (1994). The role of constraints in Hebbian learning. Neural Computation, 6(1), 100–126. Miyauchi, M., Seki, M., Watanabe, A., & Miyauchi, A. (1993). Interpretation of optical flow through complex neural network. Proc. of the International Conference on Artificial Neural Networks, (pp. 645–650). New York: Springer. Muezzino ¨ glu, ˇ M. K., Guzeli¸ ¨ s, C., & Zurada, J. M. (2003). A new design method for the complex-valued multistate Hopfield associative memory. IEEE Transactions on Neural Networks, 14(4), 891–899. Nitta, T. (1997). An extension of the back-propagation algorithm to complex numbers. Neural Networks, 10(8), 1391–1415. Nitta, T. (2000). An analysis of the fundamental structure of complex-valued neurons. Neural Processing Letters, 12(3), 239–246. Nitta, T. (2004). Orthogonality of decision boundaries in complex-valued neural networks. Neural Computation, 16, 73–97. Oja, E. (1982). A simplified neuron model as a principal component analyzer. Journal of Mathematics and Biology, 15, 267–273. Oja, E. (1989). Neural networks, principal components, and subspaces. International Journal of Neural System, 1, 61–68. Oja, E. (1992). Principal components, minor components, and linear neural networks. Neural Networks, 5(6), 927–935. Oja, E. (1994). Beyond PCA: Statistical expansions by nonlinear neural networks. Proc. of the International Conference on Artificial Neural Networks, 2, 1049–1054. O’Leary, D. P. (1990). Robust regression computation using iteratively reweighted least squares. SIAM Journal of Matrix Analysis Application, 11(3), 466–480. Palmieri, F., & Zhu, J. (1995). Self-association and Hebbian learning in linear neural networks. IEEE Transactions on Neural Networks, 6(5), 1165–1184. Pandey, R. (2001). Blind equalization and signal separation using neural networks. Unpublished doctoral disseratation, Indian Institute of Technology, Roorkee, India. Plumbley, M. D. (1992). Efficient information transfer and anti-Hebbian neural networks. Neural Networks, 6(6), 823–833. Plumbley, M. D. (2003). Algorithms for non-negative independent component analysis. IEEE Transactions on Neural Networks, 14(3), 534–543. Rubner, J., & Tavan, P. (1989). A self-organizing network for principal component analysis. Europhysics Letters, 10(7), 693–698. Sanger, T. D. (1989). Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks, 2(6), 459–473. Schmidt, R. (1986). Multiple emitter location and signal parameter estimation. IEEE Transactions on Antennas and Propagation, 34(3), 276–280. Shirazi, M. N., Peper, F., & Sawai, H. (1999). Principal component analysis by entropylikelihood optimization. Transactions of the Information Processing Society of Japan, 40(10), 3638–3644. Song, W., Yilong, L., & Feng, M. (1998). An adaptive robust PCA neural network. In Proc. of International Joint Conference on Neural Networks (pp. 2288–2293). Piscataway, NJ: IEEE. Spratling, M. W., & Johnson, M. H. (2002). Preintegration lateral inhibition enhances unsupervised learning. Neural Computation, 14(9), 2157–2179.

838

S. Fiori

Sudjianto, A., & Hassoun, M. H. (1995). Statistical basis of non-linear Hebbian learning and application to clustering. Neural Networks, 8(5), 707–715. Thirion Moreau, N., & Moreau, E. (2002). Generalized criterion for blind multivariate signal equalization. IEEE Signal Processing Letters, 9(2), 72–74. Vegas, J. M., & Zufiria, P. J. (2004). Generalized neural networks for spectral analysis: Dynamics and Lyapunov functions. Neural Networks, 17(2), 233–245. Weingessel, A., & Hornik, K. (2000). Local PCA algorithms. IEEE Transactions on Neural Networks, 11(6), 1242–1250. Widrow, B., & Winter, R. (1988). Neural nets for adaptive filtering and adaptive pattern recognition. IEEE Computer Magazine, 21, 25–39. Xu, L. (1992). Least mean square error reconstruction principle for self-organizing neural nets. Neural Networks, 6(5), 627–648. Xu, L., Oja, E., & Suen, C. Y. (1992). Modified Hebbian learning for curve and surface fitting. Neural Networks, 5(3), 441–457. Xu, L., & Yuille, A. L. (1995). Robust PCA by self-organizing rules based on statistical physics approach. IEEE Transactions Neural Networks, 6(1), 131–143. Yan, W.-Y., Helmke, U., & Moore, J. B. (1994). Global analysis of Oja’s flow for neural networks. IEEE Transactions on Neural Networks, 5(5), 674–683. Yang, B. (1995). Projection approximation subspace tracking. IEEE Transactions on Signal Processing, 43(1), 1247–1252. Zufiria, P.J. (2002). On the discrete-time dynamics of the basic Hebbian neural network node. IEEE Transactions on Neural Networks, 13(6), 1342–1352.

Received November 26, 2003; accepted August 10, 2004.

LETTER

Communicated by Kechen Zhang

Difficulty of Singularity in Population Coding Shun-ichi Amari [email protected]

Hiroyuki Nakahara [email protected] Laboratory for Mathematical Neuroscience, RIKEN Brain Science Institute, Wako, Saitama, 351-0198 Japan

Fisher information has been used to analyze the accuracy of neural population coding. This works well when the Fisher information does not degenerate, but when two stimuli are presented to a population of neurons, a singular structure emerges by their mutual interactions. In this case, the Fisher information matrix degenerates, and the regularity condition ensuring the Cram´er-Rao paradigm of statistics is violated. An animal shows pathological behavior in such a situation. We present a novel method of statistical analysis to understand information in population coding in which algebraic singularity plays a major role. The method elucidates the nature of the pathological case by calculating the Fisher information. We then suggest that synchronous firing can resolve singularity and show a method of analyzing the binding problem in terms of the Fisher information. Our method integrates a variety of disciplines in population coding, such as nonregular statistics, Bayesian statistics, singularity in algebraic geometry, and synchronous firing, under the theme of Fisher information.

1 Introduction A fundamental concern in systems neuroscience is how information is represented in the brain. Population coding is a widely accepted scheme of representation in which the accuracy of encoding is evaluated by Fisher information (Seung & Sompolinsky, 1993; Zemel, Dayan, & Pouget, 1998; Yoon & Sompolinsky, 1999; Zhang & Sejnowski, 1999; Pouget, Dayan, & Zemel, 2000; Wu, Amari & Nakahara, 2002a, 2002b; Wu, Nakahara, & Amari, 2001; Nakahara & Amari, 2002; Nakahara, Wu, & Amari, 2001; Wu, Chen, Niranjan, & Amari, 2003). The Fisher information is not only related to the mean square error of an estimator by the Cram´er-Rao theorem, but is an intrinstic quantity to measure how much information is included in the population coding. Neural Computation 17, 839–858 (2005)

© 2005 Massachusetts Institute of Technology

840

S. Amari and H. Nakahara

When two similar stimuli are presented at the same time, their responses interact mutually and cause strange phenomena in both the mathematical structure and the behavior of an animal. We explain this phenomenon by a simple example (Ingle, 1968; Amari & Arbib, 1977). When a frog observes a fly in its visual field, it is represented by an excitation pattern of neurons in the visual field (tectum). The pattern has a peak at the place corresponding to the location of the fly (see Figure 1A). Neural firing fluctuates randomly, so that the amount of information included in such population coding is analyzed by Fisher information. However, when two flies appear at the same time, the response pattern of neural excitation has two peaks at the corresponding locations of the flies (see Figure 1B). When the two flies are very close, the two peaks merge into one, and there remains only a noisy pattern of one peak (see Figure 1C). What will happen under such a condition? Neuroethological study has observed pathological behavior in that the frog cannot distinguish the two flies and sometimes jumps in between the flies, failing to catch either (Ingle, 1968). Mathematically, the Fisher information matrix degenerates in this case, so that the established Cram´er-Rao paradigm of statistics does not hold. Hence, a new framework of statistics is required to analyze such a nonregular or singular situation. Looking at the situation in more detail, we find that algebraic singularity exists in the underlying statistical manifold, which is the main cause of difficulty. There are many practical methods to resolve the problem. The frog has its own neural algorithm. Many theoreticians use the Bayesian strategy, and biologically plausible algorithms have been proposed (Pouget, Zhang, Denever, & Latham, 1998; Zemel, et al., 1998; Zemel & Pillow, 2002; Sahani & Dayan, 2003). However, the Bayesian method is not free of the difficulty caused by the singularity, because the performances of Bayesian algorithms are largely affected by the prior distributions in the singular case, which is completely different from the regular case. Computer simulations have been used so far to analyze the estimation errors of proposed algorithms. We need a new theretical framework to carry out the mathematical analysis, in particular the limit of accuracy attainable by the optimal method. The singularity also affects the Bayesian method strongly in a different way from the regular case. This study focuses on such a singular problem of population coding, providing a new framework of analysis. We do not propose a practical procedure to decide the number of stimuli or to decode information giving the estimated locations of stimuli. Instead, our framework is necessary to analyze the behaviors of algorithms mathematically. For this purpose, we must integrate various disciplines related to the scheme of population coding: neural fields (Amari, 1977), degenerate Fisher information in a nonregular model (Amari & Ozeki, 2001; Fukumizu, 2003), algebraic geometrical singularity (Fukumizu & Amari, 2000; Watanabe, 2001a) that causes such degeneration, and the information-geometrical method (Amari & Nagaoka, 2000; Amari, 1998) to elucidate the whole

Difficulty of Singularity in Population Coding

A

B

r(z)

x

z

r(z)

x1

C

841

x2

z

r(z)

x1 x2

z

Figure 1: Schematic presentation of firing patterns r (z) in a neural population, where r (z) denotes the firing rate of neurons at location z: given a single stimulus at x (A), given two stimuli at x1 and x2 that are far enough apart (B), and given two stimuli close to each other (C).

structure. (See also Dacunha-Castelle & Gassiat, 1997; Lin & Shao, 2003 and Burnashev & Amari, 2002, for recent statistical analysis of the log-likelihood ratio and the maximum likelihood estimator in singular statistical models.) We begin with the analysis of the Fisher information of population coding in a neural field. The Fourier domain is useful for analysis when the field is homogeneous. Various properties in population coding, such as the effects of the shape of a tuning curve and the correlation of noises, will be clearly understood in this framework (Wu et al., 2002b). We

842

S. Amari and H. Nakahara

summarize the method in section 2, when a single stimulus is given in a one-dimensional neural field. When two stimuli appear close together at the same time in a one-dimensional field, their mutual interaction becomes very strong. The Fisher information is a matrix in this case, but it degenerates when the two are close to each other. This violates the classic framework of statistics, and one cannot apply the Cram´er-Rao theorem because the estimation error grows without limit. It is indeed difficult to define the optimal strategy to judge if there are two targets or one. It is more difficult to identify the intensity and the location of each target and to analyze rigorously the performances of various algorithms. The difficulty is observed in animal behavior (Ingle, 1968; Treue, Hol, & Rauber, 2000; Reichardt & Poggio, 1975). When the Fisher information degenerates, its inverse diverges to infinity. Therefore, we need a careful mathematical analysis. We demonstrate that the space of parameters to be estimated from the neural excitation pattern has algebraic singularity and analyze its structure in detail. Such singularity causes the degeneration of the Riemannian structure (or the Fisher information) of the space as a black hole does in the real space-time universe. We analyze how the accuracy of estimation degenerates as the two targets draw near to each other. To this end, the Fisher information is decomposed into regular directions and singular directions, and the accuracy of estimation in the respective directions is analyzed. Such analysis gives new insight into modern statistics. The Bayes inference is another method to give a practical solution even in a singular case. However, we show that there exists a serious problem, because a smooth, positive prior over the parameters induces a singular prior on the singular model. A smooth, positive prior on the space of response functions is not absolutely continuous to a smooth prior on the parameter space. They are mutually singular. We study such a problem, showing that the same difficulty exists in Bayesian framework. From the viewpoint of model selection, the number of stimuli is decided from the data by some criteria such as the Akaike information criterion, the Bayesian information criterion, or the minimum descriptive length criterion. However, for all of these model selection procedures, their statistical validity loses its ground in such a singular case, as we explain below. We then ask if there is a way to get rid of this difficulty, finding that this is closely related to the binding problem. We demonstrate how synchronous firing resolves this difficulty by using the Fisher information. The difficulty is remarkably relaxed by synfiring, which resolves singularity. The synfiring scheme has been studied extensively, but this is the first time it is analyzed in terms of the Fisher information. We suggest a new method of analyzing a general binding problem in terms of the Fisher information in the framework of information geometry (Amari & Nagaoka, 2000).

Difficulty of Singularity in Population Coding

843

2 Fisher Information for a Single Stimulus We recapitulate the population coding in a neural field, summarizing the analysis of the Fisher information for a single stimulus. Details are found in the work of Wu et al. (2001). Consider a population of neurons located on a line, which we call a one-dimensional neural field, and the position of a neuron is denoted by a coordinate z. Let r (z) be the firing rate of the neurons at around position z. Neurons are excited in response to a stimulus applied from the outside. We consider a set of possible stimuli of which the features are specified by one-dimensional coordinates x. The feature may be the orientation of a bar, the position of an object, or something else. There is a natural one-to-one correspondence between x and z, and a stimulus with feature x is encoded by an activity pattern r (z) over the field such that the neurons at positions z ≈ x are excited strongly, with r (z) having a peak at z ≈ x (see Figure 1). More precisely, let f (z) be a unimodal function having a peak at z = 0, and when stimulus x is applied, the expected firing rate of the neurons at around z is given by f (z − x). The function f (z) is called the tuning curve, and we use the gaussian function ϕ for f , ϕ(z) = √

1 2πa

e −z /2a , 2

2

(2.1)

for the ease of analysis, where a represents the width of the tuning function. Neuronal firing is stochastic in its nature. Given stimulus x, the observed firing pattern r (z; x) is given by r (z; x) = f (z − x) + σ ε(z),

(2.2)

where ε(z) denotes random fluctuations with intensity σ . We assume that ε(z) is gaussian with mean 0. Firing of neurons at positions z and z is correlated in general, so that we put


ε(z) = 0,     ε(z)ε z = h z − z ,

(2.3) (2.4)

where < > denotes expectation and h denotes the covariance function. Here, h (z − z ) decreases to 0 as z − z  becomes larger. We assume the following form of the covariance function h(z), h(z) = (1 − β)δ(z) + βe −z /2b , 2

2

(2.5)

where δ(z) is the delta function, 1 − β represents the intensity of independent fluctuation proper to each neuron, and b represents the effective range

844

S. Amari and H. Nakahara

of correlated fluctuation. This form captures a variety of the correlation structure. Our previous articles (Wu et al., 2001, 2002a, 2002b) investigated the properties of the Fisher information for the single stimulus. Although the above h(z) does not include a multiplicative noise structure, it is possible to address such a case with a little modification (Yoon & Sompolinsky, 1999; Eurich & Wilke, 2000; Nakahara & Amari, 2002; Wu, Amari, & Nakahara, 2004), so that for simplicity we use only the above form in this study. The probability distribution of firing pattern r = {r (z)} given x is then written explicitly as Q(r|x) =

  ∞ ∞ 1 1 [r (z) − f (z − x)]h ∗ (z − z ) exp − 2 Z 2σ −∞ −∞  [r (z ) − f (z − x)]dzdz ,

(2.6)

where Z is the normalization factor. The function h ∗ (z) is the inverse kernel of the correlation function h, defined by 

∞ −∞

h ∗ (z − z )h(z − z )dz = δ(z − z ),

where δ is the delta function. The Fisher information for the single stimulus x is given by

 d 2 ln Q(r|x) d 2 ln Q(r|x) − Q(r|x) dr, I F (x) = E − d x2 d x2

(2.7)

where E denotes expectation. Simple calculation yields (Wu et al., 2001) 1 IF = 2 σ



    f  (z − x)h ∗ z − z f  z − x dzdz ,

(2.8)

which does not depend on x because of the translation invariance of the neural field. By using the Fourier transform, I F is given by 1 IF = 2πσ 2



∞

−∞

  ω2 exp −a 2 ω2 dω, H(ω)

(2.9)

where H(ω) is the Fourier transform of h(z),  2 2 √ b ω H(ω) = (1 − β) + β 2π b exp − . 2

(2.10)

Difficulty of Singularity in Population Coding

845

This form elucidates various characteristics of the Fisher information for a single stimulus (Wu et al., 2001). 3 Fisher Information for Two Stimuli Fisher information is useful for evaluating the accuracy of population coding. The Cram´er-Rao theorem shows that I F −1 provides a lower bound of ˆ The situation, the mean square error for the estimated feature or location x. changes drastically however, when two stimuli are presented at the same time. The response of neurons at z, given two stimuli at locations x1 and x2 , is modeled by r (z) = f (z; x1 , x2 ) + σ (z),

(3.1)

where f (z; x1 , x2 ) is the tuning curve when two stimuli with features x1 and x2 are presented. Let the intensities of the two stimuli be v and 1 − v, where we assume that the total intensity is regulated to be equal to 1. Then the tuning curve is defined by f (z; x1 , x2 , v) = (1 − v)ϕ(z − x1 ) + vϕ(z − x2 ),

(3.2)

where 0 ≤ v ≤ 1. Experimental results (Treue et al., 2000; Martinez-Trujillo & Treue, 2000) suggest this form of tuning curves. The probability distribution Q (r; x1 , x2 , v) of neural firing is given as equation 2.6, where f (z) is now set as equation 3.2. In this setting, we search for the Fisher information in population coding concerning two stimuli, (1 − v, x1 ) and (v, x2 ). Since the unknown parameters (x1 , x2 , v) are three in number, the Fisher information now becomes a 3 × 3 matrix. When the parameters are represented by a vector θ = (x1 , x2 , v),

∂ 2 ln Q(r|θ) I F (θ) = −E . ∂θ∂θ

(3.3)

We can calculate the Fisher information matrix I F in a similar way, and its inverse would give the accuracy of the estimated features of the stimuli. When x1 = x2 , the two targets merge into one, and f (z; x, x, v) = ϕ(z − x) does not depend on v, because of x1 = x2 , that is, a single stimulus. However, when the two targets are close, x1 ≈ x2 , a pathological situation occurs, and it causes an extreme difficulty in estimating v, causing nonidentifiability of v. Let us first discuss this difficulty in case of x1 = x2 for ease of understanding. The Fisher information matrix indeed degenerates, and its inverse diverges. Therefore, we cannot apply the Cram´er-Rao paradigm of statistics in this case. Such a pathological statistical model is said to be nonregular and has

846

S. Amari and H. Nakahara

been outside the scope of the standard theory of statistics, except for some pioneering work (Weyl, 1939; Hotelling, 1939) and recent developments (Dacunha-Castelle & Gassiat, 1997; Lia & Shao, 2003; Fukumizu, 2003). The analysis in the pathological case is the main topic of this letter. We also show how the singularity affects the Bayesian method and model selection. The pathological situation is elucidated by algebraic geometry, because algebraic singularities exist in the parameter space M = {(x1 , x2 , v)} of two stimuli (see Figure 2A). First, when x1 = x2 = x, the tuning curve is the same whatever v is, so that the points on a vertical line, l x : x1 = x2 = x, v = arbitrary,

(3.4)

have the same effect on the neural field (see Figure 2A, red bold vertical line). Second, when v = 1 (v = 0), there is no x1 (x2 ) stimulus, so that x1 (x2 ) cannot be decided. Hence, the points on the line l x,1 : v = 1, x1 are arbitrary, and the line l x,0 : v = 0, x2 is arbitrary (see Figure 2A, red bold horizontal lines). Thus, the points in the parameter space that have the same effect would ˜ consisting of the resultant be summarized as the same point in the set M probability distributions. Mathematically, we divide M by the equivalence relation of the same effect. The parameter space M then collapses along ˜ has algebraic singularity, those critical lines, and the resultant reduced M as shown in Figure 2B. The Fisher information matrix degenerates on these critical points. We now introduce a new method of analysis, focusing on the algebraic singularity. We begin with reparameterizing the features of stimuli as u = x2 − x1 w = (1 − v)x1 + vx2 . Here, u indicates the difference between the locations of the two stimuli, and w indicates their center of gravity. Using these, we can rewrite the tuning curve as f (z; u, w, v) = (1 − v)ϕ(z − w + vu) + vϕ(z − w − (1 − v)u).

(3.5)

With these parameters (w, u, v), we evaluate the accuracy of coding in the neighborhood of u ≈ 0. Without loss of generality, we rescale the z-axis to let a = 1, where x1 , x2 , and u should be rescaled as well as the width b of the correlation. Using the Chebyshev-Hermite expansion with respect to u of equation 3.5, we get, f (z; u, w, v) =

∞  c n (v) n=0

n!

 u Hn (z − w) ϕ(z − w), n

Difficulty of Singularity in Population Coding

A

847

B

v

x2 x1

~ M

M

D

C 1

θ3

v

0.75

x 10-4 1.5 1

0.5 0.25 0 0

0.04

u

0.08 0.1

v

0

-1 -1.5 0

0.4

θ2

0.8

1.2 x 10-3

Figure 2: (A) Parameter space M = {(x1 , x2 , v)}. The critical set consists of three surfaces with red lines. The three red thicker lines correspond to a set of those parameters that are equivalent to one stimuli and give the same probability ˜ where the equivalent set of parameters is distribution. (B) The reduced space M, reduced to one point. (C) Regular (u, v)-plane, where the part u < 0, v = 0.5 + v0 , is equivalent to u > 0, v = 0.5 − v0 by interchanging x1 and x2 . (D) The (u, v)plane embedded in regular space (θ1 , θ2 ) by a cusp-type singular map. Note that the line u = 0 in (C) (the light blue line) is mapped to a point in D.

where c n (v) ≡ (1 − v)(−v)n + v(1 − v)n and Hn (z) is the Chebyshev-Hermite polynomials. We approximate f (z; u, w) by taking the summation up to 3, having   1 1 2 3 f (z; u, w, v) = 1+ c 2 (v)u H2 (z − w)+ c 3 (v)u H3 (z−w) ϕ(z − w), 2 6 c 0 = 1, c 1 = 0, c 2 = v(1 − v), c 3 = −v(1 − v)(2v − 1) H0 (z) = 1, H1 (z) = z, H2 (z) = z2 − 1, H3 (z) = z3 − 3z.

848

S. Amari and H. Nakahara

Let us now introduce the set of new parameters θ = (θ1 , θ2 , θ3 ) in which f is written as f (z; θ) = {1 + θ2 H2 (z − θ1 ) + θ3 H3 (z − θ1 )} ϕ(z − θ1 ),

(3.6)

and consider a new statistical model S parameterized by θ in which the probability distributions are given by equation 2.6 and f is given by equation 3.6. Our original statistical model M is parameterized by ξ = (w, u, v) and is included in S by the mapping  c 2 (v)u2 c 3 (v)u3 θ = w, , . 2 6

(3.7)

The mapping (3.7) from ξ to θ is singular, because its Jacobian determinant,    ∂θ  |J | =   , ∂ξ

(3.8)

vanishes at u = 0. We adopt the strategy of calculating the Fisher information I FS (θ) of S, which is regular in the coordinates θ. The singular I F (ξ) of M is given through the singular mapping equation 3.7. This method elucidates the singular structure of M. We first evaluate I FS (θ) at around θ2 = θ3 = 0, which corresponds to u = 0. The elements of I FS (θ) = (Ikl ) are calculated as Ikl =

1 σ2



    ∂k f (z; θ)h ∗ z − z ∂l f z ; θ dzdz ,

where ∂k f = (∂/∂θk ) f . In terms of the Fourier transform, they are 1 Ikl = 2πσ 2



Fk (ω)H ∗ (ω)Fl (ω)dω,

(3.9)

where Fk (ω) = ∂ F (ω)/∂θk and F (ω) is the Fourier transform of f (z). The Fisher information matrix is explicitly calculated as  g1 0 −g2 I FS (θ) =  0 g2 0  , −g2 0 g3 

where gk =

1 2πσ 2



ω2k exp(ω2 ) dω > 0. H(ω)

Difficulty of Singularity in Population Coding

849

For ease of further study, we modify the parameters from θ to θ ∗ , θ1∗ = θ1 +

g2 θ3 , g3

(3.10)

θ2∗ = θ2 ,

θ3∗ = θ3 .

(3.11)

Then the Fisher information matrix is diagonalized, 

 g1 0 0 I FS (θ ∗ ) =  0 g2 0  , 0 0 g¯ 3 where g¯ 3 = g3 − g22 /g1 > 0. It is now easy to obtain the Fisher information matrix I F (ξ) in terms of the parameters ξ = (w, u, v), which represent the features of the stimuli. Let J be the Jacobian of the transformation from ξ to θ ∗ ,  1 ∂θ ∗ J = = 0 ∂ξ 0

(1/2)g2 g3−1 c 3 u2 c2 u (1/2)c 3 u2

 (1/6)g2 g3−1 c 3 u3 (1/2)c 2 u2  , (1/6)c 3 u3

(3.12)

where c i = dc i (v)/dv. Then the Fisher information I F (ξ) is obtained from I F (ξ) = J T I FS (θ ∗ ) J .

(3.13)

4 Algebraic Singularity and Accuracy of Estimation Although the Fisher information I FS (θ ∗ ) is regular, I F (ξ) is singular at u = 0, because the Jacobian J of the transformation is singular at u = 0, |J | = 0. In other words, the present model M is embedded in a regular space as ˜ and the singularity cannot be resolved by coordinate a singular subset M, transformation. Let us highlight the role of the singular mapping. Since the structure of the field is homogeneous along the w-axis, we focus on the mapping between (u, v) and (θ2∗ , θ3∗ ) =



v(1 − v)u2 v(1 − v)(1 − 2v)u3 , 2 6

.

(4.1)

Figure 2C shows the u-v plane, where the v-axis (indicated by a thick blue vertical line in the figure) corresponding to u = 0 degenerates to a single point in the θ-plane, causing the singularity. Horizontal (black) and vertical (red) lines simply indicate the grids of v and u values. Figure 2D shows the regular θ2 -θ3 plane in which the grids of (u, v) in Figure 2C are embedded.

850

S. Amari and H. Nakahara

Note that Figure 2C corresponds to a slice vertically cut from Figure 2A. Now consider the line v = 1/2 + v0 (solid horizontal blue line) in Figure 2C. It corresponds to the stimuli of intensity 1/2 + v0 , of which distance u changes arbitrarily. Note that when u < 0, it corresponds to the stimuli of intensity 1/2 − v0 , where stimulus 1 and stimulus 2 are interchanged. Hence, we consider the two blue horizontal lines as one component. The line is mapped to a cusp line (black or blue pair) in Figure 2D. The line v0 = 0 corresponds to the horizontal line θ3∗ = 0 in the right figure, and as v0 increases, the angle of the corresponding cusp becomes larger. As v0 tends to 1/2, the cusp shrinks and reduces to the origin. This gives rise to the singularity corresponding to v = 0 or 1, another type of singularity, which we do not address in this article. Note that the space of θ ∗ itself corresponds to a regular statistical model with a nondegenerate Fisher information matrix. The space of ξ is embedded in the θ ∗ space by the mapping with the singularity of the cusp structure (see  Figure  2D). ˆ u, ˆ vˆ be the maximum likelihood estimator. The errors wˆ = Let ξˆ = w, wˆ − w, uˆ = uˆ − u, vˆ = vˆ − v are given by ξˆ =

∂ξ ˆ ∗ −1 ˆ ∗ ∗ θ = J θ . ∂θ

(4.2)

The covariance matrix of the error is given by I F−1 (ξ), but the inversion of I F is complicated, so we evaluate the error directly by using equation 3.13. ∗ Since three components of θˆ are independently distributed, we obtain, −1 by evaluating J carefully, the asymptotic evaluation of the variances of the ˆ errors ξ, g3 , g1 g¯3

(4.3)

ˆ ≈ k1 u−4 + k2 u−2 , V(u)

(4.4)

ˆ ≈ k3 u−6 , V(v)

(4.5)

ˆ ≈ V(w)

where k1 = 36(1 − 2v)2 v−2 (1 − v)−2 g¯3 −1 ,  2 k2 = 4 6v2 − 6v + 1 v−2 (1 − v)−2 g2−1 ,

(4.6)

k3 = 144g¯3−1 .

(4.8)

(4.7)

When v = 1/2, k1 , the coefficient of u−4 vanishes. The above results highlight the pathological nature of estimation:

r

For w, the Fisher information is of order 1 even when u → 0. Hence, one can estimate the center of gravity of two stimuli accurately.

Difficulty of Singularity in Population Coding

r

r

851

When v = 1/2, the variance of uˆ diverges to infinity in the order of 1/u4 as u → 0. The estimation of v is worse, because its variance diverges in the order of 1/u6 . These show the difficulty in estimating them accurately. When v = 1/2, that is, the intensities of two stimuli are the same, the dominant term in equation 4.6 vanishes, so that the variance of uˆ is of order 1/u2 . Hence, estimation is carried out more accurately under such a situation.

Turning to the estimation of the original locations x1 and x2 , we have ˆ xˆ 1 = wˆ − vˆu, ˆ u, ˆ xˆ 2 = wˆ + (1 − v)

(4.9) (4.10)

  so that terms of O 1/u6 dominate in their error variances. In conclusion, one can estimate the center of gravity accurately, but it is very difficult to estimate their intensities and difference of locations, or the respective locations x1 and x2 , as u tends to 0. We have obtained the order of divergence, u−4 and u−6 . More rigorous mathematical analysis when v = 1/2 is given in Burnashev and Amari (2002) by using the technique of large deviation.

5 Model Selection and Singularity We have discussed the behavior of the maximum likelihood estimator (MLE) when two stimuli are presented. When the two are located closely, the estimation of the two locations is difficult. One may decide the number of stimuli (one or two in the present case) after the estimation of the parameters, but one may decide the number before estimation based on the encoded pattern r and then perform estimation. We apply different statistical models: one including single stimulus and the other including two stimuli. By comparing the results, one can decide the number. This is the problem of model selection. There are two typical methods frequently used in model selection. One is the Akaike information criterion (AIC), which evaluates the KullbackLeibler divergence between the true distribution and the estimated distribution based on the two models. The other is the Bayesian information criterion (BIC), which evaluates the posterior probabilities of the two models. The minimum descriptive length (MDL) criterion gives practically the same criterion as the BIC, although the underlying philosophy is different. One may apply these criteria in the singular case, obtaining a practical answer to decide the number of stimuli. However, all the criteria (AIC, BIC, MDL) are based on the statistical analysis of the behavior of the MLE under each model. In these analyses, the estimator is assumed to be asymptotically

852

S. Amari and H. Nakahara

subject to the gaussian distribution with the covariance matrix equal to the inverse of the Fisher information matrix. Since the Fisher information is degenerate in the present singular case, the MLE is not subject to the gaussian distribution even asymptotically. So one should be very careful in applying these criteria in hierarchical models including singularity, because no mathematical justification exists.

6 Bayesian Inference and Singularity The Bayesian inference presumes an adequate prior distribution over the parameters of the model. One may use a parametric family of priors, including hyperparameters, thus introducing a hierarchical structure of Bayesian inference. Given observed data r, one can calculate the posterior probability of the parameters under the model p(ξ|r) =

π(ξ) p(r|ξ) , p(r)

(6.1)

where π(ξ) is the prior distribution and p(r) is the marginal distribution of r. When one wants to combine the problem of model selection, where we presume a number of models M1 , M2 , · · · , and the posterior is p (ξi , Mi |r) =

π (Mi ) π (ξi |Mi ) p (r|ξi ) , p(r)

(6.2)

where π (Mi ) is the prior of the model Mi with parameters ξi . One may use the marginal distribution p(Mi |r) for model selection (BIC). The MAP estimator is the one that maximizes the posterior distribution. This is a point estimator, and it is known that the MAP estimator includes the same amount of Fisher information as the MLE. The Bayesian predictive distribution is obtained by using the posterior distributions of parameters to give a distribution of new data. The predictive distribution does not belong to the model, but is optimal when the prior is true. In general, it gives a good estimator, provided the prior is not extreme. The amount of Fisher information included in it is larger than that included in the MAP or MLE estimator, but the difference is small or only higher order, which can be revealed by calculating the curvature direction of the ancillary statistics (Komaki, 1996). When a smooth, positive prior is used, all the Bayesian estimators are asymptotically the same in a regular statistical model. One choice is the Jeffreys noninformative prior, which uses the square root of the determinant of the Fisher information matrix, and its optimality is guaranteed by information theory (Clarke & Barron, 1990). Hence, the performance of the

Difficulty of Singularity in Population Coding

853

Bayesian inference is asymptotically guaranteed to be good, and if one can choose an adequate prior based on prior knowledge, its performance is better. However, when one chooses an extreme prior, say diverging to infinity at some points or zero on some region, the result of the inference is biased and strongly affected by the chosen prior. When the statistical model is singular, as is our model, a serious problem arises. The Jeffreys prior converges to 0 in the critical region of parameters. Hence, this is an extreme case. One usually assumes a smooth prior, in particular, the uniform prior, on the model M. Instead of the parameter space ˜ of behaviors, which are the space of all M, we may consider the space M ˜ looks very natural to be the tuning functions f (z, ξ ). A smooth prior on M assumed, but it becomes singular on M because the critical set on M cor˜ responds to an infinite number of the same probability distributions in M. Therefore, the uniform (or smooth) prior counts a probability distribution ˜ infinitely many times in M. If we assume the uniform (or belonging to M ˜ of behaviors, then such multiplicity smooth) prior on the reduced model M disappears, as the Jeffreys prior does. However, such a prior becomes 0 on the critical region of the original parameter space. Therefore, the prior becomes singular in a singular statistical model. There has been little study on the Bayesian inference in a singular statistical model except for the algebraic geometry theory of Watanabe (2001a, 2001b, 2001c). We can observe that the uniform prior favors a smaller model (one stimulus in our case), so that it suppresses automatically larger models, avoiding overfitting. When the true distribution is in a smaller model, it gives a better performance. Watanabe and his colleagues have studied the Bayesian inference in singular models by using algebraic geometry, which we do not recapitulate here. 7 Synfiring Solution How can the brain solve the difficulty in the singular situation? We propose a possible solution in the scheme of synchronous firing. Given two stimuli representing different objects, the synfire hypothesis (Abeles, 1991; Diesmann, Gewaltig, & Aertsen, 1999; von der Malsburg, 1999; Singer, 1999) poses that the responses due to one object are synchronized, while those by the other fire synchronously in another phase. For simplicity, we assume there are only two phases, phase 1 and phase 2, and object 1 excites neurons in phase 1 with degree α and in phase 2 with the complementary degree α¯ = 1 − α. As for object 2, α and α¯ are interchanged. Hence, the responses of the neural field in phases 1 and 2 are, respectively, r1 (z) = α v¯ ϕ (z − x1 ) + αvϕ ¯ (z − x2 ) + σ ε1 (z)

(7.1)

r2 (z) = α¯ v¯ ϕ (z − θ1 ) + αvϕ (z − θ2 ) + σ ε2 (z),

(7.2)

854

S. Amari and H. Nakahara

where v¯ = 1 − v and εi (z) are independent noises. The degree of synchrony is represented by α, and there is no synchrony when α = 1/2. For r˜ = (r1 (z), r2 (z)) ,

(7.3)

the probability of r˜ is the product of Q (r1 (z)) and Q (r2 (z)) of equation 2.6, Q (˜r) = Q (r1 ) Q (r2 ) ,

(7.4)

because r1 and r2 are independent since ε1 and ε2 are independent. Hence, the Fisher information is the sum of those in phases 1 and 2. We calculate the Fisher information I F1 (ξ) included in r1 directly at u = 0. It is given by  −(α− α) ¯ (α v¯ + αv) ¯ vv¯ g1 0 ¯ 2 g1 (α v¯ + αv) . I F1 (ξ)=−(α− α) ¯ (α v¯ + αv) ¯ vv¯ g1 0 ¯ 2 (vv¯ )2 g1 (α− α) 2 0 0 ¯ g0 (α− α) 

(7.5) Its determinant vanishes so that r1 has singular information. The I F2 (ξ) in phase 2 is given by interchanging α and α¯ in I F1 . The total Fisher information is I F (ξ) = I F1 (ξ) + I F2 (ξ)  cg1 = (α − α) ¯ 2 (v − v¯ ) g1 0

¯ 2 (v − v¯ ) g1 (α − α) 2 (α − α) ¯ 2 (vv¯ )2 g1 0

 0 , 0 2 (α − α) ¯ 2 g0 (7.6)

   ¯ v¯ , which is nonsingular even at u = 0, except c = α 2 + α¯ 2 v2 + v¯ 2 + 4α αv for the case with α = α¯ = 1/2. The effect of synchrony is dramatic even when the degree of synchrony is weak. The population coding is no longer singular, keeping sufficient information to distinguish the two objects. We next discuss a possible mechanism of inducing synchrony.

8 Remarks on Binding Different Features A more general aspect of the binding problem is discussed in terms of correlated or synchronous firing. Let an object O have two different

Difficulty of Singularity in Population Coding

855

features, say, location and orientation, represented by analog features x and x  , respectively. In a typical example in the binding problem, x represents the shape parameter and x  the color of an object. Assume that two  objects   O1 and O2 are presented, with respective features x1 , x1 and x2 , x2 . If  there exists a neural field representing the features (x, x ) jointly, when one object (x, x  ) is applied, the tuning function f (z, z ) in the two-dimensional field with coordinates (z, z ) has a peak at (x, x  ) of the presented object, and the neural response is its noisy version r (z, z ) = f (z − x, z − x  ) + σ ε(z, z ). When two objects are presented simultaneously, the response is     r (z, z ) = (1 − v)ϕ z − x1 , z − x1 + vϕ z − x2 , z − x2 + σ ε(z, z ). (8.1)     Hence, r has two peaks at x1 , x1 and x2 , x2 , and there is no difficulty for binding xi with xi , provided the two peaks are separated. When they are close, the above type of singularity occurs. It is not plausible to assume the existence of a field combining any two features. There will be, in general, two different fields, Z and Z —one representing x and the other x  . Given two objects, two peaks are aroused in field Z at x1 and x2 , and two peaks in field Z at x1 and x2 , in the respective response patterns: r (z) = f (z; x1 , x2 ) + σ ε(z),       r  z = f z ; x1 , x2 + σ ε z .

(8.2) (8.3)

The problem is to find which peak, x1 or x2 , is bound to which peak, x1 or x2 . Synchronization is a solution, as we have analyzed. Synchrony is generated by the correlated firing of neurons. A simple mechanism causing pairwise and higher-order correlations has already been shown (Amari, Nakahara, Wu, & Saka, 2003). We extend that model to be applicable in our case. Here, we assume spiking neurons in a neural field Z. Let p(z) be the potential of the neuron at z, and the neuron fires when p(z) exceeds a threshold. The potential is given by the tuning function disturbed by fluctuating noises, p(z) =

1 1 ϕ (z − x1 ) + ϕ (z − x2 ) + σ ε(z), 2 2

(8.4)

where the noise term is decomposed as ε(z) = ϕ (z − x1 ) ε1 (z) + ϕ (z − x2 ) ε2 (z) + ε0 (z).

(8.5)

856

S. Amari and H. Nakahara

Here, ε1 , ε2 , and ε0 (z) are independent and gaussian random variables. The noise terms ε1 and ε2 are multiplicative, representing the Poisson character of spiking neurons. When ε1 , ε2 , and ε0 are temporally independent (or temporal correlations decay to 0 quickly), we have a model of spiking neurons. The potential in field Z has a similar nature,     p  (z ) = (1 − v)ϕ z − x1 + vϕ z − x2 + σ ε (z ),

(8.6)

where the noise in field Z is also decomposed as     ε (z ) = ϕ z − x1 ε1 (z ) + ϕ z − x2 ε2 (z ) + ε0 (z ).

(8.7)

It is plausible that noises ε1 and ε2 originate from the objects O1 and O2 , respectively, and are common to fields Z and Z . It is analyzed in Amari et al. (2003) that synchrony emerges in such a model. This can solve the binding problem. More detailed analysis using the Fisher information is in our future program. 9 Discussion This article investigated the Fisher information when two stimuli are presented together. Since ordinary statistical analysis fails in this case due to the degeneracy of the Fisher information, we introduced a novel method using algebraic singularity, and this method allowed us to examine not only the Fisher information but also the algebraic structure of the degeneracy. We analyzed the asymptotic property of how the information decays as the locations of the two stimuli near each other, although we do not propose any decoding algorithms. We then searched for a possible neural mechanism to resolve singularity. This is given by the synchronization of the neural firing in population coding. We analyzed how the Fisher information behaves under synchronization. We finally discussed a possible mechanism of generating synchronization in relation to the binding problem. Our method synthesizes various disciplines from many fields. Acknowledgments H.N. is supported by Grants-in-Aid 14780614 and 14017106 from the MEXT, Japan. References Abeles, M. (1991). Corticonics. Cambridge: Cambridge University Press. Amari, S. (1977). Dynamics of pattern formation in lateral-inhibition type neural fields. Biological Cybernetics, 27, 77–87.

Difficulty of Singularity in Population Coding

857

Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10, 251–276. Amari, S., & Arbib, M. A. (1977). Competition and cooperation in neural nets. In J. Metzler (Ed.), Systems neuroscience (pp. 119–165). San Diego: Academic Press. Amari, S., & Nagaoka, H. (2000). Methods of information geometry. New York: AMS and Oxford University Press. Amari, S., Nakahara, H., Wu, S., & Sakai, Y. (2003). Synchronous firing and higherorder interactions in neuron pool. Neural Computation, 15, 127–142. Amari, S., & Ozeki, T. (2001). Differential and algebraic geometry of multilayer perceptrons. IEICE Transactions on Fundamentals of Electronics Communications and Computer Sciences, E84-A(1), 31–38. Burnashev, M. V., & Amari, S. (2002). On some singularities in parameter estimation problems. Problems of Information Transmission, 38, 328–346 Clarke, B. S., & Barron, A. R. (1990). Information theoretic asymptotics of Bayes methods. IEEE Transactions on Information Theory, 36, 453–471 Dacunha-Castelle, D., & Gassiat, E. (1997). Testing in locally conic models, and application to mixture models. Probability and Statistics, 1, 285–317. Diesmann, M., Gewaltig, M-O., & Aertsen, A. (1999). Stable propagation of synchronous spiking in cortical neural networks. Nature, 402, 529–533. Eurich, C. W., & Wilke, S. D. (2000). Multidimensional encoding strategy of spiking neurons. Neural Computation, 12(7), 1519–1529. Fukumizu, K. (2003). Likelihood ratio of unidentifiable models and multilayer neural networks. Annals of Statistics, 31, 833–851. Fukumizu, K., & Amari, S. (2000). Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks, 13(3), 317–327. Hotelling, H. (1939). Tubes and spheres in n-spaces, and a class of statistical problems. Amer. J. Math., 61, 440–460. Ingle, D. (1968). Visual releasers of pre-catching behavior in frogs and toads. Brain Behavior and Evolution, 1, 500–518. Komaki, F. (1996). On asymptotic properties of predictive distributions. Biometrika, 83, 299–313. Liu, X., & Shao, Y. (2003). Asymptotics for likelihood ratio tests under loss of identifiability. Annals of Statistics, 31(3), 807–832. Mart´ınez-Trujillo, J. C. M., & Treue, S. (2000). Attention modulates tuning curves from two stimuli inside the receptive field of MT/MST cells in the macaque monkey. Society for Neuroscience Abstracts, 26, 251.16. Nakahara, H., & Amari, S. (2002). Attention modulation of neural tuning through peak and base rate in correlated firing. Neural Networks, 15, 41– 55. Nakahara, H., Wu, S., & Amari, S. (2001). Attention modulation of neural tuning through peak and base rate. Neural Computation, 13, 2031–2048. Pouget, A., Dayan, P., & Zemel, R. (2000). Information processing with population codes. Nature Neuroscience Reviews, 1, 125–132. Pouget, A., Zhang, K., Deneve, S., & Latham, P. E. (1998). Statistically efficient estimation using population coding. Neural Computation, 10, 373–401. Reichardt, W., & Poggio, T. (1975). A theory of the pattern induced flight orientation of the fly, Muska domestica, II. Biological Cybernetics, 18, 69–80.

858

S. Amari and H. Nakahara

Sahani, M., & Dayan, P. (2003). Doubly distributional population codes: Simultaneous representation of uncertainty and multiplicity. Neural Computation, 15, 2255–2279. Seung, H. S., & Sompolinsky, H. (1993). Simple models for reading neuronal population codes. Proc. Nat. Acad. Sci. USA, 90, 10749–10753. Singer, W. (1999). Neuronal synchrony: A versatile code for the definition of relations? Neuron, 24, 49–65. Treue, S., Hol, K., & Rauber, H. J. (2000). Seeing multiple directions of motionphysiology and psychophysics. Nature Neuroscience, 3, 270–276. von der Malsburg, C. (1999). The what and why of binding: The modeler’s perspective. Neuron, 24, 95–104. Watanabe, S. (2001a). Algebraic analysis for non-identifiable learning machines. Neural Computation, 13, 899–933. Watanabe, S. (2001b). Algebraic geometrical methods for hierarchical learning machines. Neural Networks, 14(8), 1409–1060. Watanabe, S. (2001c). Algebraic information geometry for learning machines with singularities. In T. K. Leen, T. G. Dieterrich, & V. Tresp (Eds.), Advances in neural information processing systems, 13 (pp. 329–336) Cambridge, MA: MIT Press. Weyl, H. (1939). On the volume of tubes. Amer. J. Math., 61, 461–472. Wu, S., Amari, S., & Nakahara, H. (2002a). Asymptotic behaviors of population codes. Neurocomputing, 44–46, 697–702. Wu, S., Amari, S., & Nakahara, H. (2002b). Population coding and decoding in a neural field: A computational study. Neural Computation, 14, 999–1026. Wu, S., Amari, S., & Nakahara, H. (2004). Information processing in a neuron ensemble with the multiplicative correlation structure. Neural Networks, 17(2), 205–214. Wu, S., Chen, D., Niranjan, M., & Amari, S. (2003). Sequential Bayesian decoding with a population of neurons. Neural Computation, 15(5), 993–1012. Wu, S., Nakahara, H., & Amari, S. (2001). Population coding with correlation and an unfaithful model. Neural Computation, 13, 775–797. Yoon, H., & Sompolinsky, H. (1999). The effect of correlations on the Fisher information of population codes. In M. S. Kearns, S. A. Solla, & D. A. Cohn (Eds.), Advances in neural information processing, II (pp. 167–173). Cambridge, MA: MIT Press. Zemel, R. S., Dayan, P., & Pouget, A. (1998). Probabilistic interpretation of population codes. Neural Computation, 10, 403–430. Zemel, R. S., & Pillow, J. (2002). A probabilistic network model of population responses. In R. P. N. Rao, B. A. Olshausen, & M. S. Lewicki (Eds.), Probabilistic models of the brain: Perception and neural function (pp. 223–242). Cambridge, MA: MIT Press. Zhang, K., & Sejnowski, T. J. (1999). Neural tuning: To sharpen or broaden. Neural Computation, 11, 75–84.

Received November 24, 2003; accepted August 31, 2004.

LETTER

Communicated by Laurence Abbott

Neurons Tune to the Earliest Spikes Through STDP Rudy Guyonneau [email protected] Centre de Recherche “Cerveau et Cognition,” Toulouse 31000, France, and Spikenet Technology, Revel, France

Rufin VanRullen [email protected] Centre de Recherche “Cerveau et Cognition,” Toulouse 31000, France

Simon J. Thorpe [email protected] Centre de Recherche “Cerveau et Cognition, Toulouse 31000, France, and Spikenet Technology, Revel 31250, France

Spike timing-dependent plasticity (STDP) is a learning rule that modifies the strength of a neuron’s synapses as a function of the precise temporal relations between input and output spikes. In many brains areas, temporal aspects of spike trains have been found to be highly reproducible. How will STDP affect a neuron’s behavior when it is repeatedly presented with the same input spike pattern? We show in this theoretical study that repeated inputs systematically lead to a shaping of the neuron’s selectivity, emphasizing its very first input spikes, while steadily decreasing the postsynaptic response latency. This was obtained under various conditions of background noise, and even under conditions where spiking latencies and firing rates, or synchrony, provided conflicting informations. The key role of first spikes demonstrated here provides further support for models using a single wave of spikes to implement rapid neural processing.

1 Introduction Activity-dependent learning at the systems level relies on dynamical modifications of synaptic strength at the cellular level. Experimental data have shown that these modifications depend on temporal pairing between a preand a postsynaptic spike: an excitatory synapse receiving a spike before a postsynaptic one is emitted potentiates while it weakens the other way around (Markram, Lubke, Frotscher, & Sakmann, 1997). The amount of modification depends on the delay between these two events: maximal when preand postsynaptic spikes are close together, the effects gradually decrease Neural Computation 17, 859–879 (2005)

© 2005 Massachusetts Institute of Technology

860

R. Guyonneau, R. Van Rullen, and S. Thorpe

and disappear with intervals in excess of a few tens of milliseconds (Bi & Poo, 1998; Zhang, Tao, Holt, Harris, & Poo, 1998; Feldman, 2000). Learning at the neuronal level thus heavily depends on the temporal structure of neuronal responses. Interestingly, in many brain areas, the temporal precision of spikes during stimulus-locked responses can be in the millisecond range. The evidence is clear in both the auditory cortex (Heil, 1997) and the somatosensory system (Petersen, Panzeri, & Diamond, 2001). Reproducible temporal structure can also be found in the visual system, such as in MT (Bair & Koch, 1996), and from retinal ganglion cells to the inferotemporal cortex (Sestokas, Lehmkuhle, & Kratz, 1991; Meister & Berry, 1999; Liu, Tzonev, Rebrik, & Miller, 2001; Richmond & Optican, 1990; Victor & Purpura, 1996; Nakamura, 1998). This raises the question of how a neuron with STDP will respond when repeatedly stimulated with the same pattern of input spikes. Let us consider a simple case where a neuron is exposed to an input spike pattern

Figure 1: Facing page. Single wave experiment. For a repeated spike wave under realistic conditions, a neuron learns to react faster to its target. Synaptic weights converge onto the earliest firing afferents, even with 25 ms jitter or 50 Hz background activity. (A) Typical incoming activity. Bottom: raster plot of a jittered spike wave amid spontaneous activity. At each presentation, 5 ms jitter and 5 Hz background activity are regenerated; the resulting pattern is presented to the postsynaptic neuron (with initial potential set to 0); when it spikes, the STDP learning rule is applied and its potential reset to 0 before going to the next presentation. Prior to presentation, the presynaptic neurons do not fire any spike. Top: the corresponding poststimulus time histogram (PSTH): the gaussian form in the left-most part corresponds to the reproduced spike wave. (B) Dynamics of repeated STDP. Top: sum of all synaptic weights at each presentation (the dashed line represents the output neuron threshold). The sum of the synaptic weights stored in the afferents stabilizes at threshold value. Bottom: the horizontal axis corresponds to the number of presentations (i.e., the learning step). The black line refers to the left axis and shows the reduction of postsynaptic latency during the course of learning. The background image refers to the right axis where each synapse weight is mapped by a gray-level index (see the corresponding bar on the right). Synapses are ordered by spiking latency of the corresponding neuron within the original reproducible input pattern (here, a wave), more precisely before superimposing spontaneous activity or adding some jitter. This order is decided a priori and stays fixed for the whole simulation. During learning, earliest synapses become fully potentiated and later ones are weakened. (C) Effect of jitter. Jitter is generated by a gaussian distribution. Increasing its standard deviation does not affect convergence until about 10 ms. From there, it slows the system roughly quadratically. (D) Effect of spontaneous firing rate. Increasing background activity slows convergence approximatively linearly.

Neurons Tune to the Earliest Spikes Through STDP

861

consisting of one input spike on each afferent synapse, with arbitrary but fixed (e.g., gaussian distributed, as in latencies. For one given input pattern presentation, the input spikes elicit a postsynaptic response, triggering the STDP rule. Synapses carrying input spikes just preceding the postsynaptic one are potentiated, while later ones are weakened. The next time this input pattern is re-presented, firing threshold will be reached sooner, which implies a slight decrease of the postsynaptic spike latency. Consequently, the learning process, while depressing some synapses it had previously potentiated, will now reinforce different synapses carrying even earlier spikes than the preceding time. By iteration, it follows that when the presentation of the same input spike pattern is repeated, the postsynaptic spike latency will tend to stabilize at a minimal value while the first synapses become fully potentiated and later ones fully depressed (see Figure 4 in Song, et al., 2000, as well as Gerstner and Kistler, 2002a, for related demonstrations; see also Figures 1A and 1B). Under these simplistic conditions, neurons can learn to react faster to a given stimulus pattern by emphasizing the role of the earliest firing inputs. However, input spike patterns in the brain do not consist of onespike-per-neuron waves of infinite precision. Not only do stimulus-locked responses occur with a jitter on the order of 1 or more milliseconds (Mainen

862

R. Guyonneau, R. Van Rullen, and S. Thorpe

& Sejnowski, 1995), neocortical cells also spontaneously fire at rates up to 20 Hz in awake animals (Evarts, 1964; Hubel, 1959; Steriade, Oakson, & Kitsikis, 1978). Finally, temporal structure in spike trains is often distributed over numerous consecutive spikes in long time windows. With realistic conditions, would the effect of STDP still emphasize the very first afferents? Would this be the case even when other types of neural codes (firing rate, synchrony) are present within the spike trains? This would have critical implications for our understanding of neural information coding and processing. In this theoretical study, we seek to answer these questions through a range of simulations using biologically plausible neuron models. 2 Spike Waves A target neuron is repeatedly presented with an input spike pattern. In a first example, it consists of a single asynchronous spike wave: one spike for each synapse with gaussian-distributed latencies (50 ms mean, 20 ms width). At each presentation, spike times are jittered, and Poisson spontaneous activity is added to the spike trains. We varied the amount of jitter and spontaneous firing and investigated their effect on the target neuron learning behavior. 2.1 Methods. 2.1.1 Integrate and Fire. The postsynaptic neuron is connected to a thousand afferent neurons via as many synapses and integrates spikes across time (in all simulations except one, no leakage was involved). Systematically starting at a resting level of 0 before presentation, it sums the weight of the synapses that carry each incoming spike. The presynaptic neurons are assumed to be quiet prior to presentation; they do not fire any spike. When reaching its threshold (100 for all simulations), the postsynaptic neuron fires, triggers an STDP-inspired learning rule for its first action potential only, and then sets its potential back to a resting level of 0. 2.1.2 STDP Model. The learning function F has the prototypical form of STDP according to electrophysiological results in cultured hippocampal neurons (Bi & Poo, 1998). The amount of synaptic modification arising from a single pair of pre- and postsynaptic spikes separated by time
t is expressed as follows: if τ+ ≤
t ≤ 0, if 0 <
t ≤ τ− , otherwise

F (
t) = A+ . (1 − (
t/τ +)) F (
t) = −A− . (1 − (
t/τ −)) F (
t) = 0,

where A+ and A− determine the maximum amounts of synaptic modification that occur when
t is close to zero (A+ = A− = 1 in all simulations). τ+ and τ− are the temporal windows for, respectively, potentiation and

Neurons Tune to the Earliest Spikes Through STDP

863

depression, expressed in milliseconds (here, τ+ = −20 ms and τ− = 22 ms). Synaptic growth from learning step n to n + 1 is computed as follows: wi (n + 1) = wi (n) + F (tpre − tpost ), where wi (n) is the “free” weight of synapse i at nth presentation step, tpost the postsynaptic spike timing, and tpre the presynaptic spike timing. Under these conditions, synaptic weights may grow to infinitely large values. To address this obstacle, we implemented a sigmoidal saturation function g: Wi = g(wi ), where g(x) = ((π/2) + atan(x))/π. The “free” weight, labeled wi , corresponds to the unconstrained weight of the synapse, which connects input neuroni to the postsynaptic one. It ranges from −∞ to +∞ and is the target of the strengthening/weakening process. The “effective” weights, Wi , lie in the ]wmin , wmax [ interval (here wmin = 0, wmax = 1). These “effective” weights are the ones that are considered for calculating the excitatory postsynaptic potential.1 2.1.3 Jitter. When needed, each reproducible spike was displaced in time by a value chosen randomly at each presentation from a gaussian distribution of mean 0 and standard deviation set to the expected jitter. 2.1.4 Spontaneous Activity. Spontaneous activity is generated using a Poisson process as described in section 3.1. It is redrawn for each afferent neuron, at each presentation, before being superimposed on the jittered reproducible structure to constitute the incoming activity. 2.1.5 Convergence Criterion. Convergence is met when the local average of postsynaptic latencies, in a ±5-step window, remains within 1 ms for 100 consecutive steps (reported convergence is the first of these 100 steps). We also checked that the synaptic weights were indeed selected on the basis of their earliest inputs.

1 Note that here, contrary to other update rules (Rubin, Lee, & Sompolinsky, 2001), when the effective weight approaches the upper and lower bounds, both the expected amounts of reward and punishment (in the “effective” weight domain) tend toward 0. In the multiplicative update case (Song et al., 2000), these amounts stay balanced one compared to the other: when the weight reaches a bound, for example, wmax , then reward goes to 0, whereas punition is maximal and inversely. For the additive update rule (Cateau & Fukai, 2003), both potentiation and depression are independent of the weight. If the update results in a synaptic weight outside the bounds, the weight is clipped to the boundary values.

864

R. Guyonneau, R. Van Rullen, and S. Thorpe

2.2 Resistance to Jitter and Spontaneous Activity. A typical example with 5 ms jitter and 5 Hz spontaneous activity (see Figure 1A) shows that the target neuron learns to react faster to the input pattern—in the case here, a spike wave—by selectively reinforcing the earliest firing afferents (see Figure 1B). Initially, synaptic weights are set so as to evoke the first postsynaptic response when the entire reproducible pattern, the spike wave, has been integrated (in the present case, around 120 ms after stimulus onset). Synapses are then progressively reinforced and then weakened, shortening the postsynaptic neuron latency from one step to the following. This is especially visible between steps 20 and 100, where this codependent dynamic appears as a “bright crest” going from the latest to the earliest afferences. When the postsynaptic neuron latency stabilizes, the STDP rule is systematically applied at approximately the same time: its rewarding part constantly affects the same synapses, those receiving the earliest of the reproducible spikes, thus driving them to their maximal strengths. Symmetrically, later spikes invariably arrive in the weakening part of the STDP window: the corresponding synapses are continuously depressed. This trend can also be observed in the evolution of the total amount of synaptic weight of the neuron at a given step (see Figure 1B, top). It corresponds to the potential the model neuron would reach if each of its synapses was hit by one spike only. Initially, it starts from a value well below the threshold needed for the postsynaptic neuron to fire a spike. Here, the presence of spontaneous activity allows the neuron to reach its threshold and trigger the STDP rule for the first time. As the learning process goes on, the sum of synaptic weights increases due to the increasing number of spikes, thus of synapses, falling in the rewarded part of the STDP window; tends toward the neuron’s threshold value; and stabilizes around this level. Since the maximum synaptic weight is fixed at 1.0, the neuron can reach a state where the first N spikes suffice to make it fire (where N is the output neuron’s threshold). The neuron will fire early only if it receives more or less the same pattern as the one learned during the first moments of incoming activity. In fact, it responds increasingly faster to a precise sequence of spikes than to any other (see Figure 5). Simulations showed that the convergent behavior of this trend is very robust. When jitter is increased (with spontaneous activity set to 0), the number of presentations needed for convergence stays the same until the amount of jitter reaches 10 ms; then it increases roughly quadratically (see Figure 1C). Thus, jitter has little or no effect on the consequences of repeated STDP for plausible values. Moreover, convergence of the synaptic weights onto the earliest afferents could even be obtained with jitter in the range of 20 to 25 ms, that is, as wide as the input spike times distribution itself. When increasing spontaneous activity (in the 5–50 Hz range without any jitter), the convergence is also slowed but is nonetheless obtained even at

Neurons Tune to the Earliest Spikes Through STDP

865

the highest rate (see Figure 1D). Note that in this case, the input spike wave represents less than 15% of the total spikes. As only the 10% of synapses carrying the earliest spikes are selected, this result implies that the STDP rule is able to focus exclusively on less than 2% of the spikes, those whose timing is reproducible and early, while discarding the rest. 3 Spike Trains When a single wave of spikes is the reproducible structure, it is clear that convergence to a state in which weights are heavily concentrated on the earliest firing inputs is very robust. However, it could be argued that the STDP rule focuses on the left-most part of the train only because it is the only part that contains information from one pattern presentation to the next (indeed, the right-most part contains only randomly generated spikes). 3.1 Uniform Spike Distribution. The main simulation uses an input spike pattern in the form of 500 ms–long spike trains where the temporal structure and the amount of information is statistically homogeneous across time and across afferent neurons. More specifically, spike trains were generated according to a Poisson process. Each of the 1000 excitatory afferents emits a given spike train where each interspike interval, isi, was determined according to a Poisson rule process depending on the expected rate of the train, u: isi (r ) = ln(r )/−u,

where r is a random value chosen from a uniform distribution on the interval ]0.0, 1.0[. The reproducible structure for the simulation illustrated in Figure 2 is defined once and for all using this method (u = 20 Hz for all afferent neurons). Note that we are not suggesting that neuronal responses are completely stochastic, since reproducibility implies the contrary (see, e.g., Meister & Berry, 1999). We need this only to ensure that no a priori assumptions are made as to how neural information is encoded, thanks to the homogeneous nature of Poisson processes. This basic pattern is reproduced on each presentation, undergoing 5 ms jitter and mixed with 5 Hz spontaneous activity (see Figure 2A). The initial synaptic weights are set so as to elicit the first postsynaptic response after approximately 400 ms. As in the previous simulation, latency decreases steadily from this initial value to stabilize after 1500 presentations. Conjointly, synapses carrying the first spikes become fully potentiated and later ones are fully depressed (see Figure 2B, bottom). The sum of synaptic weights displays the same dynamic as in the spike wave experiment (see section 2.2): going from a very low level, it rises until it reaches the minimal value needed to make the postsynaptic neuron fire on a single volley of

866

R. Guyonneau, R. Van Rullen, and S. Thorpe

Figure 2: Spike trains experiment. (A) Typical incoming activity. The input spike pattern consists of arbitrary 500 ms–long spike trains, where the amount of information is equally distributed over time. At each presentation, this reproducible structure is modified using 5 ms jitter and 5 Hz spontaneous activity, then presented to the postsynaptic neuron, whose potential is set to 0. Prior to presentation, the presynaptic neurons do not fire any spike. Due to the random aspect of the input design (Poisson-inspired process; see section 3.1), latencies and firing rates in the reproducible input structure were quite variable, ranging, respectively, from 0 to ≈350 ms (mean of ≈52 ± 51 ms, standard deviation) and from 4 to 44 Hz (mean 19.6 ± 6.3 Hz, standard deviation). (B) Dynamics of repeated STDP. Here again, latency decreases and stabilizes. Synapses are selected on the basis of their first-spike timings: the earliest ones are fully potentiated and the latest are weakened. The sum of synaptic weights follows the same behavior as in the spike wave experiment: it stabilizes around the output neuron’s threshold value (see Figure 1B).

spikes. At this point, the system has converged: synaptic potential flattens around a value corresponding to the postsynaptic neuron’s threshold (see Figure 2B, top). Thus, the selection of the earliest spikes through STDP is obtained even when late parts of the spike trains carry the same amount of information. 3.2 Latency versus Firing Rates and Synchrony. Note that no assumptions were made in the previous simulation as to how neurons encode information within the spike trains. Firing rates (Gerstner, Kreiter, Markram, & Herz, 1997; Shadlen & Newsome, 1998) or synchrony (Abeles, 1991) are widely believed to support the neural code. In this regard, one may think that these principles should drive the effects of STDP. Indeed, it has been argued that competition for control of the postsynaptic response would thus be won by the most correlated inputs (Song, Miller, & Abbott, 2000), or by

Neurons Tune to the Earliest Spikes Through STDP A

867

B 200

200

100

100

0

0 400

W 1 .5

300 .1 200 .01 100

0

100

200

300

Time (ms)

400

500

0

0

500

1000

1500

2000

.001

# presentations

Figure 3: Latency versus rate. (A) Typical incoming activity. The shorter the latency, the fewer spikes the input neuron emits. According to the spike train design, the tail of the PSTH contains more spikes than its head. A jitter of 5 ms is applied to each spike timing, and no spontaneous activity is superimposed on the reproducible input structure so as to rigorously control the rates of afferents. Activity is then presented to the postsynaptic neuron, whose potential is set to 0. Prior to presentation, the presynaptic neurons do not fire any spike. (B) Dynamics of repeated STDP. The same trend emerges: although receiving fewer spikes, synapses hit by the earliest trains are finally fully potentiated. Inversely, strongly firing inputs will be neglected because they fire late.

the most vigorously firing ones (Gerstner & Kistler, 2002b). However, this study points toward first-spike timing as a determining factor, emphasizing the role of temporal asynchrony in neural information coding (Gautrais & Thorpe, 1998). In the brain, short latencies are generally associated with highest firing rates, which also often result in high temporal correlations. This would make it difficult to disentangle the respective influences of these aspects of the spike train on neuronal learning. In our simulations, however, we can ask how these different aspects fare, one compared to the other, by artificially defining input spike patterns where first-spike timing is pitted against average firing rate or amount of synchrony. In the next simulation, latencies and rates have been artificially opposed: the afferent neuron with the shortest latency fires at the slowest rate over the entire window, the latest one at the highest rate, and neurons inbetween display a gradual latency-to-rate trade-off. As in the main experiment, spike times are jittered at each presentation (but spontaneous activity has been removed so as to rigorously control the firing rate of each input neuron, ranging from 4 to 44 Hz; (see Figure 3A). The result is clear as the same trend emerges (see Figure 3B). This means that a synapse receiving a very high firing rate will not be retained by STDP if it does not also correspond to one of the shortest latencies.

868

R. Guyonneau, R. Van Rullen, and S. Thorpe

A

B 400

200

200

100

0

0 400

W 1 .5

300 .1 200 .01 100

0

100

200

300

Time (ms)

400

500

0

0

500

1000 1500

2000

2500 3000 3500

.001

# presentations

Figure 4: Latency versus synchrony. (A) Typical incoming activity. The spike pattern was designed so as to oppose latency to synchrony: the shorter the latency, the shorter the synfire chain. Spontaneous activity of 5 Hz has been superimposed on the reproducible input structure and no jitter is simulated so as to rigorously keep the synchrony within the spike trains. Activity is then presented to the postsynaptic neuron whose potential is set to 0. Prior to presentation, the presynaptic neurons do not fire any spike. (B) Dynamics of repeated STDP. Once again, postsynaptic latency decreases before stabilizing, and the earliest synapses become fully potentiated while later ones are weakened. Thus, being highly correlated is not sufficient for inputs to be selected by STDP; they also have to fire with one of the shortest latencies.

In the final simulation, latency and amount of synchrony are artificially opposed: the longer the latency, the more neurons are made to share the same spike train (hence, defining longer synfire chains; Abeles, 1991). The size of these correlated groups gradually extends from 1 to 54 synchronously firing units. Here, 5 Hz spontaneous activity is added as usual, while jitter is removed so as to obtain truly synchronous waves (see Figure 4A). Once again, the results are clear: synapses carrying the very first input spikes are selected by STDP, whereas highly correlated inputs having late latencies are depressed (see Figure 4B). Note the small and eventually vanishing streaks of synaptic potential, a necessary consequence of the fact that synchronous inputs are always reinforced or depressed together by the STDP rule. Conclusively, when latency and synchrony are inversely correlated, the selection of the potentiated synapses still depends on the timing of the first reproducible spikes alone. 3.3 Selectivity Measures. Convergence of synaptic weights onto earliest afferents through STDP is now established, underlining the importance of first-spikes timing. But why do synaptic weights distribute in such a

Neurons Tune to the Earliest Spikes Through STDP

869

remarkable way? The first obvious answer is the postsynaptic spike latency reduction, as displayed in all the simulations. But having a neuron responding fast to a given event is not an interesting feature if it is not also selective to it. STDP has been shown to explain the development of direction selectivity in recurrent cortical networks, where excitatory and inhibitory synapses were modified according to their spiking activity (Rao & Sejnowski, 2000). Selectivity measures were based on whether the postsynaptic neuron fires. How should one define selectivity within the present, inhibition-less framework? Any conventional measure based on the postsynaptic firing rate is ruled out since the postsynaptic response is limited to its first spike (see sections 2.1 and 4). Under these conditions, we propose to use firing latency as a viable measure of selectivity: if a neuron spikes faster to the input pattern it learned than to any other one, it would de facto behave if not selectively, at least differentially to it. First, we generated 50,000 distractor spike trains using the same method (Poisson-inspired spike trains at 20 Hz; see section 3.1) as for the target pattern (see Figure 2A); a priori, these can be considered as equivalent to the target in terms of average latency of the spike trains, spike count, correlations, and so forth. At each learning step of the main simulation (see Figure 2B), these distractor patterns were tested for postsynaptic latency on the learning neuron and compared to 1000 responses to the target pattern, using the same conditions of background noise (5 ms jitter and 5 Hz spontaneous activity). These distractors and target presentations did not trigger the learning rule: the postsynaptic latency was measured but did not give rise to any synaptic modifications. A threshold was set around the mean response time to targets. Target spike trains yielding responses before the threshold were considered as hits and distractor ones as false alarms. Selectivity (d  ) was computed as follows: d  = z (hit rate) − z (false alarm rate), where z( p) stands for the inverse of the normal cumulative distribution function of p (Green & Swets, 1966). The expected maximum was computed using an expected hit rate of 50% and a false alarm rate of 1/2n, where n is the number of distractor patterns (here, n = 50, 000). The results (see Figure 5) show that while the neuron was initially less likely to respond to the target pattern than to arbitrary distractors (due to its initially random weight distribution), it became highly selective to its target as learning developed: after about 1500 presentations, when the postsynaptic neuron latency and weights have indeed stabilized (see Figure 2B), not one of the distractor patterns could make this neuron fire sooner than with the target one.

870

R. Guyonneau, R. Van Rullen, and S. Thorpe

Figure 5: Selectivity of the efferent neuron. The dashed line is the expected maximum value for d  (under the present conditions), gray dots stand for d  values at each learning step, and the black curve is a local average of these. The neuron becomes steadily more selective to the target pattern used for training. Initially, it tends to fire slightly more slowly to it than to 50,000 spike trains used as distractors. After 1500 presentations, when its response latency stabilizes (see Figure 2B), the postsynaptic neuron reacts to its target before any of the distractors. Note that some observations landed above the expected maximum value for d  : in certain conditions (100% hit rate and/or 0% false alarms), the observed d  can rise above the expected max value computed for an expected 50% hit rate and 1 for 100,000 false alarms.

Conclusively, a neuron exhibiting STDP will characteristically respond faster and faster to a precise repeated pattern than to any comparable one, thus becoming more selective to it, at least in the terms proposed here. 4 Discussion The temporally asymmetric STDP rule reinforces the last synapses involved in making a neuron fire. At first sight, one might think that its repeated application invariably reinforces afferents at the tail of the spike sequence firing a neuron. Instead, we have demonstrated that it focuses, in a remarkably robust manner, on the head of the spike sequence: dynamical STDP tunes a neuron to synapses transmitting reproducible spikes with the shortest latencies, thus enabling the cell to respond faster and selectively to the input

Neurons Tune to the Earliest Spikes Through STDP

871

it has learned. This result is obtained for arbitrary structures of the input spike pattern. This STDP tuning is achieved by selecting a relatively small subset of afferents, here fully determined by the model’s output threshold. In that sense, it is worth noting that only 10 to 40 fully potentiated excitatory inputs, among a possible 10,000 or so, would be enough to evoke a response (Shadlen & Newsome, 1994). The main assumption for the demonstration is that spike times within the input pattern must show a certain degree of reproducibility (which can be modulated by fairly high amounts of jitter and spontaneous activity). This is compatible with experimental observations in numerous brain structures (Heil, 1997; Petersen et al., 2001; Bair & Koch, 1996; Sestokas et al., 1991; Liu et al., 2001; Richmond & Optican, 1990; Victor & Purpura, 1996; Nakamura, 1998). This repetition of similar input patterns could quite simply be the result of multiple exposures with the same stimulus at different times in life. Alternatively, we propose that a single stimulus exposure could result in a sequence of similar processing waves through the rhythmic activity of cortical oscillations (Hopfield, 1995). Note that in both situations, one would not expect the state of the system to be the same from one step to another. In particular, the efferent neuron potential would be unlikely to have the same resting level every time the input pattern is presented. We have performed simulations where baseline fluctuations were simulated. As usual, the typical incoming activity was of the same kind as the one used in the main simulation: the reproducible input structure consisted of 1000 spike trains generated using a Poisson-inspired process (see section 3.1). At every step, a 5 ms jitter was applied to each precise spike time; then 5 Hz spontaneous activity was superimposed before being presented to the neuron. But instead of being systematically reset to 0, the postsynaptic neuron potential was set according to a gaussian process of mean 0 and width 50. Prior to presentation, the presynaptic neurons do not fire any spike. An identical convergence was reached, though less rapidly (see Figure 6). For simplicity, all of the simulations so far have been conducted with a single postsynaptic spike, no matter what the activity is afterward: a single STDP learning rule was applied at each input pattern presentation. While it may seem a controversial simplification, experimental evidence suggests that the effect of the late spikes could be neglected (Froemke & Dan, 2002). As Tsodyks (2002) commented, “The main effect of STDP is well explained by the first pair of spikes, with the additional spike having only a marginal contribution.” The first presynaptic spike is also the most relevant in evoking an excitatory postsynaptic potential (EPSP), because of synaptic depression (Thomson & Deuchars, 1994), a trend that is reinforced by synaptic redistribution, where a synapse depresses even more when potentiated (Markram & Tsodyks, 1996; see below for more details). Nonetheless, for completeness, we investigated the effects of having multiple postsynaptic spikes that repeatedly triggered the STDP mechanism. Here we implemented a potential

872

R. Guyonneau, R. Van Rullen, and S. Thorpe 200 100 0 500

W 1 0.5

400

0.1

300 0.01 200 0.001 100 0.0001 0

# presentations

Figure 6: Fluctuating potential. In this simulation, the initial efferent neuron potential was set—just before the cell is presented with the incoming activity— according to a gaussian distribution of mean 0 and standard deviation 50. Typical incoming activity is of the same kind as in the main experiment: a reproducible structure of spike trains amid 5 ms jitter and 5 Hz spontaneous activity (see Figure 2A). Prior to presentation, the presynaptic neurons do not fire any spike. Due to the fluctuation, postsynaptic neuron latencies were highly variable; the black line thus depicts a smoothed version (using a moving average of width 101 steps; ± standard deviation, dotted lines). Here, the fluctuation results in a delay of the reinforcement process applied to the reproducible structure: from step 0 to 5000, the crest-ascending motion is not obvious. But once some synapses are strong enough (i.e., displayed in white on the figure), the modification sequence starts until only the earliest firing inputs remain with large weights and the postsynaptic neuron latency has decreased to reach a stable, steady state. The sum of synaptic weights follows the same behavior as in the spike wave experiment: it stabilizes around the output neuron’s threshold value (see Figure 1B). While convergence needs more time to be reached (about 10 times more than for the original simulation; see Figure 2B), the repeated application of STDP leads the neuron with a highly fluctuating potential to tune itself on its earliest afferents and respond faster.

leak current (τ = 20 ms) in the efferent neuron. Simulation parameters and typical incoming activity were of the same kind as in the main experiment (see Figure 2A). Again, the same trend was observed with a slowing of convergence and the appearance of irregular second postsynaptic spikes that did not prevent the neuron from becoming tuned to its earliest regular afferents (see Figure 7).

Neurons Tune to the Earliest Spikes Through STDP

873

Figure 7: Leaky integrator and multiple postsynaptic responses. Here, a leak current (τ = 20 ms) is added to the neuron potential and more than one postsynaptic spike can be elicited in the course of the presentation, triggering STDP repeatedly. Typical incoming activity is of the same kind as in the main experiment: a reproducible structure of spike trains amid 5 ms jitter and 5 Hz spontaneous activity (see Figure 2A). It is presented to the output neuron after setting its potential to 0. Prior to presentation, the presynaptic neurons do not fire any spike. The black line links the first postsynaptic response latency at each presentation. Times at which the efferent neuron has subsequent discharges are displayed by empty circles. Except for some additional postsynaptic responses that did not affect the dynamics of learning, convergence on the earliest afferents was reached, slightly more slowly than in the comparative case without the use of a leaky integrator (see Figure 2B, bottom). Due to the leakage current, synapses had to be initialized at a slightly higher value than in previous simulations. The sum of synaptic weights thus started from a higher value than before, above the threshold of the postsynaptic neuron’s threshold (see Figure 2B, top). It nonetheless acted the same way to stabilize at threshold value.

One could also address the fact that theoretical studies use the idealized “smooth” STDP curve, while experimental data always display some noise: the curve is also likely to be noisy in biological settings (Bi & Poo, 1998). We thus tested this feature in a specific simulation under the same conditions as in section 3.1, except that each single synaptic modification was affected by a random offset taken from a gaussian distribution of mean 0 and with a standard deviation set to the noiseless amount of modification (see Figure 8A). Even when noise affects the synaptic modifications in a biologically realistic way, the trend emerges in a very similar manner (see Figure 8B).

874

R. Guyonneau, R. Van Rullen, and S. Thorpe

Figure 8: Noisy synaptic modifications. This simulation is identical to the main one (see Figure 2) except that in this case, each synaptic modification is affected by an offset taken from a gaussian distribution of mean 0 and standard deviation set to the noiseless amount of modification. Incoming activity is of the same kind as in the main experiment: a reproducible structure of spike trains amid 5 ms jitter and 5 Hz spontaneous activity (see Figure 2A). It is presented to the output neuron after setting its potential to 0. Prior to presentation, the presynaptic neurons do not fire any spike. (A) Typical sample of synaptic modifications. Taken from the modifications occurring at step 1 (only one out of two modifications are represented for clarity), this scatter plot shows the amount of each synaptic modification depending on the relative timing of the corresponding spike compared to the postsynaptic spike timing (dashed line). Notice how some expected rewards are in fact depressions and vice versa, as in the experimentally obtained STDP graphs (Bi & Poo, 1998). (B) Dynamics of repeated STDP. The trend emerges quite the same way as in Figure 2, illustrating the fact that STDP is able to reach for the first spikes of a reproducible input pattern, even when synaptic modifications are randomized.

Evidently, the demonstration here depends on the learning rule used in these simulations, and in particular on its temporally asymmetrical shape. This form was observed in several empirical studies (Markram et al., 1997; Bi & Poo, 1998, 2001; Zhang et al., 1998; Feldman, 2000; and Sjostr ¨ om ¨ & Nelson, 2002, for reviews) and has triggered numerous theoretical studies of STDP (Kempter, Gerstner, & van Hemmen, 1999; Song, Miller, & Abbott, 2000; van Rossum, Bi, & Turrigiano, 2000; Gerstner & Kistler, 2002b; see Abbott & Nelson, 2000, and Kepecs, Van Rossum, Song & Tegner, 2002, for reviews). Other forms exist, for example, symmetric (Egger, Feldmeyer, & Sakmann, 1999) or opposed (Bell, Han, Sugawara, & Grant, 1997), that could give rise to radically different trends. In the latter case, for example, where a pre-before-post pairing would induce depression and post-before-pre

Neurons Tune to the Earliest Spikes Through STDP

875

reinforcement, inhibiting the last spikes to make a cell fire might in fact increase the postsynaptic latency from one presentation to the other. A critical question concerns the handling of spike timing-dependent modifications when the same synapse receives more than one input spike that falls in the STDP window. Here we simply assumed that the effects of spike pairs would sum linearly, as in many theoretical studies (Gerstner, Kempter, van Hemmen, & Wagner, 1996; Song, Miller, & Abbott, 2000; Senn, Markram, & Tsodyks, 2001), but the question extends to the problem of synaptic modifications in natural spike trains. A recent experimental study showed that the efficacy of each spike in modifiying synaptic strength was suppressed by the preceding spike in the same neuron: STDP favors the earliest of two spike pairs (Froemke & Dan, 2002), that is, in our case, reinforcement over depression. Besides, in a spike sequence following a period of inactivity, each spike decreases the probability of vesicle release for the following presynaptic action potentials, reducing the amplitude of their respective EPSPs in the process (Thomson & Deuchars, 1994). This form of short-term plasticity, synaptic depression, acts at the presynaptic level through the depletion of the pool of vesicles at the synaptic release site, where recycling is a relatively slow process compared to sustained activity. Not only does it by itself suggest a relative importance of first spikes as opposed to later ones, but it also interacts with long-term potentiation in a supportive way called synaptic redistribution (Markram & Tsodyks, 1996). While increasing, or decreasing, the probability of transmitter release, STDP would at the same time decrease (resp. increase) the availability of readily releasable vesicles for later spikes: when they get stronger, synapses depress more and vice versa (Markram & Tsodyks, 1996; Volgushev, Voronin, Chistiakova, & Singer, 1997). As such, “synaptic redistribution can significantly enhance the amplitude of synaptic transmission for the first spikes in a sequence,” thus giving a predominant role to the first spikes in terms of synaptic plasticity (Abbott & Nelson, 2000). Applied to the present framework, these more realistic simulations of synaptic modifications would no doubt reinforce the “back-in-time” dynamics of STDP, as illustrated by latency reduction, a necessary consequence of its very form. 5 Conclusion The most important implications of our work concern the way the brain stores and uses information. Much work in neural coding assumes that information is encoded using either firing rate or synchrony. This study raises doubts about this assumption by demonstrating the unambigous prevalence of the earliest firing inputs. This sensitivity to spatiotemporal spike patterns, obviously inherent in STDP rules, emphasizes the importance of temporal structure in neural information, as proposed and argued by several authors (Perkel & Bullock, 1968; Bialek, Rieke, de Ruyter van Steveninck, & Warland, 1991; Engel, Konig, Kreiter, Schillen, & Singer, 1992; de Ruyter

876

R. Guyonneau, R. Van Rullen, and S. Thorpe

van Steveninck, Lewen, Strong, Koberle, & Bialek, 1997; Bair, 1999; Panzeri, Petersen, Schultz, Lebedev, & Diamond, 2001; Van Rullen & Thorpe, 2001). As stated by Froemke and Dan (2002), “Timing of the first spike in each burst is dominant in synaptic modifications.” Latencies do indeed seem to play a distinct role in terms of neural processing, as shown throughout this study. We showed how spatially distributed synapses can be made to integrate a spike wave coming from an afferent population and evoke a fast and selective response in the postsynaptic neuron, even in the presence of background noise. As a consequence, the fact that STDP naturally leads a neuron to respond rapidly and selectively on the basis of the first few spikes in its afferents lends support for the idea that even complex visual recognition tasks can be performed on the basis of a single wave of spikes (Van Rullen & Thorpe, 2002; Thorpe & Imbert, 1989). Acknowledgments This research was supported by the CNRS, the ACI Neurosciences Computationelles et Int´egratives, and SpikeNet Technology SARL. We also thank our reviewers for their critical comments. References Abbott, L. F., & Nelson, S. B. (2000). Synaptic plasticity: Taming the beast. Nat. Neurosci. 3, (Suppl.), 1178–1183. Abeles, M. (1991). Corticonics: Neural circuits of the cerebral cortex. Cambridge: Cambridge University Press. Bair, W. (1999). Spike timing in the mammalian visual system. Curr. Opin. Neurobiol., 9, 447–453. Bair, W., & Koch, C. (1996). Temporal precision of spike trains in extrastriate cortex of the behaving macaque monkey. Neural Comput., 8, 1185–1202. Bell, C. C., Han, V. Z., Sugawara, Y., & Grant, K. (1997). Synaptic plasticity in a cerebellum-like structure depends on temporal order. Nature, 387, 278–281. Bi, G. Q., & Poo, M. M. (1998). Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci., 18, 10464–10472. Bi, G. Q., & Poo, M. M. (2001). Synaptic modification by correlated activity: Hebb’s postulate revisited. Annu. Rev. Neurosci., 24, 139–166. Bialek, W., Rieke, F., de Ruyter van Steveninck, R. R., & Warland, D. (1991). Reading a neural code. Science, 252, 1854–1857. Cateau, H., & Fukai, T. (2003). A stochastic method to predict the consequence of arbitrary forms of spike-timing-dependent plasticity. Neural. Comput, 15, 597–620. de Ruyter van Steveninck, R. R., Lewen, G. D., Strong, S. P., Koberle, R., & Bialek, W. (1997). Reproducibility and variability in neural spike trains. Science, 275, 1805– 1808.

Neurons Tune to the Earliest Spikes Through STDP

877

Egger, V., Feldmeyer, D., & Sakmann, B. (1999). Coincidence detection and changes of synaptic efficacy in spiny stellate neurons in rat barrel cortex. Nat. Neurosci., 2, 1098–1105. Engel, A. K., Konig, P., Kreiter, A. K., Schillen, T. B., & Singer, W. (1992). Temporal coding in the visual cortex: New vistas on integration in the nervous system. TINS, 15, 218-226. Evarts, E. V. (1964). Temporal Patterns of Discharge of Pyramidal Tract Neurons During Sleep and Waking in the Monkey. J. Neurophysiol., 27, 152–171. Feldman, D. E. (2000). Timing-based LTP and LTD at vertical inputs to layer II/III pyramidal cells in rat barrel cortex. Neuron, 27, 45–56. Froemke, R. C., & Dan, Y. (2002). Spike-timing-dependent synaptic modification induced by natural spike trains. Nature, 416, 433–438. Gautrais, J., & Thorpe, S. (1998). Rate coding versus temporal order coding: A theoretical approach. Biosystems, 48, 57–65. Gerstner, W., Kempter, R., van Hemmen, J. L., & Wagner, H. (1996). A neuronal learning rule for sub-millisecond temporal coding. Nature, 383, 76–81. Gerstner, W., & Kistler, W. M. (2002a). Spiking neuron models. Cambridge: Cambridge University Press. Gerstner, W., & Kistler, W. M. (2002b). Mathematical formulations of Hebbian learning. Biol. Cybern., 87, 404–415. Gerstner, W., Kreiter, A. K., Markram, H., & Herz, A. V. (1997). Neural codes: Firing rates and beyond. Proc. Natl. Acad. Sci. USA, 94, 12740–12741. Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New-York: Wiley. Heil, P. (1997). Auditory cortical onset responses revisited. I. First-spike timing. J. Neurophysiol., 77, 2616–2641. Hopfield, J. J. (1995). Pattern recognition computation using action potential timing for stimulus representation. Nature, 376, 33–36. Hubel, D. H. (1959). Single unit activity in striate cortex of unrestrained cats. J. Physiol., 147, 226–238. Kempter, R., Gerstner, W., & van Hemmen, J. L. (1999). Hebbian learning and spiking neurons. Phys. Rev. E, 59, 4498–4514. Kepecs, A., Van Rossum, M. C., Song, S., & Tegner, J. (2002). Spike-timing-dependent plasticity: Common themes and divergent vistas. Biol. Cybern., 87, 446–458. Liu, R. C., Tzonev, S., Rebrik, S., & Miller, K. D. (2001). Variability and information in a neural code of the cat lateral geniculate nucleus. J. Neurophysiol., 86, 2789–2806. Mainen, Z. F., & Sejnowski, T. J. (1995). Reliability of spike timing in neocortical neurons. Science, 268, 1503–1506. Markram, H., Lubke, J., Frotscher, M., & Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science, 275, 213– 215. Markram, H., & Tsodyks, M. (1996). Redistribution of synaptic efficacy between neocortical neurons. Nature, 382, 807–810. Meister, M., & Berry, M. J., II. (1999). The neural code of the retina. Neuron, 22 (3), 435–450. Nakamura, K. (1998). Neural processing in the subsecond time range in the temporal cortex. Neural Comput., 10, 567–595.

878

R. Guyonneau, R. Van Rullen, and S. Thorpe

Panzeri, S., Petersen, R. S., Schultz, S. R., Lebedev, M., & Diamond, M. E. (2001). The role of spike timing in the coding of stimulus location in rat somatosensory cortex. Neuron, 29, 769–777. Perkel, D. H., & Bullock, P. H. (1968). Neural coding. Neurosci. Res. Prog. Sum., 3, 405–527. Petersen, R. S., Panzeri, S., & Diamond, M. E. (2001). Population coding of stimulus location in rat somatosensory cortex. Neuron, 32, 503–514. Rao, R. P. N., & Sejnowski, T. J. (2000). Predictive sequence learning in recurrent neocortical circuits. In S. A. Solla, T. K. Leen, & K.-R. Muller ¨ (Eds.), In Advances in neural information processing systems, 12. (pp 164–170).Cambridge, MA: MIT Press. Richmond, B. J., & Optican, L. M. (1990). Temporal encoding of two-dimensional patterns by single units in primate primary visual cortex. II. Information transmission. J. Neurophysiol., 64, 370–380. Rubin, J., Lee, D. D., & Sompolinsky, H. (2001). Equilibrium properties of temporally asymmetric Hebbian plasticity. Phys. Rev. Lett., 86, 364–367. Senn, W., Markram, H., & Tsodyks, M. (2001). An algorithm for modifying neurotransmitter release probability based on pre- and postsynaptic spike timing. Neural Comput., 13, 35–67. Sestokas, A. K., Lehmkuhle, S., & Kratz, K. E. (1991). Relationship between response latency and amplitude for ganglion and geniculate X- and Y-cells in the cat. Int. J. Neurosci., 60, 59–64. Shadlen, M. N., & Newsome, W. T. (1994). Noise, neural codes and cortical organization. Curr. Opin. Neurobiol., 4, 569–579. Shadlen, M. N., & Newsome, W. T. (1998). The variable discharge of cortical neurons: Implications for connectivity, computation, and information coding. J. Neurosci., 18, 3870–3896. Sjostrom, P. J., & Nelson, S. B. (2002). Spike timing, calcium signals and synaptic plasticity. Curr. Opin. Neurobiol., 12, 305–314. Song, S., Miller, K. D., & Abbott, L. F. (2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nat. Neurosci., 3, 919–926. Steriade, M., Oakson, G., & Kitsikis, A. (1978). Firing rates and patterns of output and nonoutput cells in cortical areas 5 and 7 of cat during the sleep-waking cycle. Exp. Neurol., 60, 443–468. Thomson, A. M., & Deuchars, J. (1994). Temporal and spatial properties of local circuits in neocortex. Trends Neurosci., 17, 119-126. Thorpe, S. J., & Imbert, M. (1989). Biological constraints on connectionist models. In R. Pfeifer, Z. Schreter, & F. Fogelman-Soukli´e (Eds.), Connectionism in perspective (pp. 63–92). Amsterdam: Elsevier. Tsodyks, M. (2002). Spike-timing-dependent synaptic plasticity—the long road towards understanding neuronal mechanisms of learning and memory. Trends Neurosci., 25, 599–600. van Rossum, M. C., Bi, G. Q., & Turrigiano, G. G. (2000). Stable Hebbian learning from spike timing-dependent plasticity. J. Neurosci., 20, 8812–8821. VanRullen, R., & Thorpe, S. J. (2001). Rate coding versus temporal order coding: What the retinal ganglion cells tell the visual cortex. Neural Comput., 13, 1255–1283. VanRullen, R., & Thorpe, S. J. (2002). Surfing a spike wave down the ventral stream. Vision Res., 42, 2593–2615.

Neurons Tune to the Earliest Spikes Through STDP

879

Victor, J. D., & Purpura, K. P. (1996). Nature and precision of temporal coding in visual cortex: A metric-space analysis. J. Neurophysiol., 76, 1310–1326. Volgushev, M., Voronin, L. L., Chistiakova, M., & Singer, W. (1997). Relations between long-term synaptic modifications and paired-pulse interactions in the rat neocortex. Eur. J. Neurosci., 9, 1656–1665. Zhang, L. I., Tao, H. W., Holt, C. E., Harris, W. A., and Poo, M. (1998). A critical window for cooperation and competition among developing retinotectal synapses. Nature, 395 37–44.

Received January 30, 2004; accepted August 25, 2004.

LETTER

Communicated by Alex Reyes

Rate and Synchrony in Feedforward Networks of Coincidence Detectors: Analytical Solution Shawn Mikula [email protected]

Ernst Niebur [email protected] Zanvyl Krieger Mind/Brain Institute and Department of Neuroscience, Johns Hopkins University, Baltimore, MD 21218, U.S.A.

We provide an analytical recurrent solution for the firing rates and crosscorrelations of feedforward networks with arbitrary connectivity, excitatory or inhibitory, in response to steady-state spiking input to all neurons in the first network layer. Connections can go between any two layers as long as no loops are produced. Mean firing rates and pairwise crosscorrelations of all input neurons can be chosen individually. We apply this method to study the propagation of rate and synchrony information through sample networks to address the current debate regarding the efficacy of rate codes versus temporal codes. Our results from applying the network solution to several examples support the following conclusions: (1) differential propagation efficacy of rate and synchrony to higher layers of a feedforward network is dependent on both network and input parameters, and (2) previous modeling and simulation studies exclusively supporting either rate or temporal coding must be reconsidered within the limited range of network and input parameters used. Our exact, analytical solution for feedforward networks of coincidence detectors should prove useful for further elucidating the efficacy and differential roles of rate and temporal codes in terms of different network and input parameter ranges. 1 Introduction Much of the theoretical basis for understanding information processing in complex biological systems is based on computational modeling—numerical solutions of the underlying model equations. While this approach has proven extremely useful and is the only practical one in many cases, analytical solutions would nearly always be preferable if they were available. Naturally, this is the case only for carefully selected systems of sufficient simplicity. In this report, we present one such system: an interconnected network of ideal coincidence detectors of arbitrary depth. Our solution is limited to feedforward networks, but otherwise the connectivity is arbitrary: Neural Computation 17, 881–902 (2005)

© 2005 Massachusetts Institute of Technology

882

S. Mikula and E. Niebur

neuronal thresholds can be arbitrary, synapses can be of arbitrary strengths, they can be excitatory and inhibitory, and they can be between any two neurons in any two layers provided that no loops are formed (this is the feedforward condition). The input to the network is characterized in terms of the average firing rate and the pairwise correlation, and we present iterative, closed-form solutions for all neurons (and pairs) in the network in the same form. The model is based on our previous exact, analytical solutions for the output firing rate of an individual coincidence detector receiving excitatory and inhibitory inputs, in both the presence and absence of synaptic depression (Mikula & Niebur, 2003a, 2003b, 2004) After defining methods and notations in section 2, we present our main result, the closed-form expressions for mean rate and cross-correlation, in section 3. Example networks are studied in sections 4 and 5. Implications for neural coding are discussed in section 6. 2 Methods 2.1 Notational Conventions. Matrix notation is used throughout our derivation. Matrices are denoted by boldfaced, uppercase letters; row and column vectors by boldface, lowercase letters; and scalars by regular lowercase letters. Matrices and vectors that are functions of one or more variables have their arguments denoted by subscripts, whereas scalars have their ar
j denotes a matrix that is guments within parentheses. Thus, for instance,
a function of one argument, j; the vector c
i, j is a function of two arguments, i and j; and q (i, j; s, t) denotes a scalar that is a function of four arguments, i, j, s, and t. 2.2 Model Neurons: Coincidence Detectors. The model neurons used in this study are coincidence detectors. A coincidence detector is a computational unit that fires at time t if the number of unit excitatory postsynaptic potentials (EPSPs) received within the window (t − T, t) equals or exceeds the threshold θ. In many cases, it makes sense to think of T as a period on the order of 10 ms. This is the timescale of fast ionic synaptic conductances, and it is at this timescale that synaptic events superpose and interact. We do not, however, make use of this specific setting in our analysis, other than requiring that it be sufficiently small so that a maximum of one spike can be generated in a period of this length, and our results do not depend on this setting. 2.3 Binomial Spike Trains with Specific Cross-Correlation. In a previous report Mikula & Niebur, 2003a), we introduced a systematic method for the generation of an arbitrary number of spike trains with specified pair-wise mean cross-correlations and firing rates. Action potentials are distributed according to binomial counting statistics in each spike train.

Rate and Synchrony in Feedforward Networks

883

Mean firing rates and cross-correlations are the same for all spike trains (or all pairs of spike trains, respectively), but they can be chosen independently of each other. We describe the procedure here for the convenience of the reader. Let m be the number of input spike trains, each having n time bins. All bins are of equal length δt, chosen sufficiently small so that each contains a maximum of one spike; that is, each bin is guaranteed to contain either one or zero spikes. The makes the decision whether to fire within one time bin; therefore, T = δt. Assuming a firing rate of f =

p , δt

(2.1)

the probabilitythat a spike is found in any given time bin is p; no spike is found with probability(1 − p). Bins in any given spike train are independent, which implies that the following analysis can be limited to a single time bin. A physiologically important special case is obtained if the rate of incoming spikes is low and convergence is high; the binomial statistics that governs the spikes generated by a coincidence detector is then approximated by Poisson statistics. We further note that throughout this letter, we often refer to the spike probability, p, simply as an input or output firing rate, with the understanding that the actual firing rate is obtained by dividing p by the time bin size, δt, as in equation 2.1. 2.4 Network Architecture. An example feedforward network is shown in Figure 1. To characterize the connectivity between layers in the network,
j , which quantifies the let us introduce the connectivity matrix, denoted C connectivity from the j layer to the j + 1th layer, and in which the (k, l)th entry contains the numerical value of the connection to the kth neuron in
j is m j+1 x m j , where layer j + 1 from the lth neuron in layer j. The size of C m j denotes the number of neurons in layer j. The values of the connectivity matrix are real numbers—positive for excitatory connections and negative for inhibitoryconnections. We define the connectivity vector, a row vector, denoted c
i, j , to be the
j that the length of c
i, j
j . It follows from the definition of C ith row from C is m j and that the kth element of c
i, j contains a real number describing the connection from the kth neuron in layer j to the ith neuron in layer j + 1. Although the example network in Figure 1 is limited to connections between subsequent layers, that is, between layers j and ( j + 1) for all j, we note that our formalism is sufficiently general to allow connections between any two layers j and j + J with J ≥ 1, provided layer j + J exists in the network. This can be seen by first considering a network with connections only between subsequent layers (as in Figure 1), that is, with J = 1. The connections with J > 1 are then added by formally adding virtual neurons with one incoming synapse whose strength exceeds its threshold.

884

S. Mikula and E. Niebur

Figure 1: A simple n+1-layer feedforward network with cyclical boundary conditions, period 4. Coincidence detectors are represented by a circle with a () symbol in them, and connections are shown as directed arrows between coincidence detectors. Input is shown as stylized spike trains at the very bottom. Comma-separated pairs of numbers indicate layer (second number) and neuron in this layer (first number).

Finally, we denote the output firing probability(which, by equation 2.1, is directly proportional to output firing rate) for the ith neuron located in the jth layer by the scalar firing rate, p(i, j). 3 Results 3.1 Computation of Firing Rates for the Network. As discussed in section 2.3, the m0 initial inputs to our feedforward network are binomial and originate at the zeroth layer (see Figure 1). Because each of the m0 inputs is the outcome of a Bernoulli trial, the total number of possible input states is 2m0 .1 We will find it useful to enumerate these 2m0 input states and to assign corresponding probabilities for their occurrence. 1

A Bernoulli trial is defined as a single random event for which there are two and only two possible outcomes that are mutually exclusive and have a priori fixed probabilities that sum to unity. In our case, the outcome of a trial will be either zero or exactly one

Rate and Synchrony in Feedforward Networks

885


0 . Each row of this Toward this end, we define the input state matrix,
matrix represents one input state in the form of a binary row vector whose kth entry consists of a zero or a one, denoting the absence or presence of an input spike in the kth input, respectively. From the above considerations,
0 must be of size 2m0 × m0 . we know that
We now determine how the different input states are transformed at layer 1 of our network. That is, we compute how each input state affects suprathreshold events for all of the m1 neurons in layer 1. The solution is based on the known connectivities between the m0 inputs and the m1 neu
0 (see section 2.4), and on the thresholds of the neurons in rons in layer 1, C layer 1, listed in the matrix θ
1 . This matrix has 2m0 identical rows, each consisting of the vector of all thresholds in layer 1; thus, the element in column j (and all rows) of this matrix is the threshold of neuron j in layer 1 (the usefulness of this construct will become obvious in equation 3.1). These net
0 , into work dynamics transform the input states, contained in the matrix
the resulting states in layer 1 of the network. We collect these states in the
1 , which describes the states of layer 1 and is computed as follows: matrix

0 − θ
1 ).
0C
1 = (



(3.1)

By the definition of matrix products, the element in row i and column j
0 is the input to neuron j in layer 1 when state i is present in layer 0.
0C of
Furthermore, we define the operator () in this equation as the Heaviside function acting on the individual entries of its matrix argument. It takes on the value of unity if an entry within its argument is nonnegative and zero
1 are 2m0 × m1 and that the otherwise.2 Note that the dimensions of

1 is the layer 1 state corresponding to the kth row of

0 . That is, kth row of
the transformation of the kth input state at layer 1 is given by the kth row of
1 . To be sure, while

0 is an exhaustive list of all states in the input layer,


1 is, in general, not a matrix containing all states in layer 1. Depending on m0 , m1 and the connectivity and thresholds, this matrix in general contains only a subset of all possible states in layer 1, and each of these states may appear more than once. We can recursively apply the same method used in equation 3.1 to generate the state matrices for other layers of our network to arrive at the general
j , the layer j state matrix, as the following: expression for

j−1 − θ
j ),
j−1 C
j = (



(3.2)

spike of an input neuron in a given time bin; the former happens with probability p where 0 ≤ p ≤ 1 and the latter with probability (1 − p). 2 We note that this component-wise procedure is not an operation defined in a vector space. We nevertheless use the terminology of vector spaces (rather than introducing terms like arrays or n-tuples) for convenience.

886

S. Mikula and E. Niebur


j gives us the layer j state corresponding to the kth where the kth row of

j−1 and terminates
0 . The iteration defines

j in terms of

j−1 and C row of

0 , as in equation 3.1.
0 and C at

j , which is of length 2m0 , gives us the neuronal reThe ith column of
sponses of the ith neuron in layer j to the 2m0 input states. Let this column
i, j , the state vector for the ith neuron in layer j; then vector be denoted ψ the following holds:
i, j = (

j−1 c
i, j−1 − θ(i, j)), ψ

(3.3)

where the scalar θ (i, j) is the threshold of neuron i in layer j. Note that the only difference between the right-hand sides of equations 3.3 and 3.2
j−1 and the threshold is that the latter substitutes the connectivity matrix C matrix θ
j for the connectivity vector c
i, j−1 and the threshold scalar θ (i, j), respectively. Having determined how input states are transformed across the network by equations 3.1 and 3.2, we now proceed to compute mean output spiking probabilities. We define the input state probability vector
pin , a column vector whose kth element is the probability for the occurrence of the kth input state. The output spiking probability for any neuron in the network is obtained by summing over all input states, leading to suprathreshold events times the corresponding input state probability (given by
pin ). Thus, the output spiking probabilities for all neurons located in layer j can then be written as a column vector
p j , computed as
Tj
pin ,
p j =


(3.4)

where the ith entry of
p j is the output spiking probability for the ith neuron
j.
Tj is the transpose of
located in layer j and where
Alternatively, by using equation 3.3, we can obtain an explicit expression for the output spiking probability of a single neuron located anywhere in the network. The output spiking probability for the ith neuron located in layer j, the scalar p(i, j), is given by
i,T j
pin . p(i, j) = ψ

(3.5)

Note that p(i, j) is the same as the ith entry of
p j from equation 3.4. With equations 3.4 and 3.5, we have thus obtained an analytical recurrent solution for the output spiking probabilities (or firing ratesas per equation 2.1) of the coincidence detectors comprising a feedforward network that receives an arbitrary number of binomial inputs modulated with respect to both mean rate and cross-correlation.

Rate and Synchrony in Feedforward Networks

887

3.2 Computation of Cross-Correlations. The standard Pearson correlation coefficient, denoted q (x, y), is defined by the following for random variables x and y, xy − x y
q (x, y) =
, var(x) var(y)

(3.6)

where x denotes the mean value of x, and var(x) denotes the variance of x. If x is a Bernoulli random variable (i.e., takes on values 0 or 1), then its variance is given by var(x) = x − x2 ,

(3.7)

and we may rewrite equation 3.6 as the following (Mikula & Niebur, 2003b): q (x, y) =

   prob(xy = 1) − prob(x = 1) prob(y = 1)    2  2   prob(x = 1)− prob(x = 1) prob(y = 1)− prob(y = 1) (3.8)

where prob(x=1) denotes the probability that x takes the value of unity, and prob(xy=1) denotes the probability that both x and y take the value of unity. In the notation established earlier,the spiking probability scalar p(i, j) of neuron j in layer i takes the place of prob(x=1) in equation 3.8, taking into account that it is based on a Bernoulli process. In this equation, to compute the correlation with neuron t in layer s, the probability p(s, t) takes the place of prob(y=1). And finally, since the joint probability for the ith neuron in layer j and the sth neuron in layer t to fire action potentials is given
+ψ
s,t − 2)T
pin , as can be seen by direct substitution, it follows by (ψ  i, j
s,t − 2)T
pin . We can therefore
i, j + ψ that prob(xy=1) is equivalent to (ψ rewrite equation 3.8 as the following expression for the cross-correlation, denoted q (i, j; s, t), between the ith neuron in layer j and the sth neuron in layer t,
i, j + ψ
s,t − 2)T
pin − p(i, j) p(s, t) (ψ
q (i, j; s, t) =
, p(i, j) − p(i, j)2 p(s, t) − p(s, t)2

(3.9)


i, j is obtained using equation 3.3. where ψ We have thus obtained an exact recurrent solution for the cross-correlation between any two coincidence detectors, or between any coincidence detector and an input, located within a feedforward network receiving an arbitrary number of binomial inputs modulated with respect to both mean rate and cross-correlation.

888

S. Mikula and E. Niebur

3.3 Large Networks. On a practical note, we find that it is useful to
0 for the purpose of making consider methods for reducing the size of
our solutions for rate and cross-correlation applicable to larger feedforward
0 is 2m0 × m0 , networks. As we saw in section 3.1, for m0 inputs, the size of
which increases quickly for increasing m0 . We assume that the connectivity of the network is given. Under this condition, we have discerned three
0 . One of them is to include practical methods for reducing the size of
only suprathreshold events, which means excluding those input states in which a subthreshold number of spikes occurs; all of these are mapped on a single state with a zero everywhere. Assuming uniform thresholds for
0 to all neurons,which we will denote by θ, then this method reduces
−1 m0  ) × m , which can be a significant reduction as θ/m size (2m0 − θj=0 0 0 j approaches unity. The mirror image of this simplification is obtained in the case of small θ/m0 ; in this case, a large number of input states is mapped onto one state, namely, “one” in all positions. The second method for reducing
0 is valid for low input rates (i.e., the Poisson regime), in which the size of
case the input states characterized by many spikes can be ignored due to the unlikely probability of such a state. The third method is to reduce the number of inputs while maintaining a relatively large number of neurons in each layer. In many cases, this is justified since for large convergence (the physiologically relevant case), only a small number of neurons can be simultaneously active (within one time bin of length T) to avoid the need for unrealistically high thresholds or saturation of the next layer. 4 Simple Example Let us consider the simple n-layer excitatory feedforward network shown in Figure1 with four neurons in each layer, cyclical (periodic) boundary connectivity conditions, and θ(i, j) = θ = 2 ∀i, j. We further assume that each connection is excitatory and has a weight of +1. The connectivity patterns between layers are the same for all layers; that is, each neuron in layer j connects to the three closest neurons in layer j + 1. Recalling from section 2.4 that the (k, l)th entry of the connectivity matrix is defined as the weight of the connection to the kth neuron in layer j + 1 from the lth neuron in layer j, we obtain a connectivity matrix that is the same for each layer and given by the following:  1 1 
C( j) =  0 1

1 1 1 0

0 1 1 1

 1 0 . 1 1

(4.1)

For computing the total number of input states, we note that there are four binomial inputs, and thus 24 input states. The resulting input matrix
0 is


Rate and Synchrony in Feedforward Networks



0 0  0  0  0  0  0  0

0 =  1  1  1  1  1  1  1 1

0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1

0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1

889

 0 1  0  1  0  1  0  1 . 0  1  0  1  0  1  0 1

(4.2)


0 is the input state for having zero input spikes Thus, the first row of
for all four inputs, the second row is the input state for having zero input spikes for inputs 1 through 3, whereas input 4 has a spike, and so on for all
0. of the 16 rows of
Using the network connectivity and the input state matrices, we can now
j and ψ
i, j . The result is given in apply either equation 3.2 or 3.3 to derive
Table 1, which shows all attainable states for all neurons in layers 1 through 3
0,

1,

2 , and of our network (plus the input layer, 0). We have placed


3 alongside each other in Table 1 to emphasize that this is a logical truth Table 1: Truth Table for the First Three Layers of the N-Layer Feedforward Network Shown in Figure 1.
3



2



1



0



13 ψ
23 ψ
33 ψ
43 ψ
12 ψ
22 ψ
32 ψ
42 ψ
11 ψ
21 ψ
31 ψ
41 ψ
10 ψ
20 ψ
30 ψ
40 ψ 0 0 0 0 0 1 0 1 0 1 0 1 1 1 1 1

0 0 0 0 0 0 1 1 0 0 1 1 1 1 1 1

0 0 0 1 0 1 1 1 0 0 0 1 0 1 1 1

0 0 0 1 0 0 0 1 0 1 1 1 0 1 1 1

0 0 0 0 0 0 0 1 0 1 1 1 1 1 1 1

0 0 0 0 0 1 1 1 0 0 0 1 1 1 1 1

0 0 0 1 0 0 1 1 0 0 1 1 0 1 1 1

0 0 0 1 0 1 0 1 0 1 0 1 0 1 1 1

0 0 0 0 0 1 0 1 0 1 0 1 1 1 1 1

0 0 0 0 0 0 1 1 0 0 1 1 1 1 1 1

0 0 0 1 0 1 1 1 0 0 0 1 0 1 1 1

0 0 0 1 0 0 0 1 0 1 1 1 0 1 1 1

0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1

0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1

0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1

0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1

890

S. Mikula and E. Niebur

table. This should also make it easier to see how different input states, which
0 , are transformed across different layers, which correspond to the rows of

1,

2 , and

3. correspond to the rows of
It turns out that Table 1 characterizes the behavior not only of the first three but of all layers of the (infinite) network. The reason is that all higher
2 , and similarly, all higher odd layers are even layers are identical to

1 . This periodic behavior is not a peculiarity of this specific identical to
network, is a property of all feedforward networks of coincidence detectors with identical layer properties (connectivity and thresholds) and finite numbers of neurons per layer; since they can have only a finite number of states, networks of infinite depth need to show periodic behavior. This can be seen easily by considering that there are 2m0 possible states in the input layer, layer 0 (see section 3.1), and in all higher layers. Since all layers are assumed to have the same number of neurons, at least two layers between the input layer and layer (2m0 + 1) must have identical activity patterns. Let us assume the first one of these is layer l and the second one is layer l  . Then, since the connectivities between all layers are identical, the pattern in layer (l  + 1) is the same as in (l + 1), the pattern in (l  + 2) is as in (l + 2), and so on; activity is thus periodic with a periodicity (l  − l) starting at layer l. This is what was to be shown. We note in passing that periodicity includes the case of period 1, for example, when the activity “dies out” in layer l because all activity is below threshold and therefore all higher layers have zero activity. The next step is the computation of column vector
pin , whose ith entry
0 ). We contains the probability for the ith input state (i.e., the ith row of
will first assume uncorrelated inputs (correlated input is discussed later; see equation 4.6 for an example). In this case, the components of
pin are simple binomial probabilities. We have the following for
pin :  (1 − P10 )(1 − P20 )(1 − P30 )(1 − P40 )  (1 − P10 )(1 − P20 )(1 − P30 )P40     (1 − P10 )(1 − P20 )P30 (1 − P40 )      (1 − P10 )(1 − P20 )P30 P40    (1 − P10 )P20 (1 − P30 )(1 − P40 )      (1 − P10 )P20 (1 − P30 )P40     (1 − P10 )P20 P30 (1 − P40 )     (1 − P )P P P 10 20 30 40 . 
pin =   P (1 − P )(1 − P )(1 − P ) 10 20 30 40     P (1 − P )(1 − P )P 10 20 30 40     P (1 − P )P (1 − P ) 10 20 30 40     P10 (1 − P20 )P30 P40     P10 P20 (1 − P30 )(1 − P40 )     P10 P20 (1 − P30 )P40     P10 P20 P30 (1 − P40 ) P10 P20 P30 P40 

(4.3)

Rate and Synchrony in Feedforward Networks

891

Figure 2: Effect of uniform, uncorrelated input firing rate. (A) Mean firing rates for the network. (B) Nearest-neighbor cross-correlations for the network. (C) Cross-correlation matrices for each layer.

To alleviate the notation, we replaced p(i, 0) by Pi0 for i ∈ {1, . . , 4}, that is, the mean firing probabilities for the ith input from the zeroth layer. The values of Pi0 can be freely chosen depending on the question under study (subject to the conditions 0 ≤ Pi0 ≤ 1 since they are probabilities). For example, we may choose uniform inputs, in which case all Pi0 would be equal. Alternatively, we may study how spatially localized high rate inputs are dissipated in higher levels of the network. In this case, we set some of the Pi0 to high values relative to the remaining Pi0 . Both cases are illustrated in the examples. The results for the case of uncorrelated inputs of uniform rate are shown in Figure 2. Figure 2A shows the mean rates for the input, even, and odd layers. Note that rates get propagated to arbitrarily high levels of the feedforward network. Besides the initial attenuation of rates at layer 1, there is no further attenuation. Figure 2B shows nearest-neighbor cross-correlations for the network, and Figure 2C shows cross-correlation matrices for each layer. As expected, because we have uncorrelated inputs of uniform rate, nothing much interesting happens with the cross-correlation at higher layers, except for a uniform increase in all cross-correlations, which is expected due to shared connections. What is more interesting is that crosscorrelation is not monotonically increasing as a function of network layer; rather, it remains constant from layer 1 to all higher layers. Figure 3 shows how rates and cross-correlations in higher layers vary as a function of input rates. In Figure 3A, we see that rates in higher layers are approximately linearly related to input rates, whereas in Figure 3B, we see that cross-correlations in higher layers depend on input rates in an inverted-U shaped manner, with maximum cross-correlations occurring for input rates of 0.5.

892

S. Mikula and E. Niebur

Figure 3: (A) Higher-layer output rate versus input rate curve. (B) higher-layer cross-correlation versus input rate curve for the case of uncorrelated inputs of uniform rate.

We now briefly consider the case of spatially inhomogeneous inputs. As an example, let the mean rates of the second and third inputs into our feedforward network, P20 and P30 or, in the notation of equation 3.5, p(2, 0) and p(3, 0), be equal, that is, p23 = P20 = P30 , where this equation defines p23 . Let us further assume that these inputs are correlated with a correlation coefficient q 23 , with 0 < q 23 ≤ 1 (the case q = 0 is the case of spatially inhomogeneous, uncorrelated input, which is of less interest here). To simplify upcoming expressions, we make the following definitions: √ q 23

(4.4)

q = (1 − q ).

(4.5)

q=

Rate and Synchrony in Feedforward Networks

893

Figure 4: Effect of spatially localized high input rates. (A) Mean firing rates for the network. (B) nearest-neighbor cross-correlations for the network. (C) Cross-correlation matrices for each layer.

The equivalent of equation 4.3 for
pin is now (see the appendix to Mikula and Niebur, 2003b, for detailed derivations) 



(1− P10 )((1− p23 )((1− p23 )+ p23 q )2 + p23 ((1− p23 )(1 − q ))2 )(1− P40 )  (1− P10 )((1− p23 )((1− p23 )+ p23 q )2 + p23 ((1− p23 )(1 − q ))2 )P40 (1− P10 )2((1− p23 ) p23 q ((1− p23 )+ p23 q )+ p23 ( p23 +(1− p23 )q )((1− p23 )q ))(1− P40 )   (1− P10 )2((1− p23 ) p23 q ((1− p23 )+ p23 q )+ p23 ( p23 +(1− p23 )q )((1− p23 )q ))P40    ) (1− P10 )2((1− p23 ) p23 q ((1− p23 )+ p23 q )+ p23 ( p23 +(1− p23 )q )((1− p23 )q ))(1− P40  (1− P10 )2((1− p23 ) p23 q ((1− p23 )+ p23 q )+ p23 ( p23 +(1− p23 )q )((1− p23 )q ))P40     (1− P10 )((1− p23 )( p23 q q )2 + p23 ( p23 +(1 − p23 )q )2 )(1− P40 )    (1− P10 )((1− p23 )( p23 q q )2 + p23 ( p23 +(1 − p23 )q )2 )P40
pin = P ((1− p )((1− p )+ p q )2 + p ((1− p )(1 − q ))2 )(1− P ) . 23 23 23 23 23 40   10   P10 ((1− p23 )((1− p23 )+ p23 q )2 + p23 ((1− p23 )(1 − q ))2 )P40    P10 2((1− p23 ) p23 q ((1− p23 )+ p23 q )+ p23 ( p23 +(1− p23 )q )((1− p23 )q ))(1− P40 )    P 2((1− p ) p q ((1− p )+ p q )+ p ( p +(1− p )q )((1− p )q ))P   10 23 23 23 23 23 23 23 23 40  P10 2((1− p23 ) p23 q ((1− p23 )+ p23 q )+ p23 ( p23 +(1− p23 )q )((1− p23 )q ))(1− P40 )      P10 2((1− p23 ) p23 q ((1− p23 )+ p23 q )+ p23 ( p23 +(1− p23 )q )((1− p23 )q ))P40   P ((1− p )( p q q )2 + p ( p +(1 − p )q )2 )(1− P ) 10 23 23 23 23 23 40 P10 ((1− p23 )( p23 q q )2 + p23 ( p23 +(1 − p23 )q )2 )P40

(4.6) The results for the case of uncorrelated, spatially localized high input rates are shown in Figure 4. Figure 4A shows the mean rates for the input, even, and odd layers. Notably, spatially localized high input rates get propagated to arbitrarily high levels of the feedforward network. Besides the initial attenuation of firing rates at layer 1, there is no further attenuation. Figure 4B shows nearest-neighbor cross-correlations for the network, and

894

S. Mikula and E. Niebur

Figure 5: Effect of halving input rates of Figure 4. (A) Mean firing rates for the network. (B) Nearest-neighbor cross-correlations for the network. (C) Crosscorrelation matrices for each layer.

Figure 4C shows cross-correlation matrices for each layer. Due to the convergence of inputs to higher layers, uncorrelated, spatially localized high input rates appear as cross-correlations in higher layers (see Figure 4B). That is, not only does rate information get propagated across the network, it also gets propagated as cross-correlation information. The case shown in Figure 5 is identical to that in Figure 4 except that now the input rates are halved. Note that qualitatively, the results shown are identical to those of Figure 4, which means that for even relatively low spatially localized input rates, with p(2, 0) = p(3, 0) = 0.2, rate information still gets propagated to arbitrarily high levels of the feedforward network. In addition, this rate information also appears as cross-correlation information in higher layers. Figure 6 shows the case for spatially localized cross-correlated inputs with uniform input rates. For this case, inputs 2 and 3 are cross-correlated to a value of q 2 = 0.4, and as can be seen in Figure 6B, this spatially localized cross-correlation gets propagated to arbitrarily high levels of the feedforward network. In Figure 6A, we see that the cross-correlation between inputs 2 and 3 is also represented in higher layers as higher rates. Note the appearance of increased cross-correlation between units 1 and 4 in layers 1 and higher in this and subsequent figures. It is due to the periodic boundary conditions and the specific connection scheme used: since input unit 2 projects to layer 1 unit 1, and input unit 3 projects to layer 1 unit 4, and units 2 and 3 are partially correlated, units 1 and 4 are also partially correlated but with a lower correlation coefficient than units 2 and 3, which both receive input from both correlated input units rather than from only one.

Rate and Synchrony in Feedforward Networks

895

Figure 6: Effect of spatially localized high cross-correlation (q = .4) between inputs 2 and 3. (A) Mean firing rates for the network. (B) Nearest-neighbor cross-correlations for the network. (C) Cross-correlation matrices for each layer.

Figure 7: Effect of spatially localized high cross-correlation (q = .8) between inputs 2 and 3. (A) Mean firing rates for the network. (B) Nearest-neighbor cross-correlations for the network. (C) Cross-correlation matrices for each layer.

The case shown in Figure 7 is identical to that in Figure 6 except that now the cross-correlation between inputs 2 and 3 is doubled to 0.8. This does not lead to much of an increase in higher-layer cross-correlations beyond what was seen in Figure 6B. Interestingly, in Figure 6A, we see that higher-layer rates are almost double what they were in Figure 6A. What this means is that doubling the input cross-correlation is reflected in an almost doubling

896

S. Mikula and E. Niebur

Figure 8: Effect of spatially localized high cross-correlation (q = .6) and high firing ratebetween inputs 2 and 3. (A) Mean firing rates for the network. (B) Nearest-neighbor cross-correlations for the network. (C) Cross-correlation matrices for each layer.

of the higher-layer rates, as opposed to the relatively smaller increase in higher-layer cross-correlations. Finally, in Figure 8, we see the case for spatially localized high crosscorrelation (q = 0.6) and high firing rate between inputs 2 and 3. Figure 8A shows that rates are propagated with little attenuation, which is in contrast to the cases in Figures 4 and 5, where the high input rates were not crosscorrelated. We see in Figure 8B that cross-correlations also get propagated across our network, though we also see relatively high cross-correlations between inputs 1 and 4 at higher layers, an effect likely attributed to the cyclic connectivity boundary conditions of our feedforward network.

5 More Complex Networks The simple example of section 4 is useful for making explicit the details involved with the application of equations 3.4, 3.5, and 3.9. In this section, we look at a larger feedforward network, which has the advantage of being more realistic than the previous one, but with the trade-off that many of the details cannot be made explicit due to space limitations. Our feedforward network is a slightly scaled-down version of the one introduced by Litvak, Sompolinsky, Segev, and Abeles (2003). We consider a 100-layer feedforward network containing exactly 1000 coincidence detectors per layer—500 excitatory and 500 inhibitory. The connectivity matrix is

Rate and Synchrony in Feedforward Networks

897

different between each layer3 and is randomly constructed with precisely balanced excitation and inhibition such that each neuron receives exactly 50 excitatory inputs and 50 inhibitory inputs. Further, pairs of neurons in each layer share 5% of their inputs from the preceding layer. Each excitatory connection carries a weight of +1, and each inhibitory connection has a weight of −1. We now define the inputs and the input states. For simplicity, we assume just two different input states. One state is defined with zero activity in all neurons of the input layer.4 In the other state, 10% of the inputs provide perfectly correlated input spike trains (i.e., q = 1), with a spiking probability within each spike train of pin . We designate these inputs as active inputs; since the total number of inputs is 1000, the number of active inputs is 100. All other input neurons (i.e., the other 90%) have zero spiking probability. Our requirement that the active inputs are perfectly correlated greatly simplifies what would otherwise be a difficult calculation. For example, if the active inputs were completely uncorrelated, then we would have 2100 different input states to deal with, a clear impossibility. Assuming perfectly correlated active inputs is a simple and effective way to dramatically reduce the number of input states. The results of this model using 100 different randomly generated connectivity matrices are shown in Figure 9 for average input spiking probabilities of .02 and .04 and using a threshold of 3. We show the results for only the first 20 layers of the network because higher layers are essentially the same as would be expected from Figure 9 (we studied up to layer 100). Spiking probabilities increase rapidly and immediately level off at about layer 2, remaining nearly constant for all higher layers. Because of the perfect correlation between all input neurons (q = 1), the mean firing rates at the higher layers of our network depend linearly on the initial input rates pin . For example, in Figure 9a, pin = .02 results in a mean output spiking probability of .055 at higher layers, whereas in Figure 9b, pin = .04 results in a mean output spiking probability of .11. Note the large standard deviations, indicating the richness and complexity of behavior for different connectivity matrices. The mean cross-correlations are propagated in a similar manner as the mean rates shown in Figure 9: since all active neurons in our network are perfectly correlated with other, the mean cross-correlation must be a function of the number of active neurons per layer, as is true for the mean rate per layer. Thus, if the fraction of active neurons in a given layer is r , the cross-correlation in that layer is r 2 . The propagating rates and 3 The connectivity used by Litvak et al. (2003) is apparently repeating (i.e., identical between each pair of subsequent layers). We studied this case as well and found results that were very similar to those with connectivity matrices that vary between each pair of layers and that are shown here. 4 For all nonzero thresholds, the same results would be achieved with all neurons active in the input layer.

898

S. Mikula and E. Niebur

Figure 9: Mean layer output rates as a function of network layer. (A) Mean input rate of 0.2. (B) mean input rate of 0.4.

Rate and Synchrony in Feedforward Networks

899

cross-correlations show no sign of decrement at higher layers (layer 100 looks the same as layer 3 in terms of rates and cross-correlations). 6 Discussion This article extends our previous analytical results (Mikula & Niebur, 2003a, 2003b, 2004) for an individual coincidence detector to a feedforward network of coincidence detectors receiving steady-state cross-correlated binomial inputs at the zeroth layer. Thus, our derivation is valid only for steady-state neuronal responses and does not tell us anything about transient responses, which also appear to be important. For example, Diesmann, Gewaltig, and Aertsen (1999) studied temporal structures of spike trains in feedforward networks of spiking neurons. They showed that in different parameter regimes, synchronous packets of spikes can either travel from layer to layer without loss of coherency or, alternatively, disperse within a few layers. The objectives of their study are complementary to ours as it focuses on transient activity while ours is concerned with constant firing rates and correlations. Likewise, the approach taken by Diesmann et al. is complementary to what we have taken insofar as they use a more complex single-neuron model, a leaky integrate-and-fire neuron rather than the simpler coincidence detector, but it requires a numerical solution while ours is analytical and recurrent and, furthermore, does not require introducing any approximations. In spite of the obvious limitations of feedforward networks, they are of considerable interest for theoretical considerations. Likewise, coincidence detectors are an old and tested but simple model for the units of neural networks. However, although computational approaches allow the use of much more realistic neuron models (an extreme case perhaps being a recent study by Reyes, 2003, in which replicas of an actual biological neuron were used as the underlying units), the use of the much simpler coincidence detectors as building blocks allowed us to develop a closed form for the mean rate and pairwise cross-correlation of each neuron and neuron pair, respectively. The details of our results apply only to the specific networks that we have studied here as examples to illustrate the general method we introduced. This method should, however, allow the analysis of other feedforward networks of coincidence detectors that are relevant for a specific problem. We emphasize that the methods described in section 3.3 permit one to find solutions for networks with much more realistic connectivity, including much higher numbers of synaptic connections. One way in which our approach may prove useful is in the elucidation of the neural code, which involves addressing the question of how biological neural networks represent and transform information in their patterns of activity (Perkel & Bullock, 1968). Most research in the primate cortex has focused on two coding schemes, rate and temporal coding. In rate

900

S. Mikula and E. Niebur

coding, information is coded purely in terms of average spiking activity, and variability of neural discharges is regarded as a form of noise. In temporal coding, neurons make use of the temporal structure of spike trains. Much experimental, computational, and theoretical work has been devoted to discussing this question, with evidence existing in favor of both rate codes and temporal codes. One of the tools for studying these competing models is the numerical simulation of feedforward networks. Unfortunately, two such recent simulation results involving biologically inspired feedforward networks yielded conflicting results, in which the group conducting one study (Litvak et al., 2003) could not reproduce previously published numerical results of another group (Shadlen & Newsome, 1998). As we have shown in section 5, our approach can make a contribution to the resolution of this dispute, bearing in mind that the basic units employed in those studies are different from the coincidence detectors we employed. The availability of an analytical form for the solution provided by our approach clearly precludes similar disputes. In a network of coincidence detectors, we find that aspects of both rate coding and temporal coding may be found. At least in the simple network studied in section 4, rate information is propagated up to arbitrarily high layers, even when spatially localized rate information is absent from the inputs and only spatially localized cross-correlation information is present. Also, in the case of uncorrelated inputs of uniform firing ratesshown in Figure 3A, we find that the output firing ratesof higher layers are almost linearly related to the input firing rates. Both would support that rate coding is at least a viable possibility, as proposed by Shadlen and Newsome (1998). The latter study may overemphasize the importance of rate coding, however: Salinas and Sejnowski (2000) pointed out that the influence of cross-correlations is relatively low in the Shadlen and Newsome study. The reason is that the latter study employed exactly equal correlation coefficients between inhibitory and excitatory populations (ρ E E = ρ I I = ρ E I in equation 20 of Salinas and Sejnowski, 2000), and the variance in the output layer becomes small since the contributions to the variance resulting from correlations between these populations cancel out (ρ E E + ρ I I − 2ρ E I = 0; Salinas & Sejnowski, 2000). Of course, the model described in section 5 works in the same parameter regime. On the other hand, we find that rate coding is not the only possibility of propagating information across the layers of the network. Our results show that cross-correlation information is propagated across the feedforward network to arbitrarily high layers, even when spatially localized cross-correlation information is absent from the inputs and only spatially localized rate information is present. Our results thus support the viability of temporal codes, as proposed by Litvak et al. (2003). For the types of networks we studied in sections 4 and 5, we conclude that both rate and cross-correlation information is propagated across

Rate and Synchrony in Feedforward Networks

901

feedforward networks and, furthermore, that they will interact and tend to occur together, such that when only rate or only cross-correlation information is present in the inputs, this information will appear in higher layers in the form of both rate and cross-correlation information. As such, rate and cross-correlation codes may well be intertwined and interdependent. An example of such a mixed code in complex nervous systems might be the representation of selective attention in the primate cortex (Niebur, 2002). Selective attention has been shown in electrophysiological studies to be correlated with both rate changes and changes in the fine temporal structure (on the order of milliseconds or tens of milliseconds) of neural activity (Moran & Desimone, 1985; Steinmetz et al., 2000; Fries, Reynolds, Rorie, & Desimone, 2001; Salinas & Sejnowski, 2001; Niebur, Hsiao, & Johnson, 2002). It will take more experimental as well as theoretical work to come to a conclusive answer as to which of the proposed neural coding schemes are used by the different nervous systems. Acknowledgments This work was supported by NIH/NINDS R01-NS43188-01A1 and NIH/ NEI R01-EY16281. References Diesmann, M., Gewaltig, M. O., & Aertsen, A. (1999). Stable propagation of synchronous spiking in cortical neural networks. Nature, 402, 529–533. Fries, P., Reynolds, J. H., Rorie, A. E., & Desimone, R. (2001). Modulation of oscillatory neuronal synchronization by selective visual attention. Science, 291, 1560–11563. Litvak, V., Sompolinsky, H., Segev, I., & Abeles, M. (2003). On the transmission of rate code in long feedforward networks with excitatory-inhibitory balance. J. Neurosci., 23, 3006–3015. Mikula, S., & Niebur, E. (2003a). The effects of input rate and synchrony on a coincidence detector: Analytical solution. Neural Computation, 15, 539–547. Mikula, S., & Niebur, E. (2003b). Synaptic depression leads to nonmonotonic frequency dependence in the coincidence detector. Neural Computation, 15(10), 2339– 2358. Mikula, S., & Niebur, E. (2004). Correlated inhibitory and excitatory inputs to the coincidence detector: Analytical solution. IEEE Transactions on Neural Networks, 15(5), 957–962. Moran, J., & Desimone, R. (1985). Selective attention gates visual processing in the extrastriate cortex. Science, 229, 782–784. Niebur, E. (2002). Electrophysiological correlates of synchronous neural activity and attention: A short review. Biosystems, 67(1–3), 157–166. Niebur, E., Hsiao, S. S., & Johnson, K. O. (2002). Synchrony: A neuronal mechanism for attentional selection? Current Opinion in Neurobiology, 12(2), 190–194. Perkel, D., & Bullock, T. H. (1968). Neural coding. Neurosci. Res. Programm. Bull., 6, 221.

902

S. Mikula and E. Niebur

Reyes, A. D. (2003). Synchrony-dependent propagation of firing rate in iteratively constructed networks in vitro. Nat. Neurosci., 6, 593–599. Salinas, E., & Sejnowski, T. J. (2000). Impact of correlated synaptic input on output firing rate and variability in simple neuronal models. J. Neurosci., 20(16), 6193– 6209. Salinas, E., & Sejnowski, T. J. (2001). Correlated neuronal activity and the flow of neural information. Nature Reviews Neuroscience, 2, 539–550. Shadlen, M. N., & Newsome, W. T. (1998). The variable discharge of cortical neurons: Implications for connectivity, computation, and information coding. J. Neurosci., 18, 3870–3896. Steinmetz, P. N., Roy, A., Fitzgerald, P., Hsiao, S. S., Johnson, K. O., & Niebur, E. (2000). Attention modulates synchronized neuronal firing in primate somatosensory cortex. Nature, 404, 187–190.

Received December 18, 2003; accepted August 10, 2004.

LETTER

Communicated by Alain Destexhe

Independent Variable Time-Step Integration of Individual Neurons for Network Simulations William W. Lytton [email protected] Department of Physiology, Pharmacology, and Neurology, State University of New York, Downstate, Brooklyn, NY 11203-2098, U.S.A.

Michael L. Hines [email protected] Department of Computer Science, Yale University, New Haven, CT 06520-8001, U.S.A.

Realistic neural networks involve the coexistence of stiff, coupled, continuous differential equations arising from the integrations of individual neurons, with the discrete events with delays used for modeling synaptic connections. We present here an integration method, the local variable time-step method (lvardt), that uses separate variable-step integrators for individual neurons in the network. Cells that are undergoing excitation tend to have small time steps, and cells that are at rest with little synaptic input tend to have large time steps. A synaptic input to a cell causes reinitialization of only that cell’s integrator without affecting the integration of other cells. We illustrated the use of lvardt on three models: a worst-case synchronizing mutual-inhibition model, a best-case synfire chain model, and a more realistic thalamocortical network model. 1 Introduction We have previously demonstrated some advantages of the global variable time-step integrators CVODE and CVODES (Cohen & Hindmarsh, 1994) over traditional fixed-step methods such as Euler, Runge-Kutta, or CrankNicholson for simulating single cells (Hines & Carnevale, 2001). The major advantage was due to the fact that neuron activity features spikes, requiring short time steps, followed by interspike intervals, which allow long time steps. The associated speed-up in the single-cell integration realm does not extend to simulation of networks, however. A major reason is that the global time step is governed by the fastest changing state variable. In an active network, some cell is usually firing, requiring a small time step for the entire network. A related reason is that synaptic events generally cause a discontinuity in a parameter or state variable. This requires a reinitialization, as the integrator must start again with a new initial condition problem. In a Neural Computation 17, 903–921 (2005)

© 2005 Massachusetts Institute of Technology

904

W. Lytton and M. Hines

network simulation, this means reinitialization of the entire network due to a single state variable change in one cell. With reinitialization, the integrator is working without any past history. Hence, the first step can be only firstorder accurate and must be very short. We demonstrate here that the poor network performance of the global variable time-step method can be overcome by giving each neuron in the system an independent variable time-step integrator. Thus, a single cell’s individual integrator uses a large dt when a neuron is quiescent or changing slowly, even though activity in other neurons in the network may cause those other integrators to proceed forward with many steps (small dt). When a cell receives a synaptic event, only that cell’s integrator has to be reinitialized. The critical problem in the implementation of the local time-step method (lvardt) is to ensure that when an event arrives at a cell at time te , all the state variables for the receiving cell are also at time te . This requires coordinating individual integrators so that one cell does not get so far ahead that it cannot receive a synaptic signal from another cell. To maximally challenge lvardt, we used the mutual inhibition model, a model that fully synchronizes. In this case, the expected superiority of multiple integrators is expected to be negated by the fact that all integrators are doing the same thing at the same time. At the other extreme, we show that synfire chains enjoy a dramatic performance improvement when using lvardt. We generalize the synfire chain in a simulation of multiple delay line rings to help understand the computational complexity of lvardt. Additionally, we demonstrate the performance improvement obtained using lvardt on a more biologically detailed thalamocortical network. 2 Methods The techniques and simulations described here are implemented in the NEURON simulator (http://www.neuron.yale.edu). The simulator provides several global time-step integration schemes. For global fixedstep methods (Hines, 1984; Hines & Carnevale, 1997), one can select either a first-order backward Euler integration scheme that is numerically stable for all reasonable neural models or the second-order Crank-Nicholson method. There are two global variable-step methods, both part of the Livermore SUNDIALS package , CVODES and IDA (http://www.llnl.gov/ CASC/sundials/; Hindmarsh & Serban, 2002). The current implementation of lvardt uses CVODES, which solves ordinary differential equations (ODEs) of the form d
y =
f (
y, t). dt

(2.1)

Local Variable Time-Step Method

905

CVODES, like many other variable-step integrators, has an important property for the present usage: it allows rapid interpolation within the interval of the just-executed time step. Using a fully implicit fixed-step method, accuracy is proportional to dt. With the variable-step method, an absolute error tolerance (atol) is used to bound the error. In NEURON, absolute error tolerance is used in preference to proportionate error in order to avoid infinitesimal error tolerance near zero for state variables, notably voltage, that approach or pass zero. We replicated a fully connected homogeneous inhibitory interneuron network that shows rapid synchronization through mutual inhibition (Wang & Buzsaki, 1996). As a shorthand, we will call this the mutual-inhibition model. Variations on this model have been widely studied (Traub, Jefferys, & Whittington, 1999). The basic simulation used parameters identical to those in the original article (Wang & Buzsaki, 1996). Porting this simulation to lvardt required minor alterations detailed in section 3. We replicated the single-cell model from the network, demonstrating a currentfrequency response curve identical to that reported in the article (Wang & Buzsaki, 1996). We then used both NEURON’s fixed-step method and global variable time-step methods to demonstrate comparable activity in the network simulation (precise activity is dependent on randomized initial conditions). Both the original simulation and the lvardt version are available in runnable open-source form at the ModelDB web site (http://senselab.med.yale. edu/senselab/ModelDB; Hines, Morse, Migliore, Carnevale, & Shepherd, 2004). We also present a synfire simulation available as an example in the NEURON simulation package and a thalamocortical simulation based on published sources (Bazhenov, Timofeev, Steriade, & Sejnowkski, 1998). Simulations were run on 2.40 and 2.80 GHz Intel CPUs under Linux and Solaris operating systems.

3 Results The global variable time-step method has advantages in any simulation where periods of intense simulation activity alternate with periods where state variables remain relatively constant for a period of time. In general, this situation is more likely to occur during simulation of a single cell rather than a network. The larger the network simulation, the greater is the likelihood that a neuron somewhere in the network is showing spike activity. Using a global variable time-step method, this activity slows the entire simulation to tend to that one neuron’s integration needs. Neurons that are not active will be integrated with an unnecessarily short time step. These considerations suggested the development of the local variable time-step method (lvardt) to integrate a network piecemeal, providing short time-step integration for active neurons and long time-step integration

906

W. Lytton and M. Hines

A 40

mV

Global 5 ms

−80

B 40 Local 5

−80 Figure 1: Voltage trajectories for two cells are shown for the global variable-step method (A) and lvardt(B). dt (interval between marks) varies together at top and separately at bottom. The network is shown in schematic: the stimulator cell (filled circle) fires at time 0 and drives the bottom cell weakly with a delay of 0.1 ms (+ on the graphs), and the right cell strongly with a delay of 1 ms (vertical lines on the graphs). The right cell drives the bottom cell moderately with a delay of 0.1 ms. The two cells have Hodgkin-Huxley dynamics. The right cell spike threshold is −10 mV, and the second-order correct threshold event is marked with a—on the bottom panel. The threshold event does not necessarily lie on an integration time-step boundary.

for inactive neurons, while maintaining consistent integration accuracy throughout. Neurons that fire at different times get their state variables calculated at different times and, more important, different intervals (see Figure 1). Using the global method (top graph), the two cells have their trajectories calculated at the same times. With lvardt (bottom graph), integration points are independent. This is most obvious at the beginning of the simulation, when the cell that fires first (at right on the schematic;

Local Variable Time-Step Method

907

vertical lines on graph) has only 2 integration points, while the other cell (bottom; crosses) has 12 integration points. Where the trajectories cross, the first-spike cell integrator is called frequently, while the second-spike cell integrator is using longer dt. At the peak of the first spike, both cells are being updated frequently since the second-spike cell is coincidentally approaching threshold. At the peak of the second spike, however, the firstspike cell is at the falling phase of its spike and has far fewer integration points. State variables change quickly near threshold and at the peak of the action potential. At these times, the integrators use short time steps to accurately follow the trajectory through state space. At other times, much larger dt’s can be used to achieve the same accuracy for the more slowly varying state variables. Using lvardt, a neuron that is inactive does not waste CPU time. Overall performance evaluation for this simple simulation demonstrates that the global method integrates its 8 state variables (the 4 Hodgkin-Huxley variables m, h, n, V for each cell) 177 times, for a total of 1416 state-variable integrations. The integrators in the lvardt example integrate 4 state variables 138 times (first-spike cell) and 115 times (second-spike cell) for a total of 4 · (138 + 115) = 1012 state-variable integrations. This suggests the possibility of a 40% speed-up. The lvardt method creates a separate CVODES integrator for each of Nc cells in the network. Although there are many more integrators, each integrator is more compact since it only has to handle the state variables belonging to its particular neuron. Whether using one or Nc integrators, the total number of state variables remains the same. Although the expected relative performance gain with lvardt by function call statistics in Figure 1 is 40%, there is constant overhead for each step associated with each integrator (total overhead proportional to Nc ) and overhead required to determine which cell is to be integrated next, proportional to log(Nc ) for each step (total overhead proportional to Nc · log(Nc )). We discuss queue overhead in the section, “Estimating simulation complexity,” but except for very large numbers of cells, it reduces performance only slightly. Because the system now is being calculated forward in time by multiple, independent integrators, an integration coordinator is used to maintain the overall coherence of the integration. If the various neurons in the network are not connected, as in the case of testing parameter variation over a set of neurons, such coordination is not needed. However, in a network, the integration coordinator is vital to permit synaptic signals to be communicated at appropriate times. 3.1 Handling Events. Handling events with lvardt requires that when an event arrives at a cell, all of the state variables for that cell are at their appropriate values for that time. This is accomplished with three standard variable-step-integrator operations: single-step integration, interpolation, and reinitialization. Using these operations, we ensure that incoming events

908

W. Lytton and M. Hines

are always within reach of the receiving cell’s integrator and individual integrators do not move too far beyond the network as a whole. The individual integrators maintain state and derivative information on the interval of the most recent time step. That is, each neuron’s integrator can access states over an interval between the beginning ta and the end tb of a time step: tbi − tai = dt i for the ith neuron. This gives each individual integrator the ability to provide fast, high-order interpolation to a state at any time within the interpolation range defined by the two bounding times. The integration coordinator ensures that there is always overlap in j these interpolation ranges: tai ≤ tb ∀ i, j. We define tb/e min as the time of the earliest event te or least advanced integration bound tb . To guarantee interpolation range overlap, the integration coordinator either handles the least time event or single-steps the least advanced integrator, whichever is earlier. In this way, no integrator’s tb , and no event, ever falls behind any ta . Figure 2 illustrates most of these operations in the context of a sample integration for six cells. Note that an input event normally requires a threestep sequence (1) interpolation to the event time, (2) handling the event (or all the outstanding events to the cell with that delivery time), and (3) reinitializing the integrator. This full three-step sequence is required only for events that alter the course of the integration. By contrast, an event that only records the value of a state requires only the interpolation step, since the recorded cell’s integration can then continue from tb rather than from the interpolation point. Similarly, threshold detection does not affect the integrator. However, it must be noted that threshold detection remains tentative until the threshold time is reached by tb/e min, because an event received by the cell in the interim may reinitialize the cell to a time prior to the tentatively calculated threshold time. 3.2 Porting the Mutual-Inhibition Model to lvardt. The mutualinhibition model is an all-inhibitory network with full connectivity. Each neuron is a single compartment (point neuron) with spike-generating sodium and potassium voltage-dependent currents of the Hodgkin¬Huxley type. The dynamics of the mutual inhibition model permits initially asynchronous firing to coalesce into synchronous firing within a few spikes. In the original versions of the mutual-inhibition model (Wang & Buzsaki, 1996), the entire network is implemented as one large, continuous set of linked ODEs. This is done by making the opening rate (kC→O in /ms) of the postsynaptic conductance a continuous and continuously differentiable Boltzmann function of presynaptic voltage: kC→O = 12.0/(1 + e xp(−(Vpr e /2))).

(3.2)

This continuous-activation synapse model has the advantage of making the entire simulation somewhat more tractable analytically. However,

Local Variable Time-Step Method

cell # 0 1 2 3 4 5

909

A

t min

time

b

trigger

B

0 1 2 3 4 5 t

min

b/e

0

C 12 3 4 5 t

min

b/e

0

D 12 3 4 5 t

min

b/e

Figure 2: Typical sequence of local integration steps in a six-cell example. The ta to tb intervals of possible interpolation are shown as black rectangles. The tb/e min is shown as a vertical dashed line. (A) Integration coordinator requests integration for lagging cell (# 0 with minimum tb ). Integrator advances by dt (length of hashed rectangle). In that step, we suppose cell 0 crosses the threshold, and a threshold event is generated, labeled “trigger” in panel B. This is an event whose time is tentative since unprocessed synaptic events could still influence this cell. (B) Cell 5 has minimum tb and integrates forward. (C) The trigger event is at tb/e min. The handling of this trigger creates three events to be delivered to cells 3,4,5 at varying delays (short vertical lines). We suppose that its delay places the cell 3 input event earlier than any tb . (D) The event in cell 3 is now tb/e min. Cell 3 back-interpolates, the event is handled, and cell 3 reinitializes, giving ta3 = tb3 = tb/e min. Cell 3 will be the next cell to integrate forward.

910

W. Lytton and M. Hines

1 ms Figure 3: Comparison between event-triggered (solid) and continuously activated (dashed) synaptic conductance elicited by a presynaptic action potential. Curves superimpose except for slight deviations at initiation and peak, demonstrated by 50-fold blow-ups at these locations. Threshold = −5.7 mV, Cdur = 0.41 ms.

the continuous-activation model can be criticized as being nonbiophysical, since it represents postsynaptic conductance as being activated by somatic presynaptic voltage at all voltage levels. (Destexhe, Malnen, & Sejnowski, 1994a; 1994b). At rest, this activation is generally infinitesimal and will have no effect on the simulation. A more important disadvantage of the continuous-activation synapse model is that it does not allow explicit definition of axonal and synaptic delays. In order to implement the mutual-inhibition model using lvardt, we needed to translate the continuous-activation synapses to event-driven synapses. Equation 3.2 for the continuous-activation synapse gives a steeply rising sigmoid. Thus, the transmitter release is significant only in the period in which the presynaptic action potential is above some threshold around 0 mV. Furthermore, since the action potential trajectory in this region is relatively insensitive to changing synaptic inputs, the transmitter release is well approximated by a threshold-triggered stereotypical pulse of transmitter of duration, Cdur . This latter synapse model has been extensively used (Destexhe et al., 1994a, 1994b; Lytton, 1996). Adjusting the event threshold and pulse duration parameters to least squares best fit the continuousactivation synapse conductance (see Figure 3), gives synaptic conductance trajectories so similar that simulations with the two kinds of synapses produce graphically identical results. Spike time deviations between the event-driven simulation and the original continuous simulation were minimal: 108 ± 64 µs (mean ± standard deviation), well below the duration of an action potential. Improving the fit by, for example, adjusting the maximum kC→O was unnecessary. For

Local Variable Time-Step Method

911

the mutual-inhibition model, event-driven simulations run a bit faster than the equivalent continuous simulations. This is due to the use of a single generalized synapse for each cell that accepts all of the connecting input event streams and discontinuously changes just two state variables when an event arrives. This is orders of magnitude more efficient than the continuous model, where there are about Nc synapses per cell, each with an ODE that requires an evaluation of equation 3.2 every time step. Given that fixed time-step methods remain the simulation standard, we compared the fixed time-step performance with lvardt for the mutual-inhibition model (see Figure 4). We found that a fixed time step of 0.0025 ms gave results closely comparable to those of lvardt with absolute error tolerance (atol) of 1 · 10−3 or 1 · 10−5 . In this simulation, firing of all cells is powerfully drawn into the synchronizing attractor, making the result qualitatively similar for any integration method or tolerance that produced reasonable spike trajectories for the individual cells. Figure 4A demonstrates that the lvardt simulations are relatively slow in the early phase of the simulation, where irregular firing generates high-frequency input events in all cells but then becomes far more efficient once synchrony sets in. Figure 4B explains this by showing the time steps used during the simulations. The fixed dt methods are represented here by horizontal lines. The lvardt methods produces time steps that jump around during the initial presynchrony phase of the simulation and then settle down to an alternation between large dt (>1 ms with atol = 1 · 10−3 ) in the long intervals separating the population spikes and extremely short dt during the population spike itself. This is readily understood by noting the need to calculate not only individual cell spiking during the period of the population spike but also to handle the instantaneous (zerodelay) exchange of spikes and synaptic responses to these spikes during this same brief period. Profiling of these simulations demonstrates that lvardt consumes about 12% of its total CPU time performing these event deliveries and the associated interpolations. This relatively high figure is due to the fact that spikes that are tentatively triggered in a particular cell may then need to be taken back as other inputs into that cell arrive and alter the spike time. When the global variable time-step method is used, this species of thrashing behavior sometimes resulted in severe inefficiencies. Due to the all-toall connectivity, the near-simultaneous spiking of 100 neurons places 9900 (n2 − n since no self-connectivity) near-simultaneous events on the event queue after the occurrence of spikes in all of the cells. Using lvardt, these events are generally handled in sequential order with only occasional need to recalculate threshold time (see Figure 2C). However, the global variable time-step method attempts to reconcile the mutual influence of all of these competing events using the single integrator. This led to large computer time increases (up to 60-fold) in some simulations. It is possible to provide efficient handling of such massive event influx under the global method by artificially defining a resolution interval within which events would be

912

W. Lytton and M. Hines

CPU time (min)

A 20

dt=0.0025

atol=1e−5

10 atol=1e−3 dt=0.025

100

B

200

300

400 500 model time (ms)

log(dt) (ms)

0

−2

−4

−6

Figure 4: Comparison of fixed and variable time-step methods for the mutualinhibition model. (A) CPU time increases linearly with simulation time for fixedstep method (dashed lines for dt = 2.5µs – upper curve; dt = 25µs – lower). There is a reduction in CPU load at the onset of synchrony using the variablestep method (solid lines with absolute error tolerance 1 · 10−5 – upper; 1 · 10−3 – lower). (B) Time-step size as a function of time. Log(dt) is shown for fixed dt (horizontal dashed lines) and variable dt in one neuron (solid lines).

considered to be simultaneous. However, such an ad hoc event-handling approach would not be desirable for other types of simulations. Instead, we regard the lvardt as the natural implementation for the event-driven form of the mutual-inhibition model. We further note that the all-to-all connectivity and near-perfect synchrony of the model represents an extreme simulation situation. 3.3 Generalization of Mutual-Inhibition Model Using lvardt. The use of the event-driven simulation for the mutual-inhibition model provides

Local Variable Time-Step Method

913

the desirable side effect of allowing arbitrary delays to be introduced into the simulation. We have begun exploring the effects of delays primarily in order to ensure that these would be handled readily without introducing unexpected errors or inefficiencies. We found that CPU times using lvardt decreased slightly with the introduction of a 2 ms delay (from 1.43 min to 1.07 min). CPU times were also similar with introduction of inhomogeneity in the cells’ intrinsic frequencies (1.33 min), introduction of variability in the delays (1.33 min), or variability in both intrinsic frequency and delay times (1.57 min). Introduction of brief delay shifted but did not otherwise interfere with synchronization in the homogeneous case. Introduction of a range of delays (1.9–2.1 ms; uniform distribution) also did not interfere with synchronization. Further increasing the delay range to 2 to 5 ms produced slightly broader population spikes. However, activity still synchronized within four to five cycles as before. These manipulations increased lvardt integration efficiency by 10% to 20%. With an inhomogeneous population of neurons having different natural firing frequencies, the population spike broadened considerably, again without interfering significantly with the number of cycles required to achieve synchrony. Here again, there was only a mild reduction of integration speed, comparable to that seen with randomization of delays. 3.4 Use of lvardt with a Synfire Chain. The synfire chain, introduced by Abeles (1991; Aviel, Mehring, Abeles, & Horn, 2003) is optimal for application of the lvardt method. In this classical simulation, sets of cells are fired sequentially due to synaptic connectivity density that is greatest from each set to a follower set. At any one time, activity is restricted to a small set of cells that are carrying the signal forward, while other cells in the simulation are quiescent or are firing at lower background rates. Our evaluation of simulations consisting of 100 single-compartment Hodgkin-Huxley cells showed a 20-fold speed-up using lvardt as compared to fixed dt method with similar accuracies. In general, synfire simulations can be expected to show speed-ups of one to two orders of magnitude depending on the size of the chain, background firing rates, and forward versus lateral connectivity densities. Profiling of these simulations demonstrated that event-handling overhead was insignificant. 3.5 Use of lvardt in Thalamocortical Simulation. The highly structured, stereotyped simulations described above were meant to highlight situations in which lvardt would be particularly useful (synfire chain) or would be likely to encounter problems (mutual-inhibition model). We also benchmarked lvardt with a more complex thalamocortical simulation more closely related to activity in the nervous system. This simulation features four cell types: cortical pyramidal neurons and interneurons, thalamocortical cells, and thalamic reticular neurons (Bazhenov et al., 1998). The two

914

W. Lytton and M. Hines

thalamic cell types produced bursts with prolonged interburst intervals, a situation particularly advantageous for the use of the lvardt algorithm. To preserve accurate spike times out to 150 ms of simulation time, we had to use high-accuracy simulations: an error tolerance (atol) of 1 · 10−6 for the lvardt method and a dt of 1 · 10−4 for the fixed time-step method. (The concept of accuracy is somewhat problematic when considering these complex network simulations, as will be discussed further below.) The results were striking: 10 hour 20 minute simulation time for fixed dt and 6 minutes 13 seconds for lvardt, a 100-fold speed-up. Less dramatic results were obtained when comparing to a more typical fixed dt of 1 · 10−2 , comparable in this simulation to the lvardt method with error tolerance of 1 · 10−3 . In this case, the simulations took 2 minutes for lvardt and 5 minutes 53 seconds for fixed dt, a 3-fold speed-up. As in the original Bazhenov et al., (1998) model, several cell parameters were randomized to introduce variability into the model. In general, added variability would be expected to be advantageous for lvardt, increasing the likelihood that the integrators would require different time steps at a given point in the simulation. In practice, repetitive drive in this simulation dominated dynamics so that changing the degree of cellular variability made little difference. Similarly, addition of noisy inputs had little effect in this simulation. In general, addition of strong, high-frequency noisy inputs would be expected to remove lvardt advantages, requiring interrupts and reinitialization at the input frequencies. Such an extreme case would also adversely affect performance of the global variable time-step method. Although the fixed dt method would be unaffected, it would be inaccurate: all events would be rounded off to the nearest dt, an aliasing producing false input synchrony. Addressing this by reduction to an appropriately small fixed dt would again leave the variable time steps with an advantage. 3.6 Estimating Simulation Complexity. In order to perform detailed benchmarking and complexity evaluation, we needed a hybrid simulation that would allow us to scale simulation size without altering the simulation pattern. This was not readily done with the thalamocortical simulation, where scaling to greater numbers of neurons produced activity spread that depended critically on boundary conditions and parameter-scaling choices, despite preservation of qualitative activity pattern. We therefore went back to the synfire chain, reconfiguring it as a set of rings (see Figure 5A) to make it scale well and produce continuous activity. Each neuron in this rings simulation has 10 active compartments with Hodgkin-Huxley sodium, potassium, and leak conductances, that is, 40 states. Increasing the number of neurons in a ring increases the size of the simulation without increasing the amount of parallel activity: each ring is a delay line in which only one neuron is active at a given time. An increase in the number of rings increases the amount of activity occurring simultaneously.

Local Variable Time-Step Method

915

A

128 64 32

ba

l

2

16

fix e

d

dt

glo

CPU time: log (sec)

B

8 4

1 2 1 1

2 3 4 sim size: log (# cells)

Figure 5: (A) Schematic of a rings simulation using 8 rings of 10 neurons. The 10 tightly coupled compartments of each neuron have standard Hodgkin-Huxley channels. Activity is passed around each ring independently. (B) Log-log plot of simulation time versus simulation size. Simulations range in size from 10 to 20,480 neurons (total of 400 to 819,200 state variables) in 1 to 128 rings. With global and fixed dt, results for different number of rings overlap. With lvardt (dashed lines), simulation time increases with increased number of rings (number above each line). The asterisk shows the simulation represented in A with the dashed vertical line indicating run time with fixed and global methods. Times are on a Pentium CPU running at 3 GHz.

As expected, simulation time for the global and fixed methods depends on only the number of cells, not on how they are divided into rings. Therefore, the “global” and “fixed dt” curves for different number of rings all overlap in Figure 5B. The relative location of these curves reflects the usual choice of time step = 0.025 ms for the fixed method and atol = 1 · 10−3 for the global variable-step method. The lvardt curves indicate an enormous speed advantage when run with a small number of rings (lower set of dashed curves labeled 1, 2, 4) where

916

W. Lytton and M. Hines

only 1, 2 respectively 4 cells will be active at any given time. As we increase the number of rings, hence increasing the number of cells simultaneously active, lvardt takes more computation time to simulate a given number of cells. As the number of simultaneously active cells approaches the total number of cells, lvardt will be placed at a disadvantage compared to the other integration methods, as it wastes time maintaining the integrator queue and because of the Nc –fold increase in constant integrator overhead. At the left side of each lvardt curve (small number of cells per ring), doubling the number of rings doubles the number of active cells and approximately doubles simulation time. As the number of cells per ring rises, simulation time increases only very slightly at first and then rises more steeply, causing the curves to converge somewhat as the number of cells per ring gets very large. With these very large rings, the time spent integrating the cell that is firing is swamped by the time doing maximum time steps in the large number of quiescent cells. Evaluation of the lvardt integration in the ring simulation allowed us to develop an empirical weighting of simulation time that indicates the complexity of the lvardt method:




tbig · θ + (1 − θ) ·
tsmall   · Ts · s + To + Tq · log(Nc ) .

Tr un = Nc ·

tstop
tsmall

(3.3)

Total simulation time (Trun ) for the lvardt method will be proportional to number of cells (Nc ) and total model time (tstop ). The characteristic inverse dependence on time step (
tsmall ) must here be weighted by a θ factor, indicating the proportion of neuron time spent crawling through spikes at small dt.
tbig is also included since inactive cells will be expected The proportion of
tsmall to have a characteristic time step related to subthreshold synaptic input interval in the interspike interval. For the fixed-step method,
tbig =
tsmall . For the global variable-step method, θ is dependent on the proportion of active to inactive in the network as a whole. For all methods, simulation time will be dependent on the number of state variables s per neuron and the time Ts required to integrate a single state variable. lvardt has two more dependencies: general overhead time To for handling each integrator and queue handling overhead for determining which integrator goes next. This latter term scales as Nc · log(Nc ) rather than Nc , reflecting the scaling for a queue sorting algorithm. Profiling demonstrates that integrator queue effect on simulation time is minimal (coefficient Tq is relatively small). For 128 rings of size 160 each, management of the integrator queue is under 3% of the simulation time, while integration is 94%. When using 1-compartment instead of 10-compartment cells (4 instead of 40 state variables) with this

Local Variable Time-Step Method

917

size network, state integration still dominates the calculation, with queue time increasing to only 6% of the simulation time. In Figure 2, we depicted times for events and integration boundaries as they would appear on a single queue. In the NEURON implementation, events are maintained on an event queue while integrators are maintained on a separate integrator queue. Only the latter is considered in the Tq term of equation 3.3. We did not consider the time for the event queue in this equation since it is the same regardless of which of the three integration methods is being used. 4 Discussion The opposing demands of the continuous and event-triggered aspects of neural simulation suggest that the problem can be split up. Individual neurons are computationally demanding. By contrast, connectivity does not require much CPU time, although its representation may be demanding of memory (up to n2 of neuron number). Thus, it is natural to separate the simulation problem into calculation of the continuous neural potentials and calculation of events and their effects. When the problem is split in this way, it is immediately recognized that the computational demands of each neuron will differ among themselves at any given time. This suggests the use of the local time-step method. 4.1 Event-Driven Simulation. Prior work has demonstrated a variety of methods for efficient handling of event-driven neural network simulations (Makino, 2003; Mattia & Del Giudice, 2000; Watts, 1994). However, these networks have been restricted to use with artificial cells, which permit analytical solution or approximation of cell states based on values at an arbitrary prior time. In such a network, cell states are calculated at the time of event receipt based on values determined at the prior event. In addition to external events (te ), the event queue for an artificial network may also contain self-events. For example, an artificial cell may use an event to alert itself to the end of its refractory period, permitting resetting of an internal state flag. Such self-events are typically of fixed period and can be added effortlessly to the queue. In an artificial cell network, the simulator maintains a queue of scheduled events that are then evaluated in order. To handle an event, the simulator updates the states of follower cells and places on the queue any new events generated by these followers. Since many practical simulations involve only a small set of event delay times, the need for O(logNc ) queue sorting is avoided, and the event queue can make use of efficient algorithms such as the O(1) algorithm presented by Mattia and Del Giudice (2000). Similar to the above, a network in NEURON can be constructed entirely of event-driven artificial cells. In an artificial cell network without realistic cells, there is no need for integration. In this setting, lvardt creates no

918

W. Lytton and M. Hines

integrators and does not need an integrator queue, making use of only the event queue. States for artificial cells are computed analytically on the arrival of events, and output events are then added to the event queue. By contrast with the event queue, the integrator queue always contains Nc -ordered tb events, where Nc is the number of cells. The tb times are uncorrelated with each other and with any te times. This variability in event order means that the integrator queue requires a fully general algorithm with characteristic log(Nc ) complexity per time step. In NEURON, both the event and integrator queues are based on Jones’s (1986) implementation of the splay tree algorithm of Sleator and Tarjan (1983). As we have shown, integrator queue execution time is negligible even for simulations on the order of 10,000 cells. 4.2 Contrasting Example Simulations. In order to demonstrate the general usefulness of the method, we explored its performance in several examples. The mutual-inhibition model is relatively unsuited to the use of any variable time-step method, proceeding to full synchronization within the round-off error for the double precision representation available. With regard to lvardt, such synchronization is a worst-case scenario from a performance perspective, since perfect synchrony of identical neurons means that all integrators are redundantly performing the same calculations at the same time. In addition, the mutual-inhibition model places increasing demands on event accounting, as the event-delivery algorithm attempts to reconcile the mutual effects of the n2 − n events arriving nearly simultaneously. Performance on the mutual-inhibition model simulation can be greatly improved by providing a window wherein arriving events are considered simultaneous and can be handled in the order received. We have explored this implementation but did not pursue it since it is an ad hoc solution to a peculiar simulation with pathologically synchronized behavior and since the lvardt method performed adequately despite this handicap. Moving beyond this artificially handicapped situation, it can readily be seen that the best use of lvardt will be in larger simulations involving heterogeneous neural populations where activity bounces from one area to another or spreads across the network. The synfire chain simulation represents the best-case extreme with only a small subset of neurons being active at a given time. lvardt can integrate these few, devoting no resources to the many that are done or await their moment. The ring simulation generalizes the synfire chain to allow the simulation size and the extent of simultaneous activity to be independently scaled. With a large enough number of rings, lvardt will be expected to lose its advantage, as integrator switching between simultaneously active cells will dominate. The more complex thalamocortical simulation was also assessed to demonstrate that the new method has real virtual-world application. While performance of lvardt on this simulation produced excellent results, comparison of simulations run using different integration methods with different

Local Variable Time-Step Method

919

degrees of precision raised questions of the appropriate standard of accuracy for simulations. In general, network simulations will not converge to a single, correct result since long runs will invariably yield a near-threshold event, which will produce spikes at slightly different times or not at all, ultimately dependent on round-offs, an example of sensitivity to initial conditions. From this point onward, the divergence of activity in the single neuron may spread to alter firing patterns throughout the network. In the case of a synchronizing network such as the mutual-inhibition model, the strength of the synchronizing attractor will resolve such deviations. In the general case, however, these deviations result in entirely different spike trains after a certain point, regardless of the degree of accuracy requested. This invariable variability requires some metric other than strict spike occurrence times to identify firing pattern similarities and the adequacy of an integration method (Victor & Purpura, 1996).

4.3 Parallel Computation. The use of parallel integrators for different neurons naturally raises the issue of porting this simulation method to a parallel computer. The NEOSIM project (http://www.neosim.org) has developed a parallel discrete event sample implementation for the delayed delivery of spike events from a source cell on one CPU to a target cell on another. This implementation coexists with NEURON’s lvardt method. ij High performance requires a significant minimum spike delay time td , between source cell i and target cell j. In this case, our strict integration j j ij assertion property tai ≤ tb ∀ i, j is relaxed to tai ≤ tb + td . The NEURON portion of the NEOSIM + NEURON implementation merely accepts a request from NEOSIM to integrate a specific cell or group of cells to a specified stop time consistent with the relaxed assertion. When a NEURON cell fires ij at time t, it notifies NEOSIM. At this time, no cell has a ta > t + td . Note that in our zero-delay mutual-inhibition model, this parallel method would be useless. However, most network models have a significant minimum delay between different cells and can realize significant performance gains with the NEOSIM parallel discrete event delivery technique.

5 Conclusion In summary, lvardt offers substantial speed-ups for simulations of networks of realistic ion-channel-based neurons. The advantages will be greatest in situations where some neurons are quiescent during periods when other portions of the network are active. This would be the case for simulations involving serial activation of areas, as, for example, in hippocampus, or involving cell types with very different firing properties, as, for example, in a thalamocortical or basal ganglia simulation. lvardt’s mixture of event-driven and differential equation simulation also makes it ideal for implementation

920

W. Lytton and M. Hines

of hybrid networks, where artificial cells with analytically soluble states are combined with realistic neurons requiring full ODE integration. As with any other computational method, the suitability of lvardt is dependent on the exact problem to be solved. The individual user will want to benchmark a particular simulation across methods before deciding which one to use. A general assessment of suitability can be performed by considering the factors laid out in equation 3.3. Our heuristic conclusion is that for medium-size networks in which average synaptic input intervals to a single cell are much greater than a fixed step dt, the lvardt method will have better performance than the fixed-step method. Acknowledgments This work was supported by NINDS grants NS11613 and NS32187. We thank the reviewers for many helpful suggestions. References Abeles, M. (1991). Corticonics: Neural circuits of the cerebral cortex. Cambridge: Cambridge University Press. Aviel, Y., Mehring, C., Abeles, M., & Horn, D. (2003). On embedding synfire chains in a balanced network. Neural Computation, 15, 1321–1340. Bazhenov, M., Timofeev, I., Steriade, M., & Sejnowski, T. (1998). Computational models of thalamocortical augmenting responses. Journal of Neuroscience, 18, 6444–6465. Cohen, S., & Hindmarsh, A. (1994). Cvode user guide (Tech. Rep.). Livermore, CA: Lawrence Livermore National Laboratory. Destexhe, A., Mainen, Z., & Sejnowski, T. (1994a). An efficient method for computing synaptic conductances based on a kinetic model of receptor binding. Neural Computation, 6, 14–18. Destexhe, A., Mainen, Z., & Sejnowski, T. (1994b). Synthesis of models for excitable membranes, synaptic transmission and neuromodulation using a common kinetic formalism. J. Comput. Neurosci, 1, 195–230. Hindmarsh, A., & Serban, R. (2002). User documentation for cvodes, an ode solver with sensitivity analysis capabilities (Tech. Rep.). Livermore, CA: Lawrence Livermore National Laboratory. Hines, M. (1984). Efficient computations of branched nerve equations. Int. J. BioMedical Computing, 15, 69–76. Hines, M., & Carnevale, N. (1997). The neuron simulation environment. Neural Computation, 9, 1179–1209. Hines, M., & Carnevale, N. (2001). Neuron: A tool for neuroscientists. Neuroscientist, 7, 123–135. Hines, M., Morse, T., Migliore, M., Carnevale, N., & Shepherd, G. (2004). Modeldb: A database to support computational neuroscience. J. Comput. Neurosci, 17, 73–77. Jones, D. (1986). An empirical comparison of priority-queue and event-set implementations. Comm. ACM, 4, 300–311.

Local Variable Time-Step Method

921

Lytton, W. (1996). Optimizing synaptic conductance calculation for network simulations. Neural Computation, 8, 501–510. Makino, T. (2003). A discrete-event neural network simulator for general neuron models. Neural Computing and Applications, 11, 210–223. Mattia, M., & Del Giudice, P. (2000). Efficient event-driven simulation of large networks of spiking neurons and dynamical synapses. Neural Computation, 12, 2305– 2329. Sleator, D., & Tarjan, R. (1983). Self adjusting binary trees. In Proc. ACM SIGACT Symposium on Theory of Computing (pp. 235–245). New York: ACM Press. Traub, R., Jefferys, J., & Whittington, M. (1999). Fast oscillations in cortical circuits. Cambridge, MA: MIT Press. Victor, J., & Purpura, K. (1996). Nature and precision of temporal coding in visual cortex: A metric-space analysis. J. Neurophysiol, 76, 1310–1326. Wang, X., & Buzsaki, G. (1996). Gamma oscillation by synaptic inhibition in a hippocampal interneuronal network model. J. Neurosci, 16, 6402–6413. Watts, L. (1994). Event-driven simulation of networks of spiking neurons. In J. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6. (pp. 927–934). San Mateo, CA: Morgan Kaufmann.

Received December 18, 2003; accepted August 25, 2004.

LETTER

Communicated by Anthony Burkitt

Synaptic Shot Noise and Conductance Fluctuations Affect the Membrane Voltage with Equal Significance Magnus J. E. Richardson [email protected]

Wulfram Gerstner [email protected] Laboratory of Computational Neuroscience, I&C and Brain-Mind Institute, Ecole Polytechnique F´ed´erale de Lausanne, CH-1015 Lausanne EPFL, Switzerland

The subthreshold membrane voltage of a neuron in active cortical tissue is a fluctuating quantity with a distribution that reflects the firing statistics of the presynaptic population. It was recently found that conductancebased synaptic drive can lead to distributions with a significant skew. Here it is demonstrated that the underlying shot noise caused by Poissonian spike arrival also skews the membrane distribution, but in the opposite sense. Using a perturbative method, we analyze the effects of shot noise on the distribution of synaptic conductances and calculate the consequent voltage distribution. To first order in the perturbation theory, the voltage distribution is a gaussian modulated by a prefactor that captures the skew. The gaussian component is identical to distributions derived using current-based models with an effective membrane time constant. The well-known effective-time-constant approximation can therefore be identified as the leading-order solution to the full conductance-based model. The higher-order modulatory prefactor containing the skew comprises terms due to both shot noise and conductance fluctuations. The diffusion approximation misses these shot-noise effects implying that analytical approaches such as the Fokker-Planck equation or simulation with filtered white noise cannot be used to improve on the gaussian approximation. It is further demonstrated that quantities used for fitting theory to experiment, such as the voltage mean and variance, are robust against these non-Gaussian effects. The effective-time-constant approximation is therefore relevant to experiment and provides a simple analytic base on which other pertinent biological details may be added. 1 Introduction Given a perfect model of the membrane response to synaptic input, it would be possible to infer from the distribution of the subthreshold, membranevoltage fluctuations many quantities of interest, such as the levels of activity and correlations in the excitatory and inhibitory presynaptic populations. Neural Computation 17, 923–947 (2005)

© 2005 Massachusetts Institute of Technology

924

M. Richardson and W. Gerstner

Early models of synaptic input (Stein, 1965) comprised a leaky integrator driven by a stochastic current, which generated postsynaptic potentials of fixed amplitude. Since then, great effort has been made to incorporate further biological details. Soon after the publication of Stein’s model, synaptic conductance effects began to be addressed (Stein, 1967; Johannesma, 1968; Tuckwell, 1979; Wilbur & Rinzel, 1983; Lansky & Lanska, 1987). These early models featured unfiltered, delta-pulse synapses and were primarily concerned with the statistics of the interspike interval distribution. Although the majority of studies used the diffusion approximation (i.e., the limit of high synaptic rates and low postsynaptic potential amplitudes), the effects of shot noise due to Poisson distributed pulse arrival at low rates have also been considered (see, e.g., Tuckwell, 1989) in the context of stochastic resonance (Hohn & Burkitt, 2001) and the neural response to correlations in the presynaptic population (Kuhn, Aertsen, & Rotter, 2003). Other studies have examined the filtering of the incoming pulses at the synapses and have shown it can lead to unexpected dynamical response properties: synaptic filtering can, paradoxically, allow neurons to follow high-frequency signals better (Brunel, Chance, Fourcaud, & Abbott, 2001; Fourcaud & Brunel, 2002). More recently, a number of experimental studies have directly measured the effect of synaptic drive on the membrane voltage (Kamondi, Acsady, Wang, & Buzsaki, 1998; Destexhe & Par´e, 1999; Sanchez-Vives & McCormick, 2000; Monier, Chavane, Baudot, Graham, & Fr´egnac, 2003; Holmgren, Harkany, Svennenfors, & Zilberter, 2003). The availability of such measurements has led to a renewed interest in the quantitative modeling of synaptic drive, with a view to infer presynaptic network states from voltage fluctuations (Stroeve & Gielen, 2001; Rudolph, Piwkowska, Badoual, Bal, & Destexhe, 2004), compare current and conductance-based models of synaptic drive (Tiesinga, Jos´e, & Sejnowski, 2000; Rauch, La Camera, Luscher, ¨ Senn, & Fusi, 2003; Rudolph & Destexhe, 2003; Jolivet, Lewis, & Gerstner, 2004; Richardson, 2004; La Camera, Senn, & Fusi, 2004; Meffin, Burkitt, & Grayden, 2004), and explore mechanisms for the gain control of the neuronal response (Chance, Abbott, & Reyes, 2002; Burkitt, Meffin, & Grayden, 2003; Destexhe, Rudolph, & Par´e, 2003; Fellous, Rudolph, Destexhe, & Sejnowski, 2003; Prescott & De Koninck, 2003; Grande, Kinney, Miracle, & Spain, 2004; Kuhn, Aertsen, & Rotter, 2004). In this letter, the combined effects on the membrane voltage of synaptic shot noise, filtering, and conductance increase will be examined. The central result is that the effects of synaptic shot noise on the membrane voltage statistics are as significant as those of synaptic conductance fluctuations and therefore either both (or neither) of these features of the synaptic drive should be taken into account for a consistent approach. This means that diffusion-level descriptions, such as numerical simulations or the FokkerPlanck approach, in which the drive is modeled as gaussian noise, cannot correctly describe detailed aspects of the membrane-voltage distribution, such as its skew.

Shot Noise and Conductance Fluctuations

925

2 Membrane Response to Synaptic Drive In this section, the full model of the membrane response to synaptic drive is introduced and two common approximations to this model outlined. An analysis of the aspects of the drive missed by these approximation schemes will motivate the development of a perturbative approach. 2.1 The Full Model. Following Stein (1967), the membrane voltage V(t) responds passively to synaptic drive: voltage gated channels, including spike-generating currents, are not included. The membrane is modeled by a capacitance C in parallel with a leak conductance g L and two fluctuating excitatory ge (t) and inhibitory gi (t) conductances with equilibrium potentials at E L , E e , and E i , respectively. This system therefore comprises three independent variables: C

dV = −g L (V − E L ) − ge (V − E e ) − gi (V − E i ) + Ia pp dt

(2.1)

τe


  dge δ t − tke = −ge + c e τe dt {tk }

(2.2)


  dgi = −gi + c i τi δ t − tki . dt {tk }

(2.3)

e

τi

i

The excitatory conductance is driven by pulses that arrive at the Poissondistributed times {tke } at a total rate Re summed over all input fibers. Each pulse provokes a quantal conductance increase c e , which then decays exponentially with a time constant τe . The inhibitory conductance is defined analogously. Any experimentally applied current is accounted for by Ia pp . In this letter, only the steady-state statistical properties will be considered. Thus, all expectations of a quantity x(t), written as
x(t), denote either an average over an ensemble of statistically independent systems, in which any transients due to initial conditions are no longer present, or the temporal average of x(t) in a single system. 2.2 The Diffusion Approximation. For the case in which the rates Re , Ri are relatively high, the number of pulses that arrive within the timescales τe , τi will be approximately gaussian distributed. The replacement of the synaptic shot noise in equations 2.2 and 2.3 by a constant term and gaussian white noise constitutes the diffusion approximation. Thus, using excitation as an example, τe

√ dge  ge0 − ge + 2σe ξe (t), dt

(2.4)

926

M. Richardson and W. Gerstner

where the gaussian white noise ξe (t) has a mean and autocorrelation function defined by
ξe (t)ξe (t  ) = τe δ(t − t  ).


ξe (t) = 0

(2.5)

The Ornstein-Uhlenbeck process (see equation 2.4) has been shown to capture the statistics of conductance fluctuations at the soma of compartmentalized model neurons (Destexhe, Rudolph, Fellous, & Sejnowski, 2001). The average conductance ge0 and the standard deviation σe are related to the variables c e , τe , and Re through  ge0 = c e τe Re ,

σe = c e

τe Re . 2

(2.6)

By construction, the first two moments of the diffusion approximation are identical to those of the shot noise process. Higher moments, however, are not correctly reproduced in the diffusion approximation. The conductance equation 2.4 is linear and can be integrated.1 The fluctuating component ge F of the conductance is ge F (t) ≡ ge (t) − ge0 

 √ 2σe 0

∞

ds −s/τe e ξe (t − s), τe

(2.7)

which yields (with equation 2.5) the gaussian distribution p D (ge ) = 

  (ge − ge0 )2 . exp − 2σe2 2πσe2 1

(2.8)

The subscript signifies that the calculation was made in the diffusion approximation. There are clearly some problems with distribution 2.8 if the conductance mean ge0 is of a similar magnitude to the standard deviation σe . In this regime, the diffusion approximation predicts negative conductances (Lansky & Lanska, 1987; Rudolph & Destexhe, 2003). In fact, the criterion for validity of the diffusion approximation is σe /ge0  1,

1

(2.9)

The Stratonovich formulation of stochastic calculus is used throughout this letter. However, for additive white noise or multiplicative colored noise, there is no difference between the Stratonovich or Ito forms. See, for example, Risken (1996).

Shot Noise and Conductance Fluctuations

927

suggesting that this approximation misses higher-order terms scaling with powers of σe /ge0 . Thus, the shot noise conductance fluctuations should read ge F (t) =

 √ 2σe

∞

0

  ds −s/τe σe ξe (t − s) + corrections ∝ , e τe ge0

(2.10)

where ξe (t) is the gaussian white noise defined in equation 2.5. 2.3 The Diffusion Approximation Is Inconsistent. The combination of the diffusion approximation of the synaptic drive (see equation 2.4 and its equivalent for inhibition) and the full voltage equation, 2.1, will now be examined. By separating the synaptic conductances into tonic components ge0 , gi0 and fluctuating components ge F , gi F , the voltage equation can be written as C

dV = −g0 (V − E 0 ) − ge F (V − E e ) − gi F (V − E i ), dt

(2.11)

where the total conductance g0 and drive-dependent equilibrium potential E 0 are defined by g0 = g L + ge0 + gi0

and

E0 =

1 (g LE L + ge0 E e + gi0 E i + Ia pp ). g0 (2.12)

The subscripts 0 anticipate that these quantities are correct at the zero order of a perturbation expansion that will be developed in a later section. The total conductance g0 suggests the introduction of an effective membrane time constant, τ0 = C/g0 .

(2.13)

This feature of the synaptic drive was identified in the early analytic treatment of Johannesma (1968). The fluctuation terms driving the voltage in equation 2.11 will now be examined. Taking excitation as an example, the voltage-dependent component of the drive can be expanded around the equilibrium potential E 0 , ge F (V − E e ) = ge F (E 0 − E e ) + ge F (V − E 0 ).

(2.14)

The two terms on the right-hand side have simple interpretations. The first is an additive noise term and therefore just a fluctuating current. The second is a multiplicative noise term and, in the context of equation 2.11, it can be

928

M. Richardson and W. Gerstner

seen that this term represents fluctuations in g0 , or equivalently in τ0 , the effective membrane time constant. These two noise terms are, however, not equally significant. The quantity V− E 0 grows (linearly) with the fluctuations ge F , gi F . So whereas the additive noise terms are of the order ge F , gi F , the multiplicative noise terms are of the order ge2F , gi2F , and ge F gi F . This suggests that (1) the multiplicative noise terms could be neglected if the noise strength was in some way small, and (2) if these terms were retained, the effects of the synaptic drive on the membrane voltage would be modeled in greater detail. Point 1 is valid, as will be seen in section 2.4. Point 2, however, is false due to an unexpected weakness of the diffusion approach with multiplicative noise. This will now be outlined. On reexamining equations 2.7 to 2.9, it is seen that relative to the tonic conductance, the fluctuations in the diffusion approximation scale with σe /ge0 . But equation 2.10 states that the terms missed by this approximation scale with the square of this quantity. Hence, 

σe ge F /ge0 = A ge0



 +B

σe ge0

2 + ···

(2.15)

where A is the diffusion-level term and B is the first-order correction due to shot noise. Given that σe /ge0 is the small quantity parameterizing the diffusion approximation, it is clearly inconsistent to neglect the secondorder term B in the additive noise ge F (E e − E 0 ) of equation 2.14 but keep the implicit A2 term in the multiplicative noise ge F (V− E 0 ) ∝ ge2F . This is, however, what occurs in the diffusion approximation. This result is surprising because it implies that although diffusion-based approaches (such as the Fokker-Planck equation or any simulation with filtered gaussian noise) purport to capture the effects of synapticconductance fluctuations, they miss equally important terms due to the shot noise. However, it should be stressed that almost all previous studies of conductance-based synaptic noise that used the diffusion approximation implicitly concentrated their analyses on the dominant effects coming from the tonic conductance increase and additive noise term; the conclusions of such studies remain valid. 2.4 The Effective-Time Constant Approximation. This is also known as the gaussian approximation of the voltage distribution. The treatment of the membrane voltage can easily be made consistent with the diffusion approximation of the synaptic conductance equations. This is achieved by dropping the multiplicative noise term, that is, by neglecting conductance fluctuations, to yield C

dV  −g0 (V − E 0 ) + ge F (E e − E 0 ) + gi F (E i − E 0 ). dt

(2.16)

Shot Noise and Conductance Fluctuations

929

This voltage equation is of the form of a current-based model, but the dominant effect of the synaptic conductance is accounted for through the use of an increased effective leak g0 . This approximation is in widespread use, having been applied to white noise synaptic drive (Wan & Tuckwell, 1979; Lansky & Lanska, 1987; Burkitt & Clark, 1999; Burkitt, 2001; Burkitt et al., 2003, La Camera et al., 2004), alpha-pulse synapses (Manwani & Koch, 1999), and, more recently (Richardson, 2004), to the case of exponentially filtered synapses studied here. The equation set comprising the voltage equation 2.16 and the diffusion approximations for the conductances are simple to analyze and can be integrated to give V(t) − E 0 





  (E e − E 0 ) ∞  −s/τe ds e − e −s/τ0 ξe (t − s) (τe − τ0 ) 0   √  σi (E i − E 0 ) ∞  −s/τi + 2 ds e − e −s/τ0 ξi (t − s). g0 (τi − τ0 ) 0

√

2

σe g0 

(2.17) This equation has an obvious interpretation: the quantities multiplying the noise are just the excitatory and inhibitory postsynaptic potentials for a membrane with an effective time constant τ0 . The fact that it is linear in the noise means that many quantities of interest can be easily calculated, including temporal measures such as the autocorrelation function. The distribution predicted for the voltage is the gaussian p0 (V) =

  (V − E 0 )2 , exp − 2σV2 2πσ 2 1

(2.18)

V

where, for the case where there are no correlations between excitation and inhibition, the variance is (Richardson, 2004)  σV2 =

σe g0

2 (E e − E 0 )2

τe + (τe + τ0 )



σi g0

2 (E i − E 0 )2

τi . (τi + τ0 )

(2.19)

If the limit τe , τi → 0 is correctly taken (by keeping the quantities c e τe /C and c i τi /C fixed), it can be shown that this variance is compatible with previous results derived for the gaussian approximation of white noise conductance-based synaptic drive (Burkitt et al., 2003). However, for filtered noise, the variance in equation 2.19 differs significantly from that derived in Rudolph and Destexhe (2003). In that study, a one-dimensional Fokker-Planck equation was used that could not capture the effects of synaptic filtering. Through the introduction of effective synaptic time constants (Rudolph et al., 2004), the one-dimensional Fokker-Planck equation can be

930

M. Richardson and W. Gerstner

made to yield results that correspond, at the gaussian level and in the steady state, to the distribution parameterized by equations 2.12 and 2.19. 2.5 The Aim of This Letter. The gaussian approximation provides a mathematically convenient approach to the analysis of conductance-based synaptic drive and is accurate for parameter values relevant to experiment (Richardson, 2004). Given the analysis presented above, it is clear that to improve on the gaussian approximation, both shot noise and conductance fluctuations must be included. The goal of the next two sections will be to develop a perturbative method that allows for the consistent calculation of the conductance and voltage distributions at a higher order than the gaussian approximation. These higher-order calculations will yield the skew of the voltage distribution, a quantity that is measurable experimentally. More important, the approach will provide information on the validity of fitting gaussian-level analytical forms for the mean and variance to voltage traces of cortical neurons. To aid readability, only the results of the calculations are given in the main body of the article. However, the methods developed here are applicable to other areas of theoretical neuroscience, such as the distribution of amplitudes at depressing synapses (Hahnloser, 2003) or the shape of synaptic weight distributions (van Rossum, Bi, & Turrigiano, 2000; Rubin, Lee, & Sompolinsky, 2001), and are therefore presented in full in the appendixes. 3 Synaptic Shot Noise and Conductance Distributions In this section, the effects of shot noise on the synaptic conductance distributions will be analyzed. It should be noted that relaxation processes with shot noise, for which equations 2.2 and 2.3 are examples, have been well studied, and an exact solution (Gilbert & Pollak, 1960) for the distribution, in the form of a recursion relation, does exist. However, the aim of the approach (in section 4) is to incorporate the shot-noise conductance fluctuations into a model of the membrane voltage. A perturbative approach is better suited to this purpose. For this reason, the full solution for the shot-noise distribution p S (ge ) will not be presented here but, when needed, will be obtained by numerical simulation of equation 2.2. 3.1 The Diffusion Approximation Misses the Skew. In the limit where the standard deviation σe has a similar magnitude to the conductance mean ge0 , the diffusion approximation, unlike the full model, predicts negative conductances. A second source of difference between the statistics of shot noise and the diffusion approximation is also seen in the same limit; the distribution of the shot noise conductance becomes skewed, an effect that is obviously missed by the gaussian distribution given in equation 2.8. In order to get some intuition about the skew of the distribution, a

Shot Noise and Conductance Fluctuations

C

A

Distribution

931

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

shot noise pS diff. approx. pD perturb. theory pP

0.5 0.4 0.3 0.2 0.1 0

-1

0

1

2

3

4

-1 0 1 2 3 4 5 6 7

Conductance ge /ce

Conductance ge /ce B

D

0.12

0.06

| pD - pS | | pP - pS |

Difference

0.10

0.05

0.08

0.04

0.06

0.03

0.04

0.02

0.02

0.01

0.00

0.00 -1

0

1

2

3

Conductance ge /ce

4

-1 0 1 2 3 4 5 6 7

Conductance ge /ce

Figure 1: Distribution of shot noise conductance fluctuations; the perturbation theory improves on the diffusion approximation. (A) Comparison of the full distribution p S generated by the simulation of equation 2.2 to the diffusion approximation p D (see equation 2.8) and the perturbation theory p P (see equation 3.4) for the case σe /ge0 = 0.60 (Re τe = 1.5). (B) The corresponding absolute difference between the diffusion approximation and full solution | p D − p S | and also the perturbatively generated distribution and the full solution | p P − p S |. The perturbative distribution reduces the error caused by both the negative conductances and the skew. (C, D) Analogous measures for the case σe /ge0 = 0.41 (Re τe = 3.0) for which the theoretical approaches can be expected to be more accurate. Details of the simulations are given in appendix A.

comparison can be made between the full and approximate distributions shown in Figure 1A. In this case (for which Re τe = 1.5 implying σe /ge0 = 0.60), the peak of the shot noise distribution is to the left of that of the gaussian. Because both distributions have the same mean conductance ge /c e = 1.5, the shot noise distribution is skewed; it leans to the left with

932

M. Richardson and W. Gerstner

a longer tail to the right. Any improvement of the diffusion approximation should address both the negative conductivity and the skew of the conductance distribution. 3.2 Accounting for the Shot Noise. The corrections identified in equation 2.10 will now be accounted for. A stochastic variable ζe (t), analogous to gaussian white noise ξe (t), τe

√ dge  ge0 − ge + 2 σe ζe (t), dt

(3.1)

can be constructed that has statistics that capture the shot noise fluctuations correctly up to the next order missed by the diffusion approximation. It can be shown that such a quantity must obey the same first- and second-order correlators as gaussian white noise,
ζe (t) = 0,


ζe (t)ζe (t  ) = τe δ(t − t  ),

(3.2)

but also a new third-order correlator, √
ζe (t)ζe (t )ζe (t ) = 2 





σe ge0



τe2 δ(t − t  )δ(t  − t  ).

(3.3)

It is this third-order correlator, proportional to σe /ge0 , that provides the leading-order correction to the diffusion approximation. All higher-order correlators of products of ζe (t) factorize in terms of these first-, second-, and third-order correlators. Using the rules in equations 3.2 and 3.3, the conductance distribution can be shown (see appendix B) to be    2

1 he h 4 σe h 3e p P (h e ) = √ − exp − e , 1+ 3 ge0 3! 2! 2 2π

(3.4)

where h e = (ge −ge0 )/σe is the normalized conductance and the subscript P denotes that the result was derived as a perturbative expansion in the small variables σe /ge0 . The distribution takes the form of a gaussian modulated by a prefactor. To zero order in σe /ge0 , the prefactor is equal to one, and the gaussian distribution 2.8 is recovered. The prefactor terms proportional to σe /ge0 now allow for the moments of the distribution to be calculated at higher order. The mean and variance are unchanged, as would be expected given the previous comments about the exactness of these two moments. The first new result of the perturbation theory is the skew Sge of the distribution: Sge =

4 σe 1 (ge − ge0 )3 = h 3e = . σe3 3 ge0

(3.5)

Shot Noise and Conductance Fluctuations

933

A useful aspect of the perturbation theory is that this skew is exact. The distribution itself and its higher moments are, however, correct only at the given order of the series expansion in σe /ge0 . Two examples comparing the numerically generated conductance distribution p S , diffusion approximation p D and perturbation theory p P , are plotted in Figure 1. 4 The Subthreshold Voltage Distribution The model of synaptic conductance studied in the previous section can now be incorporated into the membrane voltage equation. This will allow the voltage distribution to be calculated at the next order beyond the gaussian approximation. The method involves a perturbative solution to the voltage equation 2.1, the excitatory synaptic conductance equation 3.1, and its inhibitory analog. For the perturbative calculation of the voltage distribution, it is convenient to use the following small parameters, xe = σe /g0

xi = σi /g0 ,

and

(4.1)

which are linearly related (in σe , σi ) to the small parameters of the conductance expansion σe /ge0 and σi /gi0 . The calculation for the voltage distribution is given in appendix C and, in terms of v = V − E 0 , can be written in the form p P (v) =

1 2πσV2



v 1+ σV



µV S − σV 2!



   v3 S v2 + 3 exp − 2 , 2σV σV 3!

(4.2)

where the subscript P denotes the perturbatively generated result. The voltage appears only through the ratio v/σV , and the other terms µV /σV and S are parameters proportional to xe , xi : this distribution generates moments
vm /σVm that are correct up to order xe , xi . The quantity µV is the leading-order correction to the voltage mean E 0 and stems from the conductance fluctuations only: the shot noise does not influence the mean voltage. The standard deviation, given by equation 2.19, is identical to the gaussian value σV and is therefore unaffected by shot noise or multiplicative conductance at this order in the perturbation expansion. Thus,
V − E 0 = µV

and


(V −
V)2  = σV2 .

(4.3)

The third-order moment of the distribution 4.2 gives the skew of the voltage distribution, 1
(V −
V)3  = S = SSN + SC F . σV3

(4.4)

934

M. Richardson and W. Gerstner

From the expression given in appendix C, equation C.20, it can be seen that two distinct contributions to the skew naturally arise: one from the shot noise S SN and a second one from the conductances fluctuations SC F . These two contributions to the skew are equally significant because they are both proportional to xe , xi . This illustrates one of the central points of this study: the diffusion approximation of a conductance-based model with multiplicative noise is inconsistent because it misses the shot noise contribution SSN . The full set of equations for µV , σV , and S is given in appendix D. 4.1 An Example with Relevance to Experiment. To illustrate the effects of shot noise and conductance fluctuations, a scenario is considered in which the fluctuations due to the inhibitory component of the drive can be neglected. There are two different situations that allow this action to be taken. The first is when inhibition is absent. The second, and more interesting, case is relevant to experiments designed to isolate the effect of excitation on the membrane voltage (Silberberg, Wu, & Markram, 2004). In such experiments, the neuron is hyperpolarized through the injection of current so that the mean voltage E 0 is near the reversal of inhibition E i . In such cases, the factor E i − E 0 multiplying all inhibitory contributions to membrane fluctuations is relatively small, and such contributions can be dropped without significant loss of accuracy. Inhibition enters only through an increase of the tonic conductance g0 and the corresponding decrease of the effective time constant τ0 . For either of these scenarios, the moments that parameterize the distribution in equation 4.2 take the values τe (τe + τ0 ) τe σV2 = xe2 (E e − E 0 )2 (τe + τ0 )

µV = −xe2 (E e − E 0 )

 8 g0 (τe + τ0 )2 τe 3 ge0 (τe + 2τ0 )(2τe + τ0 ) (τe + τ0 )  2  3τe + 6τe τ0 + 2τ02 τe = −4xe . (τe + 2τ0 )(2τe + τ0 ) (τe + τ0 )

(4.5) (4.6)

S SN = xe

(4.7)

SC F

(4.8)

Equations 4.7 and 4.8 give the positive and negative contributions to the skew (see equation 4.4) that come from the shot noise and conductance fluctuations, respectively. For the case of purely excitatory drive, g0 = g L + ge0 , the relative importance of these contributions can be gauged by examining the ratio      S SN  τL (τe + τ0 )2  = 2 , S  3 (τ L − τ0 ) 3(τe + τ0 )2 − τ02 CF

(4.9)

Shot Noise and Conductance Fluctuations

935 2

Low-conductance state: g0=0.0667mS/cm , τ0=15ms 80 70 60 50 40 30 20 10 0 -0.02 0

B 0.12

diff. approx. perturb. theory shot-noise sim.

0.10

C 0.4

ETC approx. perturb. theory full model sim.

0.08 0.06 0.04

2

0.0

SSN+ SCF

-0.2

0.02 0.02 0.04 0.06

SSN

0.2

Skew

Distribution

A

0.00 -75 -70 -65 -60 -55 -50 -45

SCF

-0.4

0 0.05 0.1 0.15 0.2

Voltage V (mV)

Conductance ge (mS/cm )

xe=σe/g0 2

High-conductance state: g0=0.2mS/cm , τ0=5ms. 7 6 5 4 3 2 1 0 -0.1 0 0.1 0.2 0.3 0.4

E

F

0.04

0.4

0.03

0.0

Skew

Distribution

D

0.02

2

Conductance ge (mS/cm )

SSN+ SCF

-0.4

0.01

-0.8

0.00 -120 -100 -80 -60 -40 -20

-1.2

Voltage V (mV)

SSN

SCF 0

0.1 0.2 0.3 0.4

xe=σe/g0

Figure 2: Distribution of the membrane voltage; perturbation theory captures the skew. A neuron is subject to a purely excitatory synaptic drive with a current Iapp applied such that E 0 = −60 mV. (A, B) The conductance and voltage distributions for a low conductance state (ge0 = 0.0167, g L = 0.05 mS/cm2 ) with noise strength xe = σe /g0 = 0.2. The perturbative conductance distribution (see equation 3.4) is not accurate because σe /ge0 = 0.8. The weak skew of the corresponding voltage distribution (B) is, however, correctly predicted by the perturbation theory (see equation 4.2) because the underlying conductance skew is exact. (C) The voltage skew (see equations 4.7 and 4.8) is plotted as a function of xe for the same parameters, but with increasing noise σe . The shot noise S SN and conductance-fluctuation SC F contributions to the skew nearly cancel, explaining the almost gaussian voltage distribution in B. (D, E) A high conductance state (ge0 = 0.15 mS/cm2 ) with xe = σe /g0 = 0.4. (E) The large skew of the voltage distribution is captured by the perturbation theory. (F) The voltage skew is negative for the high-conductance case because SC F dominates. Details of the simulations are given in appendix A.

where τ L = C/g L is the leak time constant. The ratio is a monotonically increasing function of the effective time constant τ0 . In the limit of low conductance states, for which τ0 → τ L , the ratio diverges, and the contribution due to conductance fluctuations becomes negligible. For high-conductance states, for which τ0 → 0, the ratio converges to a constant value of 2/9. These

936

M. Richardson and W. Gerstner

results underline the fact that the effect of shot noise is nonnegligible: even in extremely high conductance states it still comprises just under a third, S SN /S = 2/7, of the net skew. These results are illustrated graphically in Figure 2. 5 Discussion The effect that shot noise synaptic drive has on the membrane voltage distribution was examined. A perturbative approach was developed that was first used to capture the statistics of filtered shot noise conductance fluctuations beyond both the gaussian effective-time-constant approximation and the diffusion approximation. These synaptic conductances were then incorporated into a model of the membrane voltage response. The approach allowed for the analysis of nongaussian features of the voltage distribution, such as its skew. In particular, it was shown that shot noise and synaptic conductance fluctuations affect the membrane at the same order: both effects need to be taken into account for a consistent approach. The regime in which the effects of shot noise on the voltage and firing rate might be clearly seen experimentally, is one of low presynaptic rate and large, sharp excitatory postsynaptic potentials (EPSPs). This is typical of the excitatory drive experienced by certain neocortical interneurons (Silberberg et al., 2004) for which isolated EPSPs can be many millivolts and there is little dendritic filtering. For a case in which the effects of shot noise are strong (outside the perturbative regime considered here), the voltage distribution can be considerably positively skewed with increased probability to be near threshold. It is expected that in such a case, the statistics of the generated action potentials would differ significantly from those predicted using a gaussian model of the membrane fluctuations with the same mean and variance. The gaussian, or effective-time constant approximation for the membrane distribution, is, however, mathematically simple: the mean (see equation 2.12) and variance (see equation 2.19) are transparent functions of the model parameters. Such gaussian distributions are therefore ideal to fit to experimental data (Rudolph et al., 2004) in cases where the shot noise effects are weak. The functional form of the distribution that takes into account the shot noise and conductance fluctuations is, however, somewhat less transparent as can be seen in equations D.5 and D.6 for the skew. So the question should be asked: To what extent would weak higher-order effects interfere with an attempt to fit the mean and variance to an experimental distribution? This question can be answered in the framework presented here. First, it is seen from equations 4.5 and D.3 that the correction to the mean voltage due to shot noise and conductance fluctuations is of order xe2 , xi2 ,  
V = E 0 + µV + · · · = E 0 + O xe2 , xi2 .

(5.1)

Shot Noise and Conductance Fluctuations

937

Hence, the mean is not affected at first order. The same is true for the measured variance,   
(V −
V)2  =
(V − E 0 )2  − (
V − E 0 )2 = σV2 1 + O xe2 , xi2 ,

(5.2)

which also increases only with xe2 , xi2 , despite the fact that the skew grows linearly with xe , xi . These results demonstrate that information extracted from the voltage mean using equation 2.12 and variance using equations 2.19 is not strongly affected by shot noise and conductance fluctuations missed in the gaussian approximation. Hence, fitting the gaussian-level moments to voltage traces is a robust method, given that equations 2.1 and 2.4 and their inhibitory counterpart provide a sufficiently realistic model of the effect of synaptic drive on the membrane voltage. In summary, the gaussian effective-time-constant approximation provides an accurate description of the voltage fluctuations and is a convenient tool for fitting theory to experiment. For most situations, its description of the stochastic voltage dynamics due to conductance-based synaptic drive is adequate, and it can be easily extended to include many biological details (such as voltage-dependent currents, dynamic synapses, heterogeneity, nontrivial temporal correlations in the drive, and others) missed in the simplified model considered here. Nevertheless, for the purposes of detailed modeling of conductance-based synaptic drive, it should not be overlooked that shot noise and conductance fluctuations are equally important. Our results demonstrate that diffusion-based approaches such as the FokkerPlanck equation or simulation using multiplicative filtered gaussian noise are inadequate for the description of the nongaussian statistics of the voltage. If the aim is to model or simulate the statistics of voltage fluctuations beyond the gaussian, effective-time-constant approximation, then synaptic shot noise must be included. Appendix A: Details of the Simulations The parameters used for the simulations were τe = 3 ms for the excitatory synaptic filtering, C = 1 µF/cm2 for the membrane capacitance, and g L = 0.05 mS/cm2 for the leak conductance. The reversal potentials used were E L = −80 mV for the leak and E e = 0 mV for synaptic excitation. Simulations were performed using the Euler method with the Poissonian synaptic shot noise implemented by integrating the conductance equation 2.2 to yield ge (t + dt) = ge (t) −

dt ge (t) + c e P(Re dt), τe

(A.1)

where c e is the postsynaptic conductance amplitude for a single pulse and Re is the total rate of incoming pulses. The quantity P(Re dt) is the random

938

M. Richardson and W. Gerstner

number of incoming pulses that arrive within the time step dt. The number is drawn from a Poisson distribution characterized by the mean Re dt. Appendix B: Filtered Poissonian Shot Noise The method for expanding higher-order gaussian correlators is first reviewed. The first- and second-order correlators are given in equation set 2.5. All higher odd-order correlators vanish, and higher even-order correlators (of order 2n) factorize into (2n)!/(2n n!)

(B.1)

permutations of products of n second-order correlators. As an example, and writing ξ (t1 ) = ξ1 for simplicity, the fourth-order correlator is
ξ1 ξ2 ξ3 ξ4  =
ξ1 ξ2 
ξ3 ξ4  +
ξ1 ξ3 
ξ2 ξ4  +
ξ1 ξ4 
ξ2 ξ3 .

(B.2)

A fluctuating quantity ζe (t) is now introduced with statistics that are constructed so as to capture the effects of shot noise at a higher order than gaussian white noise ξ (t). The factorization properties of high-order correlators of ζe (t) can be derived from its first-, second-, and third-order correlators defined in equation set 3.2 and equation 3.3. These rules can be derived by expanding the Poissonian distribution of the shot-noise Ornstein-Uhlenbeck equation 2.2 and by keeping terms beyond the usual diffusion approximation (see, e.g., Rodriguez, Pesquera, San Miguel, & Sancho, 1985). To order σe /ge0 , higher even-order correlators obey the usual gaussian factorization rules and higher odd-order correlators can be decomposed into permutations of a product of a single third-order correlator and an appropriate number of second-order correlators. As an example, and using the shorthand ζe (t1 ) = ζ1 , the seventh-order correlator is factorized as follows:
ζ1 ζ2 ζ3 ζ4 ζ5 ζ6 ζ7  =
ζ1 ζ2 ζ3 
ζ4 ζ5 ζ6 ζ7  + permutations,

(B.3)

where for this case, there are 7 · 6 · 5 permutations of the indices of the third-order correlator. Each fourth-order correlator can then be decomposed using the usual gaussian rules (see eq. B.2). It is important to note than no further third-order correlators are extracted out of the remaining even-order product. Otherwise, this would produce terms that go beyond the σe /ge0 correction. Hence, for a (2n + 3)-order correlator, there are (2n + 3)(2n + 2)(2n + 1) ·

(2n)! 2n n!

(B.4)

Shot Noise and Conductance Fluctuations

939

permutations. The first set of three terms comes from the different ways of arranging the single third-order correlator, and the final term comes from the gaussian statistics of the reduction of the remaining even-order correlator. B.1 The Conductance Distribution and Correlators. The normalized conductance variable h e = (ge −ge0 )/σe is introduced to simplify the following analysis. It obeys the equation τe

√ dh e = −h e + 2 ζe (t), dt

(B.5)

which can be integrated to yield √  h e (t) = 2

t −∞

ds −(t−s)/τe e ζe (s). τe

(B.6)

From this, the correlators of the conductance are found to be
h e (t) = 0
h e (t)h e (t  ) = exp(−|t − t  |/τe )
h e (t)h e (t  )h e (t  ) =

4 σe exp(−|t − t  |/τe ) exp(−|t  − t  |/τe ), 3 ge0

(B.7)

with higher-order correlators derivable from these using the underlying factorization rules for ζe (t). The steady-state distribution of the variable h e (t) can be obtained by calculating the probability density that h e (t) is found having a value near he :  p(h e ) =
δ(h e − h e (t)) =

∞

−∞

dq −iq h e iq h e (t) e . e 2π

(B.8)

The exponential is expanded to give ∞ ∞ iq h e (t)
(iq )2m 2m
(iq )3+2m 3+2m e = (t) . h e (t) + h 2m! (3 + 2m)! e m=0 m=0

(B.9)

The structure of the correlators allows this to be rewritten as

 m   ∞


1 −q 2 4σe e iq h e (t) = 1 + (iq )3 m! 2 3ge0 m=0   4σe 2 = 1 + (iq )3 e −q /2 , 3ge0

(B.10)

940

M. Richardson and W. Gerstner

which can be inserted into equation B.8,  p(h e ) =

1−

4σe d 3 3ge0 dh 3e



∞

−∞

dq −iq h e −q 2 /2 , e 2π

(B.11)

to yield the distribution given in equation 3.4. Appendix C: The Membrane Distribution The statistics of the conductance fluctuations (given in equation 3.1) now are incorporated into a model of a passive membrane (see equation 2.1). For the following analysis, it is convenient to use the shifted voltage v = (V − E 0 ), with normalized conductances h e , h i defined in equations B.5 and B.6, τ0 v˙ + v(1 + xe h e + xi h i ) = xe Ee h e + xi Ei h i ,

(C.1)

where τ0 = C/g0 , E 0 are defined by equation 2.12, Ee = E e − E 0 , and xe = σe /g0 provides the small parameter used for the perturbative analysis of the voltage (with a similar definition of xi ). Because these small parameters are linearly related to those used for the conductance perturbation theory, corrections due to shot noise and conductance fluctuations will be simultaneously accounted for. Equation C.1 can be integrated to give  v(t) =

t

−∞

t ds −(t−s)/τ0  − α(s) e s e τ0

dr τ0

β(r )



,

(C.2)

where the terms α(s) generate corrections to voltage-like quantities and β(r ) generates corrections to the effective time constant: α(s) = xe Ee h e (s) + xi Ei h i (s) β(r ) = xe h e (r ) + xi h i (r ).

(C.3)

The voltage distribution can now be obtained by evaluating the expectation  p(v) =
δ(v − v(t)) =

∞

−∞

dq −iq v iq v(t) e e , 2π

(C.4)

to the appropriate order in xe and xi . No correlations are assumed to exist between excitation and inhibition. This simplifying assumption can be relaxed, and the method used here easily extended to account for such correlations.

Shot Noise and Conductance Fluctuations

941

C.1 The Leading-Order Voltage Distribution. The derivation (Richardson, 2004) of the leading-order contribution to the voltage distribution of equation set 2.1 to 2.3 is first reviewed. The fluctuations of the voltage from its mean value are written as v(t) = σ (t) + O(xe2 , xi2 ) where  σ (t) =

t −∞

ds −(t−s)/τ0 e (xe Ee h e (s) + xi Ei h i (s)) . τ0

(C.5)

In this approximation, the leading-order probability density is a gaussian, as can be seen by examining  p0 (v) =

∞

−∞

dq −iq v iq σ (t) e e , 2π

(C.6)

where the expectation

q2 q4 1 2 2 e iq σ (t) = 1 −
σ (t)2  +
σ (t)4  · · · = e − 2 q σV 2 4!



(C.7)

is evaluated using the gaussian relation
σ (t)2n  = (2n)!
σ (t)2 n /2n n! At this order, there are no contributions from the shot noise. From equation C.5, the expectation
σ 2 (t) = σV2 is time independent and takes the value σV2 = xe2 Ee2

τe τi + xi2 Ei2 . (τe + τ0 ) (τi + τ0 )

(C.8)

Reinserting the result, equation C.7, into the probability density,     ∞   σV2 v2 v 2 dq p0 (v) = exp − 2 , exp − q −i 2 2 2σV σV −∞ 2π

(C.9)

and evaluating the integral gives a gaussian voltage distribution:   1 (V − E 0 )2 . exp − p0 (V) = 2σV2 2πσV2

(C.10)

C.2 The Next-Order Correction to the Distribution. From the previous section, it is seen that the typical difference between the voltage and its mean scales with xe , xi . To develop a systematic expansion, the dimensionless variable y = v/σV is therefore introduced. At the next order,the expansion

942

M. Richardson and W. Gerstner

can be written   y(t) = σ y (t) − φ y2 (t) + O xe2 , xi2 + · · · , where σ y (t) = σ (t)/σV and φ y2 takes the form φ y2 (t) =

1 σV



t

−∞

ds − (t−s) e τ0 τ0

 s

t

dr α(s)β(r ). τ0

(C.11)

This gives the probability density correct to order xe , xi as  p0 (y) + p1 (y) =

∞

−∞

dq −iq y iq (σ y (t)−φ y2 (t)) e e . 2π

Again the exponential within the expectation will be expanded and then evaluated to first order in φ y2 : ∞ ∞


(iq )2m 2m
(iq )3+2m 3+2m 2 e iq (σ y (t)−φ y (t)) = σy + σ (2m)! (3 + 2m)! y m=0 m=0



−

∞
  (iq )1+2m 2m 2 σ y φ y + O xe2 , xi2 . (2m)! m=0

(C.12)

The first term on the right-hand side of equation C.12 is the zero-order gaussian component treated above, the second term is the correction due to the Poissonian nature of the noise, and the third term is the correction due to the conductance-based drive. The second term is straightforward to analyze. Using the rules for the permutation of correlators, this term can be expanded out to give  m ∞ ∞

(iq )3+2m 3+2m 1 −q 2 = (iq )3 σ y3 , σy (3 + 2m)! m! 2 m=0 m=0

(C.13)

which takes the form of a gaussian with a prefactor. To obtain the third term of equation C.12, expectations of the form
σ y2m φ y2  need to be calculated. An examination of the structure of the integrals comprising this term shows that they can be written as





σ y2m φ y2 = σ y2m φ y2 + 2m·(2m − 1) ψ y4 σ y2m−2 .

(C.14)

Shot Noise and Conductance Fluctuations

943

The expectation
φ y2  can be calculated from the form given above, and
ψ y4  is defined by 4 1 ψy = 3 σ



t

−∞

ds1 ds2 ds3 τ0 τ0 τ0



t

s3

dr3
α(s1 )α(s3 )
α(s2 )β(r3 ), τ0

(C.15)

where
ψ y4  ∼ O(xe , xi ). An explicit form for this quantity will be given in appendix D. Substitution of the form C.14 into the third term of the expansion C.12 gives  m ∞ ∞

(iq )1+2m 2m 2  2 1 −q 2 σ y φ y = iq φ y + (iq )3 ψ y4 . (2m)! m! 2 m=0 m=0

(C.16)

Inserting the results of equations C.13 and C.16 into the expansion C.12 gives

  1 2 2 e iq (σ y (t)−φ y (t)) = 1 + (iq )3 σ y3 − iq φ y2 − (iq )3 ψ y4 e − 2 q ,

(C.17)

where the fact that
σ y2  = 1 has been used. Inserting this into the leading order correction to the distribution,  ∞  1 2 dq −iq y  (iq )3 σ y3 − iq φ y2 − (iq )3 ψ y4 e − 2 q p1 (y) = e −∞ 2π   ∞ 2 d  4 3  d 3 dq −iq y − 1 q 2 = φy e 2 + ψy − σy e dy dy3 2π −∞     1 2 = − φ y2 y + ψ y4 − σ y3 (y3 − 3y) √ e −y /2 , 2π

(C.18)

yields the perturbatively generated distribution, correct to order xe , xi , with the following functional form, 1 p(y) = √ 2π





S 1 + y µy − 2!



  2 y S +y exp − , 3! 2 3

(C.19)

where µ y is the correction to the mean voltage and S is the skew, µ y =
y = − φ y2 and

  S =
(y −
y)3  = 6 σ y3 − ψ y4 , (C.20)

and the equalities hold to first order in the perturbation theory. The first correction to the mean of y is affected only by the synaptic conductance. However,there are two components of the skew S = SSN + SCF : a contribution

944

M. Richardson and W. Gerstner

SSN = 6
σ y3  from the Poissonian nature of the drive and a contribution SCF = −6
ψ y4  from synaptic conductance fluctuations. The functional forms of µ y and S will be evaluated by the quantities
φ y2 ,
σ y3  and
ψ y4  in the next section. Appendix D: The Voltage Mean and Skew At this order in perturbation theory, any of the higher-order cumulants of the voltage distribution can be expressed in terms of the mean µ y and the skew S,
y2m  =

(2m)! 2m m!

and


y2m+1  =

(2m + 2)!  m  + µ S , y 2m+1 (m + 1)! 3

(D.1)

where m = 0, 1, 2 · · · Only the odd correlators are different from the gaussian approximation at this order. D.1 Voltage Mean. The first quantity to be evaluated is the correction to the mean. Because of equation C.20, the integral

1 φ y2 = σ



t −∞

ds − (t−s) e τ0 τ0

 s

t

dr
α(s)β(r ) τ0

(D.2)

must be evaluated. These integrals can be performed using the equation set B.7 and yield for µV =
v:  µV = −

xe2 Ee

 τe τi 2 + xi Ei . (τe + τ0 ) (τi + τ0 )

(D.3)

D.2 Voltage Skew: The Poissonian Contribution. Due to equation C.20, this requires the evaluation of

σ y3 =

 3  t ds ds  ds  − (3t−s−sτ  −s  ) 1 0 e
α(s)α(s  )α(s  ), σ −∞ τ0 τ0 τ0

(D.4)

which can be performed using the result for the third-order correlator given in equation set B.7. This yields for S SN = 6
σ y3 , S SN =

1 σ3



 8Ei3 τi3 xi4 (g0 /gi ) 8Ee3 τe3 xe4 (g0 /ge ) + . 3(τe + 2τ0 )(τ0 + 2τe ) 3(τi + 2τ0 )(τ0 + 2τi )

(D.5)

Shot Noise and Conductance Fluctuations

945

D.3 Voltage Skew: The Conductance Contribution. This is given by −6
ψ y4  and therefore requires the evaluation of the integral given in equation C.15. After some algebraic effort, the result can be written in the form SC F

 2   2 3τe + 6τe τ0 + 2τ02 (τe + 2τ0 )(2τe + τ0 )     2 3τi2 + 6τi τ0 + 2τ02 4xi4 Ei3 τi − σ3 τi +τ0 (τi + 2τ0 )(2τi + τ0 )   2x 2 x 2 E 3 Ei τe τi (2τe τi +τ0 (τe +τi ))(2τe (τi +τ0 ) − τi τ0 ) 2+ − 3 e i e σ (τe +τ0 )(τi +τ0 ) (2τe + τ0 )(2τi + τ0 )(τe τi + τe τ0 + τi τ0 )   2 2 3 2xi xe Ei Ee τi τe (2τi τe +τ0 (τi +τe ))(2τi (τe +τ0 ) − τe τ0 ) 2+ . − 3 σ (τi +τ0 )(τe +τ0 ) (2τi + τ0 )(2τe + τ0 )(τi τe + τi τ0 + τe τ0 )

4x 4 E 3 = − e3 e σ



τe τe +τ0

(D.6) If only one synaptic input type is present or if the average voltage is near the reversal of inhibition such that Ei = E i − E 0  0, this result greatly simplifies. This case is given in equation 4.8 and compared to simulations of the full model in Figure 2. References Brunel, N., Chance, F. S., Fourcaud, N., & Abbott, L. F. (2001). Effects of synaptic noise and filtering on the frequency response of spiking neurons. Phys. Rev. Lett., 86, 2186–2189. Burkitt, A. N. (2001). Balanced neurons: Analysis of leaky integrate-and-fire neurons with reversal potentials Biol. Cybern., 85, 247–255. Burkitt, A. N., & Clark, G. M. (1999). New technique for analyzing integrate and fire neurons. Neurocomputing, 26–27, 93–99. Burkitt A. N., Meffin, H., & Grayden, D. B. (2003). Study of neuronal gain in a conductance-based leaky integrate-and-fire neuron model with balanced excitatory and inhibitory synaptic input. Biol. Cybern., 89, 119–125. Chance, F. S., Abbott, L. F., & Reyes, A. D. (2002). Gain modulation from background synaptic input. Neuron, 35, 773–782. Destexhe, A., & Par´e, D. (1999). Impact of network activity on the integrative properties of neocortical pyramidal neurons in vivo. J. Neurophysiol., 81, 1531–1547. Destexhe, A., Rudolph, M., Fellous, J.-M., & Sejnowski, T. J. (2001). Fluctuating synaptic conductances recreate in vivo–like activity in neocortical neurons. Neuroscience, 107, 13–24. Destexhe, A., Rudolph, M., & Par´e, D. (2003). The high-conductance state of neocortical neurons in vivo. Nature Rev. Neurosci., 4, 739–751. Fellous, J.-M., Rudolph, M., Destexhe, A., & Sejnowski T. J. (2003). Synaptic background noise controls the input/output characteristics of single cells in an in vitro model of in vivo activity. Neuroscience, 122, 811–829.

946

M. Richardson and W. Gerstner

Fourcaud, N., & Brunel, N. (2002). Dynamics of the firing probability of noisy integrate-and- fire neurons. Neural Comput., 14, 2057–2110. Gilbert, E. N., & Pollak, H. O. (1960). Amplitude distributions of shot noise. Bell. Syst. Tech. J., 39, 333–350. Grande, L. A., Kinney, G. A., Miracle G. L., & Spain W. J. (2004). Dynamic influences on coincidence detection in neocortical pyramidal neurons. J. Neurosci., 24, 1839– 1851. Hahnloser, R. H. R. (2003). Stationary transmission distribution of random spike trains by dynamical synapses. Phys. Rev. E, 67, 022901. Hohn, N., & Burkitt, A. N. (2001). Shot noise in the leaky integrate-and-fire neuron. Phys. Rev. E, 63, 031902. Holmgren, C., Harkany, T., Svennenfors, B., & Zilberter, Y. (2003). Pyramidal cell communication within local networks in layer 2/3 of rat neocortex. J. Physiol. London., 551, 139–153. Johannesma, P. I. M. (1968). Diffusion models for the stochastic activity of neurons. In E. R. Caianello (Ed.), Neural networks (pp. 116–144). New York: Springer. Jolivet, R., Lewis, T. J., & Gerstner, W. (2004). Generalized integrate-and-fire models of neuronal activity approximate spike trains of a detailed model to a high degree of accuracy. J. Neurophysiol., 92, 959–976. Kamondi A., Acsady, L., Wang, X.-J., & Buzsaki, G. (1998). Theta oscillations in somata and dendrites of hippocampal pyramidal cells in vivo: Activity-dependent phaseprecession of action potentials. Hippocampus, 8, 244–261. Kuhn, A., Aertsen, A., & Rotter, S. (2003). Higher-order statistics of input ensembles and the response of simple model neurons. Neural Comp., 15, 67–101. Kuhn, A., Aertsen, A., & Rotter, S. (2004). Neuronal integration of synaptic input in the fluctuation-driven regime. J. Neurosci., 24, 2345–2356. La Camera, G., Senn, W., & Fusi, S. (2004). Comparison between networks of conductance and current-driven neurons: Stationary spike rates and subthreshold depolarization. Neurocomputing, 58–60, 253–258. Lansky, P., & Lanska, V. (1987). Diffusion approximation of the neuronal model with synaptic reversal potentials. Biol. Cybern., 56, 19–26. Manwani, A., & Koch, C. (1999). Detecting and estimating signals in noisy cable structures, I: Neuronal noise sources. Neural Comp., 11, 1797–1829. Meffin, H., Burkitt, A. N., & Grayden, D. B. (2004). An analytical model for the “large, fluctuating synaptic conductance state” typical of neocortical neurons in vivo. J. Comput. Neurosci., 16, 159–175. Monier, C., Chavane, F., Baudot, P., Graham, L. J., & Fr´egnac, Y. (2003). Orientation and direction selectivity of synaptic inputs in visual cortical neurons: A diversity of combinations produces spike tuning. Neuron, 37, 663–680. Prescott, S. A., & De Koninck, Y. (2003). Gain control of firing rate by shunting inhibition: Roles of synaptic noise and dendritic saturation. P. Natl. Acad. Sci., 100, 2076–2081. Rauch, A., La Camera, G., Luscher, ¨ H.-R., Senn, W., & Fusi, S. (2003). Neocortical pyramidal cells respond as integrate-and-fire neurons to in vivo–like input currents. J. Neurophysiol., 90, 1598–1612. Richardson, M. J. E. (2004). Effects of synaptic conductance on the voltage distribution and firing rate of spiking neurons. Phys. Rev. E, 69, 051918.

Shot Noise and Conductance Fluctuations

947

Risken, H. (1996). The Fokker-Planck equation. New York: Springer-Verlag. Rodriguez, M. A., Pesquera, L., San Miguel, M., & Sancho, J. M. (1985). Master equation description of external Poisson white noise in finite systems. J. Stat. Phys., 40, 669–724. Rubin, J., Lee, D. D., & Sompolinsky, H. (2001). Equilibrium properties of temporally asymmetric Hebbian plasticity. Phys. Rev. Lett., 86, 364–367. Rudolph, M., & Destexhe, A. (2003). Characterization of subthreshold voltage fluctuations in neuronal membranes. Neural Comput., 15, 2577–2618. Rudolph, M., Piwkowska Z., Badoual, M., Bal, T., & Destexhe, A. (2004). A method to estimate synaptic conductances from membrane potential fluctuations. J. Neurophysiol., 91, 2884–2896. Sanchez-Vives, M. V., & McCormick, D. A. (2000). Cellular and network mechanisms of rhythmic recurrent activity in neocortex. Nat. Neurosci., 3, 1027–1034. Silberberg, G., Wu, C. Z., & Markram, H. (2004). Synaptic dynamics control the timing of neuronal excitation in the activated neocortical microcircuit. J. Physiol-London, 556, 19–27. Stein, R. B. (1965). A theoretical analysis of neuronal activity. Biophys. J., 5, 173–193. Stein, R. B. (1967). Some models of neuronal variability. Biophys. J., 7, 37–68. Stroeve, S., & Gielen, S. (2001). Correlation between uncoupled conductance-based integrate-and-fire neurons due to common and synchronous presynaptic firing. Neural. Comp., 13, 2005–2029. Tiesinga, P. H. E., Jos´e, J. V., & Sejnowski, T. J. (2000). Comparison of currentdriven and conductance-driven neocortical model neurons with Hodgkin-Huxley voltage-gated currents. Phys. Rev. E, 62, 8413–8419. Tuckwell, H. C. (1979). Synaptic transmission in a model for stochastic neural activity. J. Theor. Biol., 77, 65–81. Tuckwell, H. C. (1989). Stochastic processes in the neurosciences. Philadelphia: SIAM. van Rossum, M. C. W., Bi, G. Q., & Turrigiano, G. C. (2000). Stable Hebbian learning from spike timing-dependent plasticity. J. Neurosci., 20, 8812–8821. Wan, F. Y. M., & Tuckwell, H. C. (1979). The response of a spatially distributed neuron to white noise current injection. Biol. Cybern., 33, 39–55. Wilbur, W. J., & Rinzel, J. (1983). A theoretical basis for large coefficient of variation and bimodality in neuronal interspike distribution. J. Theor. Biol., 105, 345–368.

Received August 4, 2004; accepted October 1, 2004.

LETTER

Communicated by Michael Cohen

Dynamical Behaviors of a Large Class of General Delayed Neural Networks Tianping Chen [email protected]

Wenlian Lu [email protected] Laboratory of Nonlinear Mathematics Science, Institute of Mathematics, Fudan University, Shanghai, 200433, China

Guanrong Chen [email protected] Electronic Engineering Department, City University of Hong Kong, Kowloon, Hong Kong, China

Research of delayed neural networks with varying self-inhibitions, interconnection weights, and inputs is an important issue. In the real world, self-inhibitions, interconnection weights, and inputs should vary as time varies. In this letter, we discuss a large class of delayed neural networks with periodic inhibitions, interconnection weights, and inputs. We prove that if the activation functions are of Lipschitz type and some set of inequalities, for example, the set of inequalities 3.1 in theorem 1, is satisfied, the delayed system has a unique periodic solution, and any solution will converge to this periodic solution. We also prove that if either set of inequalities 3.20 in theorem 2 or 3.23 in theorem 3 is satisfied, then the system is exponentially stable globally. This class of delayed dynamical systems provides a general framework for many delayed dynamical systems. As special cases, it includes delayed Hopfield neural networks and cellular neural networks as well as distributed delayed neural networks with periodic self-inhibitions, interconnection weights, and inputs. Moreover, the entire discussion applies to delayed systems with constant self-inhibitions, interconnection weights, and inputs. 1 Introduction Recurrently connected neural networks, sometimes called GrossbergHopfield neural networks, have been extensively studied, and many applications in different areas have been found. It is well known that many applications depend heavily on the dynamical behaviors of the networks. Therefore, analysis of these dynamic behaviors is a necessary step toward practical design of these neural networks. Neural Computation 17, 949–968 (2005)

© 2005 Massachusetts Institute of Technology

950

T. Chen, W. Lu, and G. Chen

A recurrently connected neural network is described by the following differential equations: n
dui (t) a i j g j (u j (t)) + Ii = −di ui (t) + dt j=1

i = 1, . . . , n,

(1.1)

where g j (x) are activation functions, di , a i j are constants, and Ii are constant inputs. The dynamical behaviors of recurrently connected neural networks have been studied since the early period of research on neural networks. For example, multistable and oscillatory behaviors were studied by Amari (1971, 1972) and Wilson and Cowan (1972). Chaotic behaviors were studied by Sompolinsky and Crisanti (1988). Hopfield and Tank (1984, 1986) studied the dynamical stability of symmetrically connected networks and showed their practical applicability to optimization problems. Cohen and Grossberg (1983) gave more rigorous results on the global stability of the neural networks. A number of researchers have studied local and global stability of asymmetrically connected networks (e.g., Hirsch, 1989; Matsuoka, 1992; Michel & Gray, 1990; Chen & Amari, 2001; Chen, Lu, & Amari, 2002). Wang and Zou (2002) and Lu and Chen (2003) discussed the global stability of CohenGrossberg neural networks. In practice, the system is unsynchronized. Therefore, one often needs to investigate the following delayed dynamical systems with fixed delays or time-varying delays: n
dui (t) a i j g j (u j (t)) = − di ui (t) + dt j=1

+

n


b i j f j (u j (t − τi j )) + Ii

i = 1, . . . , n

(1.2)

j=1

There are also many articles discussing dynamical behaviours of delayed neural networks. The delayed system (see equation 1.2) was investigated by Belair (1993), Belair, Campbell, and van den Driessche (1996), Cao and Wu (1996); Cao and Zhou (1998), Cao (2000), Chen and Amari (2001), Chen (2001), Gopalsamy and He (1991), Marcus and Westervelt (1989), Zhang and Jin (2000), and others. In the literature, various conditions ensuring global and exponential stabilities have been given. For more general delayed functional-differential equations, see early work by Hale (1965) and Miller (1965).

On a Large Class of Delayed Neural Networks

951

In this letter, we discuss a large class of delayed dynamical systems, which can be represented as n
dui a i j (t)g j (u j (t)) = −di (t)ui (t) + dt j=1

+

n 
j=1

∞

f j (u j (t − τi j − s))ds K i j (t, s) + Ii (t)

(1.3)

0

or n
dui a i j (t)g j (u j (t)) = −di (t)ui (t) + dt j=1

+

n 
j=1

∞

f j (u j (t − τi j (t) − s))ds K i j (t, s) + Ii (t),

(1.4)

0

where for any fixed t, ds K i j (t, s) are Lebesgue-Stieljies measures, which satisfy dsK i j (t + ω, s) = ds K i j (t, s), di (t), a i j (t), b i j (t), Ii (t), τi j (t) : R+ → R are continuously periodic functions with period ω > 0, that is, di (t + ω) = di (t), a i j (t) = a i j (t + ω), b i j (t) = b i j (t + ω), Ii (t) = Ii (t + ω), τi j (t + ω) = τi j (t) for all t > 0 and i, j = 1, 2, . . . , n. In section 3, we investigate the existence of a periodic solution for systems 1.3 and 1.4 and its global stability and exponential stability. 2 Some Definitions and Assumptions In this section, we give some definitions and assumptions that are used throughout the letter: Definition 1. Lip(G) denotes the class of functions with Lipshitz constant G > 0. Definition 2. The following two norms are defined for every t ∈ (−∞, ∞):  1. {ξ, 1}-norm. u(t){ξ,1} = i |ξi ui (t)|, where ξi > 0, i = 1, . . . , n. 2. {ξ, ∞}-norm. u(t){ξ,∞} = max i |ξi−1 ui (t)|, where ξi > 0, i = 1, . . . , n. Definition 3. For any real number a , denote a + = ma x{a , 0}. ¯ Throughout this article, we assume |ds K i j (t, s)| ≤ |dK¯ i j (s)|,  ∞where dK i j (s) is a Lebesgue-Stiejies measure independent of t and satisfies 0 s|dK¯ i j (s)| < +∞, di (t) > di > 0 with di being constants, |a i∗j | = sup{0 0, τi∗j = sup{0
952

T. Chen, W. Lu, and G. Chen

3 Main Results In this section, we establish several theorems. Theorem 1. Suppose that G i , Fi > 0, gi (·) ∈ Li p(G i ) and is nondecreasing, f i (·) ∈ Li p(Fi ), i = 1, 2 · · · , n. If there are positive constants ξ1 , . . . , ξn such that  +
ξi |a i∗j | G j −ξ j d j + ξ j a j j (t) + i= j

+

n


 ξi

∞

|dK¯ i j (s)|F j ≤ −η < 0,

j = 1, . . . , n,

(3.1)

0

i=1

then the dynamical system, equation 1.3, has a unique periodic solution v(t) = [v1 (t), . . . , vn (t)], and for any solution u(t) = [u1 (t), . . . , un (t)] of equation 1.3, lim [ui (t) − vi (t)] = 0,

i = 1, . . . , n.

t→∞

(3.2)

Proof. Denote u¯ j (t) = u j (t + ω) − u j (t), g¯ j (u j (t)) = g j (u j (t + ω)) − g j (u j (t)),f¯j (u j (t)) = f j (u j (t + ω)) − f j (u j (t)). Then we have n
d u¯ i a i j (t)g¯ j (u j (t)) = −di (t)u¯ i (t) + dt j=1

+

n 
j=1

∞

f¯j (u j (t − τi j − s))ds K i j (t, s).

(3.3)

0

Define the Lyapunov function: L(t) =

n


ξi |u¯ i (t)| +

i=1

n




∞

ξi

i, j=1

 |d K¯ i j (s)|

0

t t−τi j −s

| f j (u¯ j (y))|dy.

(3.4)

Differentiating it, we have ˙ L(t) =

n


 ξi sign{u¯ i (t)} −di (t)u¯ i (t)

(3.5)

i=1

+

n
j=1

a i j (t)g¯ j (uj (t)) +

n 
j=1

0

∞

 f¯j (uj (t − τi j − s))ds K i j (t, s)

On a Large Class of Delayed Neural Networks

+



n




n


∞

ξi

| f¯j (uj (t − τi j − s))||d K¯ i j (s)|

0

i, j=1

≤

| f¯j (uj (t))||d K¯ i j (s)|

0

i, j=1

−

∞

ξi



n




ξi −di |u¯ i (t)| + sign{u¯ i (t)}

i=1

+



n


∞

ξi

a i j (t)g¯ j (uj (t))

| f¯j (uj (t))||d K¯ i j (s)|

0





n


−ξ j d j |u¯ j (t)| + sign{u¯ j (t)}ξ j a j j (t)

j=1

+



n
j=1

i, j=1

=

953








sign{u¯ i (t)}ξi a i j (t) g¯ j (uj (t))

i= j

+

n




| f¯j (uj (t))||d K¯ i j (s)|

0

i, j=1

≤

∞

ξi

 n 


−ξ j d j + ξ j a j j (t) +

j=1

+



n


i= j

 ξi

i=1

≤ −η




n


∞

+ ξi |a i∗j |

Gj

 |d K¯ i j (s)|F j |u¯ j (t)|

0

|u¯ j (t)|.

(3.6)

j=1

˙ Then L(t) ≤ 0. Therefore, L(t) is bounded, which implies that u¯ i (t), i = 1, . . . , n, are bounded. Thus, the right side of equation 3.3 is bounded, and every u¯ i (t) is uniformly continuous. Combining equation 3.6 with 

∞ 0

n


|u¯ j (t)|dt ≤

j=1

1 L(0), η

(3.7)

we conclude that lim u¯ j (t) = 0.

t→∞

(3.8)

It is clear that with equation 3.7, for any positive integer k, when m → ∞,

954

T. Chen, W. Lu, and G. Chen

the following integral, n 


(k+1)ω

|u j (t + (m + p)ω) − u j (t + mω)|dt

kω

j=1

=

n 


(k+1)ω

kω

j=1

   ∞
p−1 n   m+  
u¯ j (t + lω)dt ≤ |u¯ j (t)|dt,    l=m (k+m)ω j=1

(3.9)

converges to zero. By the Cauchy convergence principle (see Halmos, 1950), there are Lebesque integrable functions v j (t), j = 1, . . . , n, defined in 0 ≤ t < ∞ such that lim

n 


m→∞

j=1

(k+1)ω

|u j (t + mω) − v j (t)|dt = 0

(3.10)

kω

for all positive integer k. Thus, we can select a subsequence mk such that lim u j (t + mk ω) = v j (t)

mk →∞

(3.11)

for almost all t ∈ [0, ∞) with respect to the Lebesgue measure (see Halmos, 1950). It is clear by equation 3.10 that vi (t + ω) = vi (t) when t > 0. By extending vi (t) periodically to −∞ < t < 0, we obtain a periodic function vi (t). Now we prove that vi (t) is a periodic solution of the dynamical system 1.3. By equation 3.10 and the Fatou lemma (see Halmos, 1950), for any t1 > 0, t2 > 0, we have 

t2

lim

m→∞ t 1

|g(u j (t + mω)) − g(v j (t))|dt 

≤ G j lim

t2

m→∞ t 1

 lim

m→∞ t 1

t2

|u j (t + mω) − v j (t)|dt = 0

(3.12)

| f (u j (t + mω)) − f (v j (t))|dt 

≤ F j lim

m→∞ t 1

t2

|u j (t + mω) − v j (t)|dt = 0

(3.13)

On a Large Class of Delayed Neural Networks





∞

|ds K¯ i j (s)|

lim

m→∞ 0



t1 ∞

≤

t2

| f (u j (t + mω − τi j − s)) − f (v j (t − τi j − s))|dt 

|ds K¯ i j (s)| lim

t2

m→∞ t 1

0

955

| f (u j (t + mω − τi j − s))

− f (v j (t − τi j − s))|dt = 0.

(3.14)

Let t1 > 0, t2 > 0 be two points where equation 3.11 holds. Then, vi (t1 ) − vi (t2 ) = lim {ui (t1 + mk ω) − ui (t2 + mk ω)} mk →∞

 = lim

t1

mk →∞ t 2

+

n 
j=1



t1

=

∞



−di (t)ui (t + mk ω) +



f j (u j (t + mk ω − τi j − s))ds K i j (t, s) + Ii (t) dt

0



−di (t)vi (t) +

n


a i j (t)g j (v j (t))

j=1 n 
j=1

a i j (t)g j (u j (t + mk ω))

j=1

t2

+

n


∞



f j (v j (t − τi j − s))ds K i j (t, s) + Ii (t) dt.

(3.15)

0

Therefore, vi (t) is absolutely continuous on the set where equation 3.11 is valid. Modifying vi (t) at those points t where equation 3.11 might be invalid (a set of Lebesgue measure 0) by continuity, we conclude that every vi (t) is absolutely continuous and n
dvi (t) a i j (t)g j (v j (t)) = −di (t)vi (t) + dt j=1

+

n 
j=1

∞

f j (v j (t − τi j − s))ds K i j (t, s) + Ii (t).

(3.16)

0

In summary, v(t) = [vi (t), . . . , vn (t)] is a periodic solution of the dynamical system 1.3. Now we will prove that for every solution u(t) = [ui (t), . . . , un (t)], we have lim ui (t) = vi (t).

t→∞

(3.17)

956

T. Chen, W. Lu, and G. Chen

In fact, define ui∗∗ (t) = ui (t) − vi (t), g ∗∗ j (u j (t)) = g j (u j (t)) − g j (v j (t)), f j∗∗ (u j (t)) = f j (u j (t)) − f j (v j (t)), i = 1, . . . , n, and construct a Lyapunov function, L ∗1 (t) =

n


ξi |ui∗∗ (t)| +

i=1

N


 ξi

i, j=1

∞



0

t

t−τi j −s

| f j∗∗ (u j (y))|dy|d K¯ i j (s)|. (3.18)

∗ Replacing ui (t + ω) − ui (t) by ui∗∗ (t), g ∗j (u j (t)) by g ∗∗ j (u j (t)), f j (u j (t)) by f j∗∗ (u j (t)), by using the same argument in proving equation we can prove

lim [ui (t) − vi (t)] = 0.

(3.19)

t→∞

In the following, we investigate the exponential stability of the delayed networks 1.3 and 1.4. We prove two theorems. Theorem 2. Suppose thatα > 0 , G i , Fi > 0 , gi (·) ∈ Li p(G i ) and is nondecreas∞ ing, f i (·) ∈ Lip(Fi ), and 0 e αs |d K¯ i j (s)| < ∞, i, j = 1, 2, . . . , n. If there are positive constants ξ1 , . . . ,ξn such that ξ j (−d j + α) + ξ j a j j (t) +


i= j

+

n
i=1

 ξi

∞

+ ξi |a i∗j |

e α(s+τi j ) |d K¯ i j (s)|F j ≤ 0

Gj j = 1, . . . , n,

(3.20)

i0

then the dynamical system 1.3 has a unique periodic solution v(t) = [v1 (t), . . . , vn (t)], and for any solution u(t) of equation 1.3, there hold ||u(t + jω) − v(t)||{ξ,1} = O(e − jωα ).

(3.21)

Proof. Let ui∗ (t) = ui (t + ω) − ui (t), gi∗ (ui (t)) = gi (ui (t + ω)) − gi (ui (t)) f i∗ (ui (t)) = f i (ui (t + ω)) − f i (ui (t)). Direct calculations lead to the following equations: 
d{e αt u∗ (t)} αt a i j g ∗j (u j (t)) = e −di (t)ui∗ (t) + αui∗ (t) + dt j  n  ∞
∗ f j (u j (t − τi j − s))ds K i j (t, s) i = 1, . . . , n. + j=1

0

On a Large Class of Delayed Neural Networks

957

Define a Lyapunov function by L 2 (t) =

n


ξi |e αt ui∗ (t)|

i=1

+

n


 ξi

∞



t−τi j −s

0

i, j=1

t

e α(s+y+τi j ) | f j∗ (u j (y))|dy|d K¯ i j (s)|.

Differentiating L 2 (t), we have

L˙ 2 (t) =

n


 ξi sign{ui∗ (t)}e αt

−di (t)ui∗ (t) + αui∗ (t)

i=1

+

n


a i j g ∗j (u j (t))

+

n 


j=1

+



− τi j − s))ds K i j (t, s)

e αt | f j∗ (u∗j (t − τi j − s))||d K¯ i j (s)|

0

i, j=1

≤

∞

ξi

 f j∗ (u j (t

e α(s+t+τi j ) | f j∗ (u j (t))||d K i j (s)|

0



n


n


∞

ξi

i, j=1

−

0

j=1

n


∞

 ξi e

αt

(−di + α)|ui∗ (t)| + a ii (t)sign{ui∗ (t)}gi∗ (ui (t))

i=1

+ sign{ui∗ (t)}

+

n


|

n
j=1, j=i

f j∗ (u j (t))|



j=1

=

n


|a i∗j |g ∗j (u j (t)) 

∞

e

α(s+τi j )

|d K¯ i j (s)|

0

 e

αt

ξ j (−d j + α)|ui∗ (t)|

j=1

+ ξ j a j j (t) +


i= j

+

n
i=1

ξi |a i∗j |

ξi | f j∗ (u j (t))|





∞

e 0

sign{u∗j (t)}g ∗j (u j (t)) α(s+τi j )

|d K¯ i j (s)|

958

T. Chen, W. Lu, and G. Chen n


≤



 +
ξi |a i∗j | G j ξ j (−d j + α) + ξ j a j j (t) + i= j

j=1

+

n




∞

ξi F j 0

i=1



e α(s+τi j ) |d K¯ i j (s)| e αt |u∗j (t)| ≤ 0.

Therefore, L 2 (t) is bounded, and n


ξi |ui (t + ω) − ui (t)| = O(e −αt ).

(3.22)

i=1

Now define a function v(t) = [v1 (t), . . . , vn (t)]T by vi (t) = lim ui (t + jω). j→∞

Because of ui (t + jω) = ui (t) +

j


{ui (t + kω) − ui (t + (k − 1)ω)}

k=1

and equation 3.22, v(t) is well defined and is a periodic function with period ω. Moreover, ||u(t + jω) − v(t)||{ξ,1} = O(e − jωα )

when

j → ∞.

Theorem 3. Suppose that G i , Fi > 0, gi (·) ∈ Li p(G i ) and is nondecreasing, ∞ f i (·) ∈ Li p(Fi ), 0 e αs |d K¯ i j (s)| < ∞, i, j = 1, 2, . . . , n. If there are positive constants ξ1 , . . . , ξn and α > 0 such that for i = 1, 2, . . . , n, −(di − α)ξi + ξi {a ii (t)}+ +


j=i

+

n
j=1

 ξj Fj

∞

|a i∗j |G j ξ j

∗

e α(s+τi j ) |d K¯ i j (s)| ≤ 0.

(3.23)

0

Then the dynamical system 1.4 has a unique ω-periodic solution v(t) = [v1 (t), . . . , vn (t)], and for any solution u(t) = [u1 (t), . . . , un (t)] of equation 1.4, there hold ||u(t + jω) − v(t)||{ξ,∞} = O(e −jωα )

j → ∞.

(3.24)

On a Large Class of Delayed Neural Networks

959

Proof. Let C = C([−∞, 0], Rn ) be the Banach space of continuous functions, which map [−∞, 0] into Rn with norm ϕ =

sup ϕ(θ){ξ,∞} .

(3.25)

−∞≤θ ≤0

Define zi (t) = [ui (t + ω) − ui (t)], gi∗ (zi (t)) = gi (ui (t + ω)) − gi (ui (t)), f i∗ (zi (t)) = f i (ui (t + ω)) − f i (ui (t)), and M1 (t) = sup e α(t−s) z(t − s){ξ,∞} . 0≤s≤∞

It is clear that e α(t) z(t){ξ,∞} ≤ M1 (t) and M1 (t) is nondecreasing. In the following, we will prove that M1 (t) = M1 (0) for all t > 0. In fact, for every t0 > 0, there are two possible cases: Case 1. e αt0 z(t0 ){ξ,∞} < M1 (t0 ). In this case, M1 (t) is not increasing at t = t0 . Case 2. e αt0 z(t0 ){ξ,∞} = M1 (t0 ). In this case, let i t0 . be an index such that   αt e 0 zi (t0 ) = e αt0 z(t0 ){ξ,∞} = M1 (t0 ). ξi−1 t0 t 0

Then we have 

d|e αt zit (t)| dt  × +

 t = t0

= sign(zit0(t0 )e αt0

     −αzit0 (t0 ) dit0 uit0 (t0 + ω) − dit0 uit0 (t0 )

n


a it0 j (t0 )g ∗j (z j (t0 ))

j=1

+

n 
j=1

∞

f j∗

0

   z j t0 − τit0 j (t0 ) − s d K it0 , j (t0 , s)



 n    
   a ∗ G j |z j (t0 )| ≤ e αt0 − dit0 − α + a i+t it (t0 ) zit0 (t0 ) + i t0 j 0 0  j=i t0

+

n
j=1

 Fj 0

∞

     z j t0 − τi j (t0 ) − s d K¯ i , j (s) t0 t0 

960

T. Chen, W. Lu, and G. Chen

       αt0 e zi (t0 ) ≤ − dit0 − α ξit0 + ξit0 a i+t it (t0 ) ξi−1 t0 t0 0 0  n


+

    αt e 0 z j (t0 ) ξ j a i∗t j G j ξ −1 j 0

j=i t0

+

n


 ξj Fj

  α(t0 −τi j (t0 )−s) t0 e ξ −1 z j (t0 − τit0 j (t0 ) − s) j

∞

0

j=1

    × e α(s+τit0 j (t0 )) d K¯ it0 , j (s)  

n 
 ∗  + ≤ −(dit0 − α)ξit0 + ξi0 a it it (t0 ) + ξ j a it j G j e αt0 z(t0 ){ξ,∞} 0 0 0  j=i t0

+

n


 ξj Fj

  e α(s+τit0 j (t0 )) d K¯ i0 , j (s)

∞

0

j=1

    α(t0 −τi j (t0 )−s) t0 z(t0 − τit0 j (t0 ) − s){ξ,∞} × e  

n
   − dit0 − α ξit0 + ξit0 a i+t it (t0 ) + ξ j a i∗t j G j

≤



0 0

+

n


ξj Fje

j=1

ατi∗t

0

 j

∞

e 0



αs 

j=i t



0

   d K¯ it0 , j (s) M1 (t0 ) ≤ 0.

(3.26)

Therefore, M1 (t) is not increasing at t = t0 . In summary, we have M1 (t) = M1 (0) for all t > 0. Therefore, u(t + ( j + 1)ω) − u(t + jω){ξ,∞} ≤ e − jα sup u(s + ω) − u(s). s∈(−∞,0]

(3.27) Similar to the proof of the previous theorem, define a periodic function v(t) = [v1 (t), . . . , vn (t)]T by vi (t) = lim ui (t + jω). j→∞

We know that v(t) is well defined and ||u(t + jω) − v(t)||{ξ,∞} = O(e − jωα ) when

j → ∞.

On a Large Class of Delayed Neural Networks

961

As a special case, let ds K i j (t, 0) = b i j (t), and ds K i j (t, s) = 0, for s = 0. Then dynamical systems 1.3 and 1.4 reduce to n
dui (t) a i j (t)g j (u j (t)) = −di (t)ui (t) + dt j=1

+

n


b i j (t) f j (u j (t − τi j )) + Ii (t) i = 1, . . . , n

(3.28)

j=1

and n
dui (t) a i j (t)g j (u j (t)) = −di (t)ui (t) + dt j=1

+

n


b i j (t) f j (u j (t − τi j (t))) + Ii (t) i = 1, . . . , n,

(3.29)

j=1

respectively. It should be noted that model 3.28 has been investigated in detail by Zhou, Liu, and Chen (2004) in terms of L 2 norm and Zheng and Chen (2004) in terms of L p norm. In this special case, d K¯ i j (0) = |b i∗j |, d K¯ i j (s) = 0 for s = 0 and  ∞ ∗ ∗ e α(s+τi j ) |d K¯ i j (s)| = e ατi j |b i∗j |. 0

Therefore, as consequences of theorems 2 and 3, we obtain following two theorems: Theorem 4. Suppose that G i , Fi > 0 , gi (·) ∈ Li p(G i ) and is nondecreasing, f i (·) ∈ Li p(Fi ), i = 1, 2, . . . , n. If there are positive constants ξ1 , . . . , ξn such that  +
−ξ j (d j − α) + G j ξ j a j j (t) + ξi |a i∗j | i= j

+ Fj

n
i =1

ξi e ατi j |b i∗j | ≤ 0 ,

j = 1, . . . , n,

(3.30)

j = 1, . . . , n,

(3.31)

or in particular, −ξ j (d j − α) + G j

n
i =1

+ Fj

n
i =1

ξi |a i∗j |

ξi e ατi j |b i∗j | ≤ 0,

962

T. Chen, W. Lu, and G. Chen

then the dynamical system, 3.28, has a unique periodic solution v(t) = [v1 (t), . . . , vn (t)], and for any solution u(t) = [u1 (t), . . . , un (t)] of that equation: ui (t) − vi (t) = O(e −αt ) i = 1, . . . , n.

(3.32)

Theorem 5. Suppose that G i , Fi > 0, for all i = 1, 2, . . . , n. g(x) = (g1 (x), . . . , gn (x))T , where gi (·) ∈ Li p(G i ) is nondecreasing, f (x) = ( f 1 (x), . . . , f n (x))T , where f i (·) ∈ Li p(Fi ). If there are positive constants ξ1 , . . . , ξn and α > 0 such that
−(di − α)ξi + ξi a ii + (t) + ξ j |a i∗j |G j j=i

+

n


∗

ξi e ατi j |b i∗j |F j ≤ 0

i = 1, 2, . . . , n,

(3.33)

j=1

in particular, −(di − α)ξi +

n


|a i∗j |G j ξ j +

j=1

n


∗

ξ j e ατi j |b i∗j |F j ≤ 0 i = 1, 2, . . . , n,

j=1

(3.34) then the dynamical system 3.29 has a unique ω-periodic solution v(t) = [v1 (t), . . . , vn (t)], and for any solution u(t) = [u1 (t), . . . , un (t)] of that equation, ui (t) − vi (t) = O(e −αt ) i = 1, . . . , n.

(3.35)

Suppose that ds K i j (t, s) = b i j (t)ki j (s)ds. Then dynamical systems 1.3 and 1.4 reduce to following dynamical systems with distributed delays: n
dui (t) a i j (t)g j (u j (t)) = −di (t)ui (t) + dt j=1

+

n




∞

b i j (t)

ki j (s) f j (u j (t − τi j − s)) ds + Ii (t)

(3.36)

0

j=1

and n
dui (t) a i j (t)g j (u j (t)) = −di (t)ui (t) + dt j=1

+

n
j=1

respectively.



∞

b i j (t) 0

ki j (s) f j (u j (t − τi j (t) − s)) ds + Ii (t),

(3.37)

On a Large Class of Delayed Neural Networks

963

In this special case, d K¯ i j (s) = |b i∗j ||ki j (s)|, and 

∞

0

∗ e α(s+τi j ) |d K¯ i j (s)| = |b i∗j |



∞

∗

e α(s+τi j ) |ki j (s)| ds.

0

Therefore, as consequence of theorem 1, we have: Theorem 6. Suppose that G i , Fi > 0, for all i = 1, 2, . . . , n, g(x) = (g1 (x), . . . , gn (x))T , where gi (·) ∈ Li p(G i )  is nondecreasing, f (x) = ∞ ( f 1 (x), . . . , f n (x))T , where f i (·) ∈ Li p(Fi ), and 0 s|ki j (s)|ds < ∞. If there are positive constants ξ1 , . . . , ξn such that  −ξ j d j + ξ j a j j (t) +


i= j

+ Fj

n


ξi |b i∗j |



∞

+

ξi |a i∗j |

Gj

|ki j (s)| ds < 0,

j = 1, . . . , n,

(3.38)

0

i=1

then the dynamical system 3.36 has a unique periodic solution v(t) = [v1 (t), . . . , vn (t)], and for any solution u(t) = [u1 (t), . . . , un (t)] of that equation, lim ui (t) = vi (t) i = 1, . . . , n.

(3.39)

j→∞

Similarly, the following two theorems can be proven as consequences of theorems 2 and 3, respectively: Theorem 7. Suppose that G i , Fi > 0 , gi (·) ∈ Li p(G i ) and is nondecreasing, ∞ f i (·) ∈ Li p(Fi ), i = 1, 2, . . . , n, and 0 s|ki j (s)| ds < ∞. If there are positive constants ξ1 , . . . , ξn such that −ξ j (d j − α) + ξ j a j j (t) +


i= j

+ Fj

n
i=1

ξi |b i∗j |



∞

+ ξi |a i∗j |

Gj

e α(s+τi j ) |ki j (s)|ds < 0,

j = 1, . . . , n,

(3.40)

0

then the dynamical system 3.36 has a unique periodic solution v(t) = [v1 (t), . . . , vn (t)] and, for any solution u(t) = [u1 (t), . . . , un (t)] of that equation, there hold ui (t) − vi (t) = O(e −αt ) i = 1, . . . , n.

(3.41)

964

T. Chen, W. Lu, and G. Chen

Theorem 8. Suppose that G i , Fi > 0 , gi (·) ∈ Li p(G i ) and is nondecreasing, ∞ f i (·) ∈ Li p(Fi ), i = 1, 2, . . . , n, and 0 s|ki j (s)|ds < ∞. If there are positive constants ξ1 , . . . , ξn such that −(di − α)ξi + ξi a ii + (t) +


j=i

+

n


ξ j F j |b i∗j |



j=1

∞

ξ j |a i∗j |G j

∗

e α(s+τi j ) |ki j (s)|ds ≤ 0 i = 1, 2, . . . , n,

(3.42)

0

then the dynamical system 3.37 has a unique periodic solution v(t) = [v1 (t), . . . , vn (t)], and for any solution u(t) = [u1 (t), . . . , un (t)] of that equation, there hold ui (t) − vi (t) = O(e −αt ) i = 1, . . . , n.

(3.43)

Remark. Because constants can be regarded as ω-periodic functions with any period ω > 0, all the theorems apply to dynamical systems when di (t), a i j (t), b i j (t), and Ii (t) are constants. The details are omitted. It is natural to ask whether the conditions −ξ j d j + ξ j a j j +




+ ξi |a i j |

Gj

i= j

+

N


 ξi

∞

|d K¯ i j (s)|F j < 0

j = 1, . . . , n

(3.44)

0

i=1

can guarantee exponential convergence of the dynamical system 1.3. The answer is no. Let us consider the following delayed differential equation,  u(t) ˙ = −u(t) +

∞

u(t − s)dK (s),

(3.45)

0

where dK (τ ) = rewritten as

1 2

and dK (t) = 0 if t = τ . The differential equation can be

1 u(t) ˙ = −u(t) + u(t − τ ). 2

(3.46)

The solution of the differential equation is u(t) = e −αt ,

(3.47)

On a Large Class of Delayed Neural Networks

965

where α satisfies e ατ = 2(1 − α)

(3.48)

τ = log{2(1 − α)}/α.

(3.49)

or

If τ → ∞, then α → 0. It means that there exists no such fixed β that u(t) = O(e −βt ) holds for any delay τ . 4 A Numerical Experiment In this section, we present a numerical experiment to verify our main results. Consider the following delayed neural networks with two neurons: u˙ 1 = −2u1 (t) − 2 tanh(u1 (t)) + sin2 (t) tanh(u2 (t)) + | sin(2t)|arctg(u1 (t − 1)) +

1 cos2 (t)arctg(u2 (t − 2)) 2

+ sin(2t) + 3 u˙ 2 = −u2 (t) + 2 cos2 (t) tanh(u1 (t)) − tanh(u2 (t)) 1 2 sin2 (t)arctg(u1 (t − 1)) + | cos(t)|arctg(u2 (t − 2)) 3 3 + cos(t) + 4, +

(4.1)

where d1 = 2 ∗ b 11 =1

∗ ∗ d2 = 1 a 11 = −2 a 22 = −1 a 12 = 1 a 21 = 2 ∗ b 12 =

1 2

∗ = b 21

2 3

∗ = b 22

1 3

G 1 = G 2 = F1 = F2 = 1.

With ξ1 = ξ2 = 1, we have 2 1 = − <0 3 3 1 1 1 = −1 + [−1 + 1]+ + + = − < 0. 2 3 6

∗ + ∗ ∗ ] + b 11 + b 21 = −2 + [−2 + 2]+ + 1 + − d1 + [−a 11 + a 21 ∗ + ∗ ∗ − d2 + [−a 12 + a 22 ] + b 12 + b 22

By theorem 2 or theorem 4, we conclude that there exists a unique periodic solution that is globally exponentially stable. Figure 1 indicates that u1 (t) u2 (t) all converge to periodic trajectories, respectively.

966

T. Chen, W. Lu, and G. Chen

Figure 1: Dynamical behavior of two components.

Instead, if we hope to use the results given in Zheng and Chen (2004) or Zhou et al. (2004), we need to prove that the following matrix, P =

d1

∗ ∗ ∗ ∗ −[a 12 + a 22 + b 12 + b 22 ]

2 − 17 3 = , − 17 1 6

∗ ∗ ∗ ∗ −[a 11 + a 21 + b 11 + b 21 ]



d2

is an M matrix. It is easy to see that P is not an M matrix. Therefore, by the results given in Zheng and Chen (2004) and Zhou et al. (2004), we cannot conclude that this system has a unique periodic solution. 5 Conclusion In this article, we investigate dynamical behaviors of a large class of delayed neural networks with periodic self-inhibition, interconnection weights, and inputs. We prove that if the activation functions are of Lipschitz type and some set of inequalities, for example, equation 3.1, is satisfied, the delayed system has a unique periodic solution, which is stable globally. We also prove that if the set of inequalities 3.20 or 3.23 is satisfied, then the system is exponentially stable globally. This class of delayed dynamical

On a Large Class of Delayed Neural Networks

967

systems provides a general framework for many delayed dynamical systems. As a special case, it includes delayed Hopfield neural networks and cellular neural networks as well as distributed delayed neural networks with periodic self-inhibition, interconnection weights, and inputs as special cases. Therefore, it is easy to derive corresponding results for these neural networks from the results we have obtained. We also compare the results with those given in other articles by a numerical experiment. Moreover, the discussion also applies to delayed systems with constant interconnection weights and inputs. Acknowledgments We are grateful for the reviewers, comments, which were very helpful in the revision of the article. This project is supported by NSF of China under grant 60374018. G.C. is supported by the Hong Kong Research Grants Council under CERG grant City U 1018/0lE.

References Amari, S. (1971). Characteristics of randomly connected threshold element networks and neural systems. Proc. IEEE, 59, 35–47. Amari, S. (1972). Characteristics of random nets of analog neuron-like elements. IEEE Trans., SMC-2, 643–657. Belair, J. (1993). Stability in a model of a delayed neural network. Journal of Dynamical Differential Equations, 5, 607–623. Belair, J., Campbell, S. A., & van den Driessche, P. (1996). Stability and delay-induced oscillations in a neural network model. SIAM Journal of Applied Mathematics, 56, 245–255. Cao, Y. J., & Wu, Q. H. (1996). A note on stability of analog neural networks with time delays. IEEE Transactions on Neural Networks, 7, 1533–1535. Cao, J. (2000). On exponential stability and periodic solution of CNNs with delay. Physics Letters A, 267, 312–318. Cao, J. D., & Zhou, D. (1998). Stability analysis of delayed neural networks. Neural Networks, 11, 1601–1605. Chen, T. P. (2001). Global exponential stability of delayed Hopfield neural networks. Neural Networks, 14(8), 977–980. Chen, T. P., & Amari, S. (2001). Stability of asymmetric Hopfield networks. IEEE Transactions on Neural Networks, 12(1), 159–163. Chen, T. P., Lu, W. L., & Amari, S. (2002). Global convergence rate of recurrently connected neural networks. Neural Computation, 14(12), 2947–2957. Cohen, M. A., & Grossberg, S. (1983). Absolute stability and global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactions on Systems, Man and Cybernetics, SMC-13, 815–821. Gopalsamy, K., & He, X. (1991). Stability in asymmetric Hopfield nets with transmission delays. Phys. D, 76, 344–358.

968

T. Chen, W. Lu, and G. Chen

Hale, K. (1965). Sufficient conditions for stability and instability of autonomous functional-differential equations. Journal of Differential Equations, 1, 452–482. Halmos P. R. (1950). Measure theory. New York: Van Nostrand. Hirsch, M. W. (1989). Convergent activation dynamics in continuous time networks. Neural Networks, 2, 331–349. Hopfield, J. J., & Tank, D. W. (1984). Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Nat. Acad. Sci., 79, 3088–3092. Hopfield, J. J., & Tank, D. W. (1986). Computing with neural circuits: A model. Science, 233, 625–633. Lu, W., & Chen, T. (2003). New conditions on global stability of Cohen-Grossberg neural networks. Neural Computation, 15(5), 1173–1189. Marcus, C. M., & Westervelt, R. M. (1989). Stability of analog neural networks with delay. Physics Review A, 39, 347–359. Matsuoka, K. (1992). Stability conditions for nonlinear continuous neural networks with asymmetric connection weights. Neural Networks, 5, 495–500. Michel, A. N., & Gray, D. L. (1990). Analysis and synthesis of neural networks with lower block triangular interconnecting structure. IEEE Trans. Circuits Syst., 37, 1267–1283. Miller, K. (1965). Asymptotic behavior of nonlinear delayed-differential equations. Journal of Differential Equations, 1, 293–305. Sompolinsky, H., & Crisanti, A. (1988). Chaos in random neural networks. Physical Review Letter, 61, 259–262. Wang, L., & Zou, X. (2002). Exponential stability of Cohen-Grossberg neural networks. Neural Networks, 15, 415–422. Wilson, H. R., & Cowan, J. D. (1972). Excitatory and inhibitory interactions in localized populations of model neurons. Biophysical J., 12, 1–24. Zhang, J., & Jin, X. (2000). Global stability analysis in delayed Hopfield neural models. Neural Networks, 13(7), 745–753. Zheng, Y., & Chen, T. (2004). Global exponential stability of delayed periodic dynamical systems. Physics Letters A, 322(5–6), 344–355. Zhou, J. Liu, Z., & Chen, G. (2004). Dynamics of delayed periodic neural networks. Neural Networks, 17(1), 87–101.

Received March 26, 2003; accepted August 11, 2004.

LETTER

Communicated by Kechen Zhang

A Subjective Distance Between Stimuli: Quantifying the Metric Structure of Representations Dami´an Oliva [email protected] Laboratorio de Neurobiolog´ıa de la Memoria, Departamento de Fisiolog´ıa, Biolog´ıa Molecular y Celular, Facultad de Ciencias Exactas y Naturales, Universidad de Buenos Aires, Ciudad Universitaria, 1428, Buenos Aires, Argentina

In´es Samengo [email protected] Centro At´omico Bariloche, San Carlos de Bariloche, R´ıo Negro, Argentina

Stefan Leutgeb [email protected] Centre for the Biology of Memory, Norwegian University of Science and Technology, NO-7489 Trondheim, Norway

Sheri Mizumori [email protected] Psychology Department, University of Washington, Seattle 98195-1525, U.S.A.

As subjects perceive the sensory world, different stimuli elicit a number of neural representations. Here, a subjective distance between stimuli is defined, measuring the degree of similarity between the underlying representations. As an example, the subjective distance between different locations in space is calculated from the activity of rodent’s hippocampal place cells and lateral septal cells. Such a distance is compared to the real distance between locations. As the number of sampled neurons increases, the subjective distance shows a tendency to resemble the metrics of real space.

1 How Different Are Two Stimuli Perceived? Conside a subject that is labeling the elements of a given set of stimuli S = {s 1 , s 2 , . . . , s N }. Every time a stimulus s j ∈ S is shown, he or she identifies it as s k ∈ S, where j may or may not be equal to k. Successful trials are those where the stimulus is correctly identified, that is, when j = k. By writing down the succession of presented stimuli and, in each case, the response of the subject, one can build up a list of pairs (s j , s k ), where the first element, Neural Computation 17, 969–990 (2005)

© 2005 Massachusetts Institute of Technology

970

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

s j , is the real stimulus, and the second one, s k , is the choice made by the subject. Notice that by looking into the list of pairs, one cannot determine precisely what the subject actually perceived, since only the final choice sk is accessible. The subject may in fact hesitate to identify the stimulus as a member of S, or even think that it does not truly match any of the s i . However, even if the mental representation elicited by stimulus s j is unknown, the researcher can assess that under the requirement to classify the stimulus as an element of S, the subject chooses s k . In a way, whatever the neural activity brought about by s j , out of all the elements in S, the one whose representation is most similar to the actual one is s k . The object of this work is to provide a quantitative measure of such a criterion of similarity. The approach does not rely on a model of mental representations; it only makes use of the statistics of actual and chosen stimuli. In what follows, we assume that a sufficiently large number of samples has been taken, so that the conditional probability Q(s k |s j ) of showing s j and perceiving s k may be evaluated for all j and all k. The matrix Q is henceforth called the confusion matrix. The elements of matrix Q are positive numbers, ranging from 0 to 1. In addition, normalization must hold,


Q(s k |s j ) = 1.

(1.1)

k

It should be noticed that Q need not be symmetrical. For any fixed j, one can define an N-dimensional vector q j such that its kth component is equal to Q(s k |s j ). The positivity of the elements of Q and the normalization condition, equation 1.1, determine a domain D, where q j can live. It is a finite portion of a hyperplane of dimension N − 1. Figure 1 depicts the domain D for N = 3. For some sets of stimuli, the confusion matrix may show clustering. That is, choosing a convenient ordering of the stimuli, Q may show a block structure, where cross-elements between stimuli belonging to different blocks are always zero. In this case, the stimuli belonging to different blocks are never confounded with one another. Moreover, they are never confounded with a third common stimulus. The phenomenon of clustering exposes a very particular structure perceived by the subject in the set of stimuli. In this work, we are interested in studying the statistics of mistakes. More specifically, if the subject happens to systematically confound, for example, two of the stimuli, it could be argued that from his or her subjective point of view, those two stimuli are particularly similar. We would like to quantify such an amount of similarity by introducing a distance between stimuli. This distance, being defined with the statistics of mistakes, is of course subjective. It may happen that in a particular experiment, the subjective distance between stimuli can actually be explained in terms of some physical parameter qualifying the stimuli (e.g., orientation angle, pitch, color). But it could

A Subjective Distance Between Stimuli

q3

971

j

1

1

q2

j

1 q1

j

Figure 1: Domain D where the vector q j can exist, when N = 3.

also happen that the subjective metric structure had no physical correlate in the stimuli themselves, but instead depended on the previous semantic knowledge of the subject, or on the presence or absence of distractors, or on the statistical distribution with which the stimuli are presented, or on the attention being paid by the subject. The aim, hence, of defining a subjective distance between stimuli is to provide a quantitative measure that may serve to determine the degree up to which these different processes— if present—contribute to the confusion of some stimuli with others or, in contrast, to their clear differentiation. In order to define a distance between stimuli, it is necessary to have a notion of equality. Here, two stimuli i and j are considered subjectively equal if qi = q j . That is, if for all k, Q(s k |s i ) = Q(s k |s j ). Hence, in the notion of equality, not only the way stimulus i is confounded with stimulus j is relevant. One must also compare the way each of those two stimuli are confounded with the rest of the elements in set S. If one of them is perceived as very similar to a third stimulus k but the other is not, then a noticeable

972

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

difference between i and j can be pointed out, and the stimuli cannot be considered subjectively equal. The equality of qi and q j , in addition, is not equivalent to a high confusion probability between the two of them. If stimulus i is always perceived as stimulus j and vice versa, then, taking s i and s j as representing the first and second components, respectively, in the q vectors, (qi )t = (0, 1, 0, . . . , 0), whereas (q j )t = (1, 0, 0, . . . , 0). In this case, the two stimuli are perfectly distinguishable from one another. The fact that the subject chooses to label stimulus i as j (and vice versa) does not mean he or she confuses them. It is only a question of names. Correspondingly, it may happen that two stimuli are never confounded with one another, and yet they are equal. This happens when Q(s i |s j ) = Q(s j |s i ) = Q(s i |s i ) = Q(s j |s j ) = 0, and in addition, Q(s k |s i ) = Q(s k |s j ), for all k different from i and j. Starting from the notion of subjective equality, in the next section a number of desirable properties of a subjective distance are discussed. Out of all the distances that fulfill these requirements, a single one is selected in section 3. In section 4, the relationship of our subjective distance to other measures of similarity is discussed. Section 5 extends the definition of subjective distance to the case where the response of the subject is given as a neural pattern of activity. In section 6, an example is presented using extracellular recordings from the rodent hippocampus and lateral septum. Finally, in section 7, a brief summary of the main ideas and results is given. 2 Properties of a Subjective Distance What are the desirable properties of a subjective distance? First, since the distance D between elements i and j is intended to reflect the statistics of confusion on presentation of these two stimuli, it is convenient to define it in terms of the vectors qi and q j . As a distance, it is required to fulfill the following conditions: 1. D(qi , q j ) ≥ 0, and D(qi , q j ) = 0 ⇔ qi = q j . 2. D is symmetrical: D(qi , q j ) = D(q j , qi ). 3. D obeys the triangle inequality: D(qi , q j ) + D(q j , qk ) ≥ D(qi , qk ). These are general requirements, defining a distance. The third condition implies that the set of q vectors that lie all at the same distance of one particular qi conform a convex figure, in the domain D. In addition, in the case presented here, the distance between two elements should not depend on the ordering of the stimuli. Hence, if the components k and
are interchanged, in both qi and q j , the distance D(qi , q j ) should remain invariant. That is, if C k
is a matrix that interchanges the kth and
th component, then 4. D(qi , q j ) = D(C k
qi , C k
q j ).

A Subjective Distance Between Stimuli

973

This requirement, though plainly obvious from the intuitive point of view, imposes quite serious restrictions. Consider, for example, all the distances D(qi , q j ) that can be derived from a scalar product <, >, namely, D(qi , q j ) =  j >. Once an orthonormal basis is given, this may be writ< qi − q j , qi − q i j ten as D(q , q ) = (qi − q j )t M(qi − q j ), where M is a hermitian, positive definite matrix representing the scalar product. Condition 4 imposes symmetry among the components of the vectors, which means that M must be proportional to the unit matrix. Therefore, out of all the distances that have a scalar product associated with them, the only one that fulfills condition 4 is the Euclidean distance—apart from a scale factor, fixing the units. What should be the meaning of the maximum subjective distance? The maximum distance should be reserved to those pairs of objects that the subject distinguishes unambiguously from one another: that is, to those s i and s j that are never confounded with a common stimulus. Mathematically, this means that for each k, either Q(s k |s i ) or Q(s k |s j ) (or both) must vanish, that is, whenever Q(s k |s i ) = 0, Q(s k |s j ) = 0 (and vice versa). In this case, whatever the response of the subject to stimulus s i , it never coincides with his or her response to stimulus s j . This situation corresponds to the intuitive notion of unambiguous segregation: the response of the subject to stimulus i is enough to ensure that the stimulus was not j. And vice versa, the response to stimulus j is enough to discard stimulus i. The fifth requirement, hence, reads: 5. If D(qi , q j ) is maximal if and only if s i and s j are unambiguously segregated. And conversely, if D(qi , q j ) is not maximal, then s i and s j are not unambiguously segregated. Imposing requirement 5 ensures that the stimuli that are unambiguously segregated are all at the same distance, no matter any other particular characteristic of the stimuli. And conversely, if two stimuli do not elicit segregated responses, they are not allowed to be at the maximum distance. Condition 5 establishes the cases that correspond to the maximum distance, in the same way that condition 1 does to the minimum distance. Adding the triangular inequality 3 ensures that those pairs of stimuli whose distance lies in between the minimal and the maximal one be consistently ordered. Condition 5 has two important consequences. In the first place, it ensures that the clustering structure present in Q is also reflected in the matrix of distances D. In D, cross-terms between different blocks are equal not to zero but to the maximum distance. Conversely, if D shows a block structure, it may be shown that Q has the same block structure. In the second place, if the maximum distance is finite, a similarity matrix S may be defined, S = d M I − D, where d M is the maximum distance and I the unit matrix. The matrix S also inherits, if present, the clustering structure of Q. This correspondence between the clustering structure of D (and of S) with the one of Q cannot be ensured if condition 5 is not fulfilled.

974

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

 j 2 i The Euclidean distance DE (qi , q j ) = k (q k − q k ) does not fulfill condition 5. Taking into account that the q vectors are normalized (see equation 1.1), the √ maximum value of the Euclidean distance between two stimuli is 2. It can be attained, for example, for (q1 )t = (1, 0, 0, 0) and (q2 )t = (0, 1, 0, 0). In this example, in fact, stimulus 1 shares no common response with stimulus 2. However, not all stimuli with no common responses lie at the maximum Euclidean distance. Consider, for example, (q3 )t = (1/2, 1/2, 0, 0) and (q4 )t = (0, 0, 1/2, 1/2). Though showing disjoint response sets, their Euclidean distance is equal to 1, which is less than the maximum distance. This means that none of the distances that can be associated with a scalar product are useful as a measure of subjective dissimilarity. 3 Choosing a Subjective Distance There are still many distances fulfilling requirements 1 through 5. In what follows, a single one is selected on the basis of a maximum likelihood decoding. Imagine that someone observing the subject’s responses to either stimulus i or j has to guess which of the two has been presented. For the moment, we assume for simplicity that both stimuli appear with the same frequency; this requirement will be abandoned later. We assume the observer is familiar with the confusion matrix of the subject. There are several ways in which he or she can decide between stimuli i and j given the subject’s response. Here, a maximum likelihood strategy is considered, since this is the algorithm that maximizes the fraction of stimuli correctly identified. It consists of taking the choice of the subject—say, stimulus k—and deciding whether the actual stimulus was i or j on the basis of which of them has the larger qk component. If Q(s k |s i ) > Q(s k |s j ), the observer chooses stimulus i; if the opposite holds, the observer chooses stimulus j. If both conditional probabilities are equal, then the observer chooses any of the two stimuli, with equal probabilities. Under this scheme, the fraction of times the observer correctly identifies stimulus i is P(s i |s i ) =




q ki + j

k/q ki >q k

1
i q . 2 i j k

(3.1)

k/q k =q k

The fraction of times stimulus j is presented but the observer chooses stimulus i is P(s i |s j ) =




j

qk + j

k/q ki >q k

1
i q . 2 i j k k/q k =q k

(3.2)

A Subjective Distance Between Stimuli

975

Correspondingly, when the observer decides for stimulus j,


P(s j |s j ) =

j

qk +

j

k/q k >q ki




P(s j |s i ) =

j

k/q k >q ki

1
j q . 2 i j k

(3.3)

1
j q . 2 i j k

(3.4)

k/q k =q k

q ki +

k/q k =q k

The distance D(qi , q j ) is defined as the difference between the fraction of correct and incorrect maximum likelihood choices, namely, D(qi , q j ) = =

 1  1 P(s i |s i ) − P(s i |s j ) + P(s j |s j ) − P(s j |s i ) 2 2 N 1
j |q i − q k |. 2 k=1 k

(3.5)

In other words, the distance between stimulus i and stimulus j is defined in terms of the performance of the maximum likelihood decoding, assuming that the response statistics of the subject are known. This definition is easily shown to fulfill all five conditions. A distance equal to zero means that the observer is deciding at chance between the two stimuli. Given that he or she uses a maximum likelihood strategy, that means that the two underlying vectors are equal. A distance equal to 1 implies that the observer always makes the right choice. In what follows, some mathematical properties of the distance D are analyzed. In order to get a geometrical flavor of D, Figure 2A shows a contour plot of the distance of all the vectors in D to the vectors (1/3, 1/3, 1/3) and Figure 2B to the vectors (0, 0, 1). The subjective distance D is translation invariant. That is, if the vectors qi and q j are displaced by a fixed vector q, the distance between them remains unchanged. Mathematically, D(qi , q j ) = D(qi + q, q j + q).

(3.6)

Equation 3.6 is valid for any displacement q. However, in the context here, the displacement should be such that both qi + q and q j + q fall inside the domain D. This means that the components of q must sum up to zero, and its magnitude must be bounded (the value of the bound depends on the location of qi and q j ). As a further characterization, the distance D between a stimulus that is perfectly identified by the subject—say, stimulus i—and another stimulus j, N j is given. We take (qi )t = (1, 0, 0, . . . , 0). In this case, D(qi , q j ) = k=2 qk =

976

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

Figure 2: Contour plot of the distance D to a fixed vector qi , in D. (A) qi = (1/3, 1/3, 1/3). (B), qi = (1, 0, 0).

j

j

1 − q 1 . The distance between qi and q j is fully determined by q 1 . It does not matter whether the probability of not selecting stimulus 1 is spread out over the last N − 1 components of q j or is entirely concentrated in a single one. This result generalizes to any vector qi having two or more null components: the distance depends on the sum of those same components of q j , not on their individual values. Finally, consider the case where the subject has a probability α of identifying any of the stimuli correctly (Q(s i |s i ) = α), and that whenever he or she makes a mistake, the error is equally distributed among all other stimuli (Q(s i |s j ) = β, for all i = j). The normalization condition equation 1.1 implies α + (N − 1)β = 1. In this case, D(s i , s j ) = |α − β|. 3.1 Extension to Continuous Stimuli. Consider the case where there is a continuous parameter x labeling the stimuli, such that stimulus i corresponds to an interval of x values ranging from i to (i + 1)/, with  = 1/N. It is now convenient to vary i between 0 and N − 1. The scale of x is chosen in such a way that its maximum value is 1. For large N, the

A Subjective Distance Between Stimuli

977

confusion matrix can be written as Q(s j |s i ) = u( j|s i ), where u(x|s i ) is a piecewise continuous probability density. The distance between stimuli i and j reads D(qi , q j ) =

  1
 1 1  |u(x|s i )−u(x|s j )|d x, u(k|s i )−u(k|s j ) → 2 k 2 0 (3.7)

where the right-hand limit corresponds to making the number of stimuli N tend to infinity. In other words, the distance between stimuli i and j is equal to the area between the two corresponding densities. The normalization condition for u(x|s) ensures that D lies between 0 and 1. One can easily show that its value remains invariant when a different parameterization of the variable x is used—as long as the new variable is in a one-to-one relation to x. 3.2 Extension to Stimuli with Nonuniform Prior Probabilities. One could ask whether the measure D can be extended to stimuli that are not all presented with the same probability. Let Q(s i ) denote the probability of presenting stimulus s i . In this case, the maximum likelihood algorithm has to be substituted by a maximum a posteriori one. That is, the observer chooses stimulus s i upon response s k from the subject whenever pki = Q(s k |s i )Q(s i ) j j is larger than pk = Q(s k |s j )Q(s j ), and vice versa. Whenever the pki = pk , the choice between s i and s j is proportional to their corresponding priors. In this case, the difference between the correct and incorrect fractions of maximum likelihood estimations is D0 (qi , q j ) =


 1 j i p − p  k k. Q(s j ) + Q(s i ) k

(3.8)

This is, of course, also a valid distance between stimuli, since it fulfills requirements 1 through 5. Its maximum value is also 1, and it still carries the same block structure as Q. Notice that in this case, the distance not only depends on the subject’s perception, characterized by Q(s j |s i ), but also on the statistics of the stimuli (described by Q(s i )). 4 Comparison with Other Measures of Dissimilarity The distance D0 is no more than a geometrical view of the matrix Q(s i , s j ). It has the advantage of being true distance, that is, of obeying conditions 1 through 3. In addition, it fulfills the symmetry constraints imposed by condition 4 and preserves the block structure in Q, as ensured by condition 5. However, there are also other distances that still obey requirements 1 through 5; the angle between the vectors qi and q j can be taken

978

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

as an example. The advantage of D0 is that it has a simple interpretation in terms of maximum likelihood decoding performance. There have been previous attempts aiming at quantifying how different two stimuli are perceived. Perhaps the most similar to ours was proposed by Green and Swets (1966) in their definition of the discriminability d  between two stimuli. Strictly speaking, their approach can be used only when the response to different stimuli is described by gaussian functions whose mean depends on the stimulus but whose variance remains fixed. They defined the discriminability d  between two stimuli as the ratio between the difference of the two corresponding mean values to the standard deviation. Can the concept of d  be extended to more general response distributions? In the gaussian case, one can make a correspondence between a given d  value and the expected fraction of errors when estimating the stimulus from the response of the subject using a maximum likelihood decoding. In terms of this correspondence, d  not only represents the distance at which the gaussian functions sit from one another, but more generally, the fraction of decoding mistakes, which is something that does not depend on the shape of the probability distribution of the responses of the subject. Large discriminability is associated with a small probability of making a mistake. The scale of the measure, however, since it is inherited from the gaussian case, has a non linear relationship with the fraction of mistakes. With this idea in mind, d  can be extended to nongaussian stimuli (Rieke, Warland, de Ruyter van Steveninck, & Bialek, 1997). Whatever the shape of P(s k |s i ) and P(s k |s j ), given the response s k , an observer can decide in favor of stimulus s i or s j depending on which of them has a higher probability of eliciting response s k . For each pair of stimuli s i and s j , there will be, on average, a certain fraction f e of errors. One could extend the definition of discriminability to be the d  value that would give, in the gaussian case, the same fraction f e of errors. This extension, being intimately related to the performance of a maximum likelihood decoding, is grounded on the same rationale as our subjective distance D. The problem is that if one is constrained to choose the d  scale to match the equivalent gaussian case, then the definition of discriminability does not obey the triangular inequality. It is easy to construct an example where P(s k |s i ) overlaps in a certain region with P(s k |s j ), which in turn overlaps in a different region with P(s k |s
), but such that P(s k |s i ) and P(s k |s
) are never simultaneously different from zero. In this case, d  (s i , s j ) and d  (s j , s
) are both finite, whereas d  (s i , s j ) is infinite. Hence, though both D and d  can be defined in terms of maximum likelihood performance, D is a proper distance, whereas d  is not. Another well-known notion of distance can be defined (for continuous stimuli) in terms of the Fisher information metric tensor J . There is a natural scalar product associated with J and also a notion of distance in the space of stimuli (see Amari & Nagaoka, 2000). However, the entire Fisher geometry becomes meaningless for discrete stimuli. Our aim is to show that even in the discrete case, a definition of distance is possible.

A Subjective Distance Between Stimuli

979

The distance defined in terms of the Fisher metric tensor is a bilinear form. Its matrix elements are not constant, but depend on the point q that one is interested in. Hence, distances are defined in terms of a curvilinear integral, which may actually involve very difficult calculations. When two stimuli have disjoint response sets, the Fisher distance between them diverges. However, a Fisher distance between two stimuli equal to infinity does not necessarily imply that the responses to those two stimuli conform disjoint sets. The Fisher distance between two stimuli may diverge, for example, when the probability density u(x|s) is discontinuous. This implies a discrepancy with requirement 5. The Kullback-Leibler divergence (Cover & Thomas, 1991) between the vectors qi and q j can also be used to measure how different two stimuli are perceived. It has the appealing property of being intimately related to many concepts in information theory, and as such, it has an informationbased intuitive interpretation: it is a measure (in number of additional bits of the mean code length) of the inefficiency of assuming that the distribution of a given variable is qi when its true distribution is q j . It is not a distance, however, since it does not fulfill requirements 2 and 3. Its symmetrized version is sometimes called the Jensen-Shannon measure, which is still not a distance because it does not obey the triangular inequality 3. For this measure, condition 4 is always true. The maximum value of the JensenShannon divergence is infinite. This value, however, is not only reached for pairs of stimuli whose response sets are disjoint, but also whenever there j is a component k such that q ki = 0, and q k = 0. This means requirement 5 is not in general fulfilled. There has been another previous proposal of a pseudo-distance (Treves, 1997), which was also defined in terms of the confusion matrix. As opposed to D0 , and also to the Jensen-Shannon measure, the distance between stimuli s i and s j depends only on Q(s i |s j ), Q(s j |s i ), Q(s i |s i ) and Q(s j |s j ) (other stimuli do not appear). Just like the Jensen-Shannon divergence, however, it does not fulfill requirements 3 and 5. We claim that our measure allows a geometrical picture of a set of stimuli. Multidimensional scaling (Young & Hamer, 1994; Cox & Cox, 2000) also aims at this goal. The two approaches, however, bear certain differences. The use of D0 aims at a definition of a distance. It is particularly useful when one can vary a certain parameter in the experiment (in the example below, the number of cells being sampled) and wants to know the effect of that parameter in the structure of confusions. Nevertheless, D0 in itself does not provide a way of visualizing the stimuli in a particular space. Since the structure of confusions may depend on a very large number of factors (e.g., the semantic knowledge of the subject), there may be actually no small-dimensional space where the stimuli can be placed. Multidimensional scaling, in contrast, starts with a given matrix of distances, or dissimilarities. The algorithm is designed to place the stimuli in a finite dimensional space producing the minimum possible distortion of the

980

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

pairwise distances. Except for very particular sets of stimuli, it is an approximate method. What is gained is the optimal set of coordinates of the stimuli out of the matrix of distances. The subjective distance could be, in certain applications, a good starting point with which to feed the multidimensional scaling algorithm. Finally, we point out that our definition of subjective distance emphasizes the probability of confounding the stimuli, as opposed to other possible measures, that stress the differences in the elicited neural representations. In this sense, our approach is, broadly speaking, complementary to Victor and Purpura’s (1997) proposal of constructing a metric in the set of responses. There, different distances between spike trains were considered. Each proposed distance captured specific aspects of the neural response. For example, the distance between two spike trains was defined in terms of how different the timing of individual spikes was, or how different the interspike intervals were, and so forth. Their aim was to decide which of those distances could better cluster the neural responses corresponding to the different stimuli in their set and thus make inferences about the way stimuli were encoded into spike trains. In the back of this reasoning is the assumption that the stimuli themselves are all sufficiently different from one another. Here, in contrast, we are interested in the perceived distances and similarities of the stimuli, as can be deduced from the structure of confusions.

5 From Neural Representations to the Subjective Distance In order to use the distances defined in section 3, the conditional probabilities Q(s i |s j ) are needed. Such probabilities may be extracted, as described in section 1, from an experiment where the subject is asked to identify the stimuli. However, this is not the only way a matrix Q can be obtained. In the case of nonhuman subjects, many times the stimuli are presented while the activity of one or several neurons is being recorded by microelectrodes implanted in the animal’s brain. From these experiments, the conditional probability ρ(r k |s j ) of recording response r k during the presentation of stimulus s j may be calculated. One way of deriving Q from ρ is to use a decoding procedure, that is, to define a mapping going from the set of neural responses to the set of stimuli (Rolls & Treves, 1998; Dayan & Abbott, 2001; Rieke et al., 1997). Different rules defining such a transformation give rise to different decoding procedures. Among them, one can point out the maximum a posteriori k j k decoding, where  r is associated with the stimulus that maximizes ρ(s |r ) = ρ(r k |s j )P(s j )/
ρ(r k |s
)P(s
), that is, the conditional probability of having presented stimulus s j when response r k was measured. This rule is, among all possible decoding rules, the one that maximizes the fraction of correct decodings.

A Subjective Distance Between Stimuli

981

Another widely used option is the Bayesian approach, where Q(s j |s i ) =




ρ(s j |r k )ρ(r k |s i ).

(5.1)

k

Strictly speaking, in this case there is no decoding, since one does not choose a single stimulus for each response. One rather keeps all the probability distribution for each stimulus, given the neural response. The only assumption in the back of equation 5.1 is that the decoded stimulus is conditionally independent of the actual stimulus, that is, P(s j |r k , s i ) = P(s j |r k )P(r k |s i ). Passing from matrix ρ to matrix Q always means a loss of information. The amount of information that is lost can sometimes be estimated (Samengo, 2001). A maximum a posteriori decoding maximizes the fraction of correct identifications (that is, the trace f of the resulting Q) but probably loses some of the structure of ρ whenever the response r j is not the most probable response elicited by a given stimulus. A procedure like equation 5.1 is intended to preserve the statistical regularities of mistakes, but typically implies a lower fraction of correct choices f . Therefore, whenever neural responses are used to construct matrix Q, one must bear in mind that the results depend, at least partially, on how Q is calculated. The dependence of the Q matrix on the decoding may seem dangerous. Could one define a distance in terms of ρ without having to pass through matrix Q? Of course, this is possible in general terms. One option, for example, would be to calculate the mean response mi to stimulus i, and then define the distance between stimuli i and j as the distance Dn between the corresponding means.1 This is a distance between vectors living in the response set, not in D. The subindex n, hence, stands for neural. The distance Dn may be a sensible approach when one is interested in the neural representations of the stimuli. It is not very convenient, however, if the distance is meant to reflect the structure of confusions. As is shown below, Dn does not in general fulfill condition 5. Confusion is defined at the behavioral level; the subject is asked to identify the stimuli. At the neural level, one can talk about confusion between two stimuli only when a decoding procedure is introduced. In this sense, the process of decoding should not be viewed as an arbitrary step: one shifts the question of how distinguishable two stimuli are to the problem of inferring the stimulus from the neural response. To test whether a distance defined in the response set fulfills condition 5, we point out that for any reasonable decoding procedure, two stimuli that 1 Of course, even more sophisticated distances are possible, for example, one taking into account the covariance matrix, or even higher moments of the distributions ρ(r k |s i ). Here, we use only a very simple definition of a distance Dn in the response space, since we want to point out only that there is a qualitative difference between defining a distance in terms of the Q matrix and in terms of the ρ matrix.

982

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

elicit responses occupying convex, nonoverlapping portions of the response space are never confounded with one another. Is this enough to have them at the maximum Dn ? In general, no. Being Dn defined in terms of a distance in the response set, the more separate the responses to the two stimuli are, the larger Dn is. Hence, neural-based distances do not in general fulfill condition 5.

6 Application to Place Cells: Comparing the Actual and the Subjective Distance Between Two Locations in Space It is known that many of the pyramidal cells in the rodent hippocampus selectively fire when the animal is in a particular location of its environment (O’Keefe & Dostrovsky, 1971). Such a response profile allows one to define, for each cell, a place field, that is, the region in space for which the neuron is responsive. A classic experiment in the study of the rat’s hippocampal neurophysiology consists in letting the awake animal wander in a given environment while the activity of one or several of its hippocampal pyramidal cells is being recorded (for an overview see, e.g., O’Keefe & Dostrovsky, 1971, or Redish, 1999). In the experiment we analyze next, cells in the lateral septum have also been recorded. The lateral septum receives a massive projection from hippocampal pyramidal neurons. The activity of septal cells has also been shown to be informative about the location of the animal, though not as neat as in hippocampal neurons. The set of stimuli in this case consists of all the possible locations where the animal can be placed. This set is endowed with a natural metric: we know what the distance between two locations is. Other sets of stimuli, for example, a set of pictures of human faces, lack an obvious, natural distance. In order to test our definition of subjective distance, it is convenient to work with a metric set of stimuli, which allows the comparison of D0 with the natural physical distance in the set. A description of the experiment follows. Nine young adult Long-Evans rats were tested while moving in an eight-arm radial maze, as shown in Figure 3A. Each arm contained a small amount of chocolate milk in its distal part (for details, see Leutgeb & Mizumori, 2002). In a given trial, a rat initially placed in the center of the maze visited the eight arms in a random order (see Figure 3B), taking the food reward. Several trials were recorded per animal, sometimes in darkness and sometimes in light conditions. Septal and hippocampal cells were simultaneously registered while the animal moved in the maze. Single units were separated using an online and off-line separation software. Units were then classified according to their anatomical location and the characteristics of the spikes. Hippocampal pyramidal cells and lateral septal cells were identified. In what follows, each arm of the maze is taken as a different stimulus, and it is labeled by its angle, as in Figure 3A.The center of the maze was excluded so as to have eight similar

A Subjective Distance Between Stimuli

983

Figure 3: (A) Eight-arm maze where the animals move. (B) Path traveled by a rat in a given trial, as measured by the diode on its head. The rat goes straight to the end of the arm and drinks the chocolate milk. When turning around, it typically sweeps its head in a circular movement, peeping outside the maze.

and evenly visited stimuli. Thus, in the experiment, the physical distance between any two stimuli is the (actual) angle between the corresponding arms. The subjective distance is deduced from the responses. As an example, in Figure 4 the location of the animal is shown whenever a lateral septal (see Figure 4A) and a hippocampal place cell (see Figure 4B) fire a spike. It is clear that both cells fire selectively when the animal is in a particular location on the maze, the septal cell having a somewhat more distributed response.

984

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

Figure 4: Location of the animal when a (A) given lateral septal and (B) hippocampal pyramidal neuron fired a spike. The dots that fall outside the maze indicate that the nose of the animal is peeping outside the walls of the labyrinth.

The decoding procedure and the calculation of the Q matrix were carried out first for a single neuron and then for pairs, triplets, and sets of four cells measured simultaneously. The response of the animal was, correspondingly, a scalar, or a two-, three- and four-dimensional vector r, where the component rc stands for the firing rate of cell c. Hence, upon entrance to arm s j , rc was calculated as the ratio of the number of spikes fired by cell c to the time spent in the arm s j in that particular trial. To decode the arm corresponding to a given response r, the M responses most similar to r were taken into account. Those M firing rates included M0 , responses that corresponded to the animal in arm 0 degrees, M45 responsescorresponded to the animal in arm 45 degrees, and so forth. That is, M = i Mi , and the response r of the present trial is not included. The probability ρ(s j |r ) for the rat to be in arm s j when response r was observed was set as Mj /M. The

A Subjective Distance Between Stimuli

985

A 1

D0

From arm 0

o

180

o

0 1

D0

45

o

225

o

0 1

D0

90

o

270

o

0 1

D0

315 135

0 0

90

180

o

o

0

270

To arm

90

180

270

To arm

1

o

D0

From arm 180 From arm 0

0 1

D0

45

o

o

225

o

270

o

0 1

D0

90

o

0 1

D0

135

o

315

o

0 0

90

180

To arm

270

0

90

180

270

To arm

Figure 5: Matrix of distances D0 for the (A) lateral septal cell of Figure 4A, and (B) hippocampal pyramidal cell of Figure 4B.

decoded arm was the one maximizing ρ(s j |r ). If there was a draw between two arms, the decoded one was chosen at chance between those two, with probabilities that were proportional to the two corresponding priors. With this procedure, Q(s i |s j ) is defined as the fraction of times s i was decoded, whenever s j was the actual arm.

986

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

The procedure was carried out for several values of M ranging from 2 to 20. Each M gave rise to a different Q matrix. We observed that the fraction of correct decodings (the trace of Q) typically showed a maximum as a function of M. The M for which tr(Q) was maximal was taken to be the final one. With this procedure, we obtained a Q matrix for each neuron, each pair of neurons, each triplet, and each quadruplet. In what follows, the results for the subjective distance as derived from the chosen Q matrices are shown. As examples from the single neuron behavior we show, in Figure 5, the matrix D0 (qi , q j ), for the same cells of Figure 4. In each plot, the arm i is kept fixed, while the arm j is varied (it does not really matter which is i and which is j, since the distance is symmetrical). In all graphs, D0 = 0 corresponds to i = j. The plot in Figure 5A corresponds to a cell in the lateral septum. It may be seen that most arms are far away from the one at 135 degrees. Not surprisingly, this is precisely the arm where the cell fires most. If a strong, reliable response is obtained for a given stimulus and if this response differs from the response to all other stimuli, then there is little probability of misidentifying it. Among the remaining arms, the one closest to the one at 135 degrees is the one at 0 drgree, and it corresponds to the second-largest firing rate. In the case of the hippocampal cell of Figures 4B and 5B, the arm that is farthest away from all others is the one at 0 degree, the one where the place field is located. Surprisingly, however, its distinction from all other arms is not as clear as in the septal cell of Figures 4A and 5A even though Figure 4B indicates that this cell is much more selective to the location of the animal. The fact is that Figure 4 alone is not enough to depict the selectivity of the cell, because it does not show the statistics. The rat in Figures 4B and 5B entered 18 times into the arm at 0 degree, but in only half of those trials, the cell fired a burst of spikes. The other half had no response. Therefore, a burst of spikes most probably means the animal is the arm at 0 degree, but a silent response does not discard this arm. In all those trials when there was no activity upon entrance to the arm at 0 degree, this arm can well be confounded with any other arm. If all the silent entrances to the arm at 0 degree are discarded, then the matrix of distances changes drastically, as is shown in Figure 6. There, the distance from the arm at 0 degree to any other arm is shown to be equal to one, implying no mistakes at all. This shows that the trial-to-trial variability has an important influence on the distance between any two stimuli, even for those stimuli that may elicit very strong responses. Every cell has a different spatial distribution of responses and, hence, a different matrix of subjective distances. In what follows, therefore, instead of analyzing the specific characteristics of individual cells, the average behavior is studied. In Figure 7, the average distance of a given arm with all the others is shown. The numbers in the x-axis indicate the angle separating the two arms under consideration. Thus, the average D0 between all pairs of arms

A Subjective Distance Between Stimuli

987

D0

1

From arm 0

From arm 180

o

o

0

D0

1

45

o

90

o

225

o

0

D0

1

270

o

D0

0 1

135

o

315

o

0 0

90

180

To arm

270

0

90

180

270

To arm

Figure 6: Matrix of distances D0 for the cell of Figure 4B, when all the trials with silent entrances to the arm at 0 degree have been removed.

that lie at 45 degrees from one another is shown as a single point in the plot. Each data point represents an average among cells (62 units have been considered: 29 pyramidal cells in the hippocampus and 33 lateral septal cells). The error bars show the standard deviation of the cell average. In the lower curve, the arm is decoded from the activity of a single cell. As the curves rise, the decoding makes use of more cells (from 1 to 4) recorded simultaneously. The first thing that can be noticed is that as the number of neurons increases, all the subjective distances grow. This means that the representation of the different arms becomes more and more distinctive, and therefore the fraction of mistakes goes down. One can also observe that the single-neuron case looks pretty flat. That is, there is no evident structure in the set of stimuli, since each arm has roughly the same probability as being confounded with any other arm. An analysis of variance (ANOVA) of the distances obtained for pairs of arms at 45 and 180 degrees shows that they are not significantly different ( p > 0.3). In contrast, as the number of neurons increases, the curves begin to bend, showing larger distances for pairs of arms that lie farther apart. The ANOVA test shows that the distances obtained for pairs of arms at 45 degrees are significantly different from the ones at 180 degrees ( p < 0.0001). This means that if the response of four neurons is considered, then it is more probable

988

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

D0

0.5

1 cell 2 3 4

0.0 0

90 180 270 angle to neighbor

360

Figure 7: Average distance of a given arm with all the others, numbered counterclockwise, as calculated from 62 units (29 hippocampal pyramidal neurons and 33 lateral septal cells). The error bars are the standard deviations from the averages. Stars, triangles, squares, and circles correspond to 1, 2, 3, and 4 cells, respectively.

to confound a given location with a nearby location than with a distal one. Topography, hence, seems to emerge from population coding, not from the single-cell response. This example shows that the subjective distance is, when read from four simultaneously recorded cells, a monotonic function of the true distance. This indicates that near in the actual world corresponds to near in the subjective perception, and far in the actual world corresponds to far in the subjective perception. In this sense, one can assert that there is a continuous mapping between the actual and subjective spaces. If the subjective distance were strictly equal to the actual distance, however, the upper curve of Figure 7 would have a linear rise from 0 to 180 degrees, and then a similar linear decrease from 180 to 360 degrees. The curved shape of the upper trace of Figure 7 indicates that the scale in the subjective representations somehow shrinks as the actual stimulus moves farther away. This, in turn, shows that the system is better designed to make fine distinctions between nearby stimuli than between distal ones. The singlecell response instead makes no distinction at all between near or far. It

A Subjective Distance Between Stimuli

989

only recognizes whether the two arms are the same or not. All these nontrivial characteristics of the coding properties of hippocampal and septal cells have been visualized in terms of our definition of the subjective distance.

7 Summary The subjective distance as introduced here is a way of measuring how differently stimuli are perceived. The distance between any two elements may be interpreted in terms of the average performance when trying to infer the actual stimulus, if only the response of the subject is known. In such a performance, the trial-to-trial variability of the responses to each stimulus is as important as the mean responses. It should be noted that the probability of confusion depends not only on the characteristics of the two stimuli under consideration, but also on all the other stimuli in the set S. Hence, the distance between two given items may vary when the remaining stimuli in S are modified. So here, as well as in many other information or discrimination analyses the choice of the set of stimuli is a highly relevant (and sometimes difficult) issue in itself, which should not be neglected. The distances D and D0 may have several applications; for example, it may be of interest to compare the subjective distance with some other objective measure of distance. Here, as shown in section 6, D0 has proven useful to show that an increasingly topographic encoding of spatial location arises as the number of cells grows. In this case, as the population of neurons increases, the topography of real space seems to emerge. There are other cases, though, where the subjective perception of certain objects raises clear differentiating bounds between stimuli that are actually near, in the so-called physical space. For example, it is known that during the first year of life, the exposure of infants to their mother tongue builds up a very particular way of perceiving phonetic information. Two sounds that are physically very similar but correspond to two different phonemes in the child’s language are easily discriminated. Yet the researcher can design two sound waves differing even more in their physical characteristics, but that the infant cannot distinguish simply because they are not distinct building blocks (phonemes) in his or her own language (see, e.g., Kuhl, 1994). Another example is the perception of facial expression (Young et al., 1997), where although the researcher can continuously morph the picture of a happy face into that of an angry one, human observers have a tendency to categorize them into distinct emotions (full happiness or full anger). Our subjective distance would be a good way to quantify these effects. Another possible application would be to use the subjective distance as the input to a multidimensional scaling algorithm. This would allow placing

990

D. Oliva, I. Samengo, S. Leutgeb, and S. Mizumori

the stimuli in a finite-dimensional space and gaining further geometrical intuition about them. Acknowledgments We thank Alessandro Treves for his very useful suggestions. This work has been supported with a fellowship of the Alexander von Humboldt Foundation, a grant of Fundacion ´ Antorchas, one of the Human Frontier Science Program No. RG 01101998B, and NIMH grant 58755. References Amari, S., & Nagaoka, H. (2000). Methods of information geometry. New York: Oxford University Press and American Mathematical Society. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley. Cox, T. F., & Cox M. A. (2000). Multidimensional scaling, (2nd ed.). London: Chapman and Hall. Dayan, P., & Abbott L. F. (2001) Theoretical neuroscience: Computational and mathematical modeling of neural systems. Cambridge, MA: MIT Press London. Green, D. M., & Swets J. A. (1966). Signal detection theory and psychophysics. New York: Willey. Kuhl, P. K. (1994). Learning and representation in speech and language. Curr. Op. Neurobiol., 4 812–822. Leutgeb, S., & Mizumori S. J. (2002). Context-specific spatial representations by lateral septal cells. J. Neurosci., 112 (3), 655–663. O’Keefe, J., & Dostrovsky, J. (1971). The hippocampus as a spatial map: Preliminary evidence from unit activity in the freely moving rat. Brain Res., 34, 171–175. Redish, A. D. (1999). Beyond the cognitive map: From place cells to episodic memory. Cambridge, MA: MIT Press. Rieke F., Warland D., de Ruyter van Steveninck R., & Bialek W. (1997). Spikes: Exploring the neural code. Cambridge, MA: MIT Press. Rolls, E. T., & Treves, A. (1998). Neural networks and brain function. New York: Oxford University Press. Samengo, I. (2001). The information loss in an optimal maximum likelihood decoding. Neural Comput., 14, 771–779. Treves, A. (1997). On the perceptual structure of face space. Biosystems 40, 189–196. Victor, J. D., & Purpura K. P. (1997). Metric-space analysis of spike trains: Theory, algorithms and application. Network: Comput. Neural Syst., 8, 127–164. Young F. W., & Hamer R. M. (1994) Theory and applications of multidimensional scaling. Hillsdale, NJ: Eribaum. Young, A. W., Rowland, D., Calder, A. J., Etcoff, N. L., Seth, A., & Perrett, D. I. (1997). Facial expression megamix: Tests of dimensional and category accounts of emotion recognition. Cognition, 63 271–313.

Received April 30, 2003; accepted August 10, 2004.

NOTE

Communicated by Emilio Salinas

Gain Control by Concerted Changes in IA and IH Conductances Denis Burdakov [email protected] Neurosciences Division, Biological Sciences, University of Manchester, Manchester M13 9PT, U.K.

Stability of intrinsic electrical activity and modulation of input-output gain are both important for neuronal information processing. It is therefore of interest to define biologically plausible parameters that allow these two features to coexist. Recent experiments indicate that in some biological neurons, the stability of spontaneous firing can arise from coregulated expression of the electrophysiologically opposing IA and IH currents. Here, I show that such balanced changes in IA and IH dramatically alter the slope of the relationship between the firing rate and driving current in a Hodgkin-Huxley-type model neuron. Concerted changes in IA and IH can thus control neuronal gain while preserving intrinsic activity. 1 Introduction Maintaining stable intrinsic activity patterns in the absence of relevant stimuli is thought to be important for proper information processing in neurons and neural circuits (Turrigiano & Nelson, 2000; Davis & Bezproznanny, 2001; Marder & Prinz, 2002). But the ability of neurons to change gain, that is, to alter the slope of the relationship between driving stimulus and firing response, is also critical for computation in diverse neural systems (Salinas & Thier, 2000; Salinas & Sejnowski, 2001). It is therefore of interest to define biologically plausible parameters that allow the gain of a neuron to be modulated without changes in its intrinsic (stimulus-independent) activity. It was recently demonstrated, using biological and model neurons, that one of the ways to achieve such “silent” modulation of gain is by covarying the frequencies of excitatory and inhibitory background synaptic input currents (Chance, Abbott, & Reyes, 2002). Another recent report indicated that biological neurons can intrinsically covary the magnitudes of certain excitatory and inhibitory ionic currents: increasing the expression of the transient outward current (IA ) led to proportional increases in hyperpolarization-activated inward current (IH ) in lobster somatogastric neurons (MacLean, Zhang, Johnson, & Harris-Warrick, 2003). This biological coregulation of IA and IH did not result in significant changes in intrinsic firing properties (MacLean et al., 2003). Neural Computation 17, 991–995 (2005)

© 2005 Massachusetts Institute of Technology

992

D. Burdakov

Therefore, this note addresses the question of whether the concerted changes in IA and IH can act as a cellular mechanism for “silent” gain modulation. 2 Results and Discussion To explore the effects of concerted changes in IA and IH on neuronal gain, a recently published model of lobster somatogastric neurons (Prinz, Thirumalai, & Marder, 2003) was used. This is a Hodgkin-Huxleytype, single-compartment model that comprises seven membrane currents (INa , ICaS , IA , IKCa , IKd , IH , and Ileak ) and an intracellular calcium buffer, and exhibits tonic intrinsic firing (Prinz et al., 2003). All maximal conductances (except those for IA and IH , which were varied as indicated) were as in Prinz et al., and simulations were performed using MATLAB stiff systems numerical integrator ode15s, at time resolution of 25 µs. The maximal conductances (g) of IA and IH were changed from the values in Prinz et al. (10 and 0.05 mS/cm2 , respectively) in such a way so as to keep the intrinsic firing characteristics (tonic pattern and firing frequency) unaltered. The relationship between the two conductances for the latter condition is shown in Figure 1C. Changes in gA and gH markedly altered the firing responses of the model neuron to a tonic excitatory current: balanced increases in gA and gH decreased the firing response (see Figures 1A and 1B). This implies that the concerted changes in gA and gH selectively altered the gain, since the tonic nature of firing and the firing frequency remained unaffected. To quantify the changes in gain in a simplified but plausible way, the firing rate (f) was plotted against the driving excitatory current (I) for

Figure 1: (A, B) Examples of simulations illustrating that concerted changes in IA and IH affect neuronal gain without changes in unstimulated firing rate or pattern. (A) Neuron with a small IA and IH (gA and gH are 10 and 0.05 mS/cm2 , respectively) responds to a tonic excitatory current with a large increase in firing rate. (B) Increasing IA and IH in a balanced manner that leaves unstimulated firing unaltered (here, gA and gH are 50 and 1 mS/cm2 , respectively) leads to a marked reduction of the firing response to the same excitatory current. Simulation protocols used to elicit IA , IH , and firing responses in A and B are shown schematically below the corresponding traces. (C) The relationship between gA and gH that satisfies the condition of unaltered intrinsic firing characteristics. (D) Examples of f-I relationships of neurons with different balanced combinations of gA and gH (respectively, in mS/cm2 : black circles = 10 and 0.05, white squares = 20 and 0.18, black triangles = 50 and 1, white diamonds = 100 and 6). (E) Gain values (the slopes of f-I relationships such as those shown in D) plotted against the sum of gA and gH for combinations of gA and gH that do not alter intrinsic firing characteristics.

Gain Control by Concerted Changes in IA and IH Conductances

993

994

D. Burdakov

a range of combinations of gA and gH that resulted in the same intrinsic firing rate. The balanced increases in gA and gH led to progressive decreases in the gain (slope) of the f-I relationship (see Figure 1D). The gain was altered most steeply when the sum of gA and gH was below 100 mS/cm2 (see Figure 1E); this conductance range is consistent with physiological gA and gH values in lobster somatogastric neurons (MacLean et al., 2003; Prinz et al., 2003). Decreases in gain saturated when the sum of gA and gH exceeded about 100 mS/cm2 (see Figure 1E). These results indicate that concerted changes in IA and IH may allow neurons to vary gain while preserving intrinsic activity patterns. To the best of my knowledge, such cellular mechanism of gain control has not been previously reported. This mechanism is biologically plausible (MacLean et al., 2003), and could potentially be of general physiological importance considering that IA and IH are expressed together in many types of biological neurons and are under the control of a variety of neuromodulators (Yang et al., 2001; Ramakers & Storm, 2002; Schweitzer, Madamba, & Siggins, 2003; Frere & Luthi, 2004).

References Chance, F. S., Abbott, L. F., & Reyes, A. D. (2002). Gain modulation from background synaptic input. Neuron, 35(4), 773–782. Davis, G. W., & Bezprozvanny, I. (2001). Maintaining the stability of neural function: A homeostatic hypothesis. Annu. Rev. Physiol., 63, 847–869. Frere, S. G., & Luthi, A. (2004). Pacemaker channels in mouse thalamocortical neurons are regulated by distinct pathways of cAMP synthesis. J. Physiol., 554, 111–125. MacLean, J. N., Zhang, Y., Johnson, B. R., & Harris-Warrick, R. M. (2003). Activityindependent homeostasis in rhythmically active neurons. Neuron, 37(1), 109–120. Marder, E., & Prinz, A. A. (2002). Modeling stability in neuron and network function: The role of activity in homeostasis. Bioessays, 24(12), 1145–1154. Prinz, A. A., Thirumalai, V., & Marder, E. (2003). The functional consequences of changes in the strength and duration of synaptic inputs to oscillatory neurons. J. Neurosci., 23(3), 943–954. Ramakers, G. M., & Storm, J. F. (2002). A postsynaptic transient K(+) current modulated by arachidonic acid regulates synaptic integration and threshold for LTP induction in hippocampal pyramidal cells. Proc. Natl. Acad. Sci. U. S. A, 99(15), 10144–10149. Salinas, E., & Sejnowski, T. J. (2001). Gain modulation in the central nervous system: Where behavior, neurophysiology, and computation meet. Neuroscientist, 7(5), 430–440. Salinas, E., & Thier, P. (2000). Gain modulation: A major computational principle of the central nervous system. Neuron, 27(1), 15–21. Schweitzer, P., Madamba, S. G., & Siggins, G. R. (2003). The sleep-modulating peptide cortistatin augments the H-current in hippocampal neurons. J. Neurosci., 23(34), 10884–10891.

Gain Control by Concerted Changes in IA and IH Conductances

995

Turrigiano, G. G., & Nelson, S. B. (2000). Hebb and homeostasis in neuronal plasticity. Curr. Opin. Neurobiol., 10(3), 358–364. Yang, F., Feng, L., Zheng, F., Johnson, S. W., Du, J., Shen, L., Wu, C. P., & Lu, B. (2001). GDNF acutely modulates excitability and A-type K(+) channels in midbrain dopaminergic neurons. Nat. Neurosci., 4(11), 1071–1078.

Received August 23, 2004; accepted October 1, 2004.

NOTE

Communicated by Klaus Obermayer

Winner-Relaxing Self-Organizing Maps Jens Christian Claussen [email protected] ¨ Theoretische Physik und Astrophysik, Institut fur Christian-Albrechts-Universit¨at zu Kiel, 24098 Kiel, Germany

A new family of self-organizing maps, the winner-relaxing Kohonen algorithm, is introduced as a generalization of a variant given by Kohonen in 1991. The magnification behavior is calculated analytically. For the original variant, a magnification exponent of 4/7 is derived; the generalized version allows steering the magnification in the wide range from exponent 1/2 to 1 in the one-dimensional case, thus providing optimal mapping in the sense of information theory. The winner-relaxing algorithm requires minimal extra computations per learning step and is conveniently easy to implement. 1 Introduction The self-organizing map (SOM) algorithm (Kohonen, 1982) served both as a model for topology-preserving primary sensory processing in the cortex (Obermayer, Blasdel, & Schulten, 1992) and for technical applications (Ritter, Martinetz, & Schulten, 1992). Self-organizing feature maps map an input space, such as the retina or skin receptor fields, into a neural layer by feedforward structures with lateral inhibition. Defining properties are topology preservation, error tolerance, plasticity, and self-organized formation by a local process. Compared to other clustering algorithms and vector quantizers, its apparent advantage for data visualization and exploration is its approximative topology preservation. In contrast to the elastic net (Durbin & Willshaw, 1987) and the Linsker (1989) algorithm, which perform gradient descent in a certain energy landscape, the Kohonen algorithm lacks an energy function in the general case of a continuous input distribution. Although the learning process can be described in terms of a Fokker-Planck equation (Ritter & Schulten, 1988), the expectation value of the learning step is a nonconservative force (Obermayer et al., 1992) driving the process so that it has no associated energy function. Despite much research, the relationships between the Kohonen model and its variants to general principles remain an open field (Kohonen, 1991). 1.1 Kohonen’s Self-Organizing Feature Map. Kohonen’s selforganizing map is defined as follows. Every stimulus v of an input space V Neural Computation 17, 996–1009 (2005)

© 2005 Massachusetts Institute of Technology

Winner-Relaxing Self-Organizing Maps

997

is mapped to a “center of excitation,” or winner, s = argminr∈R |wr − v|,

(1.1)

where |·| denotes the Euclidean distance in input space. In the Kohonen model, the learning rule for each synaptic weight vector wr is given by δwr = η · grs · (v − wr ),

(1.2)

where grs defines the neighborhood relation in R, and throughout this article will be a gaussian function of the Euclidean distance |r − s| in the neural layer. Topology preservation is enforced by the common update of all weight vectors whose neuron r is adjacent to the center of excitation s; the adjacency function grs prescribes the topology in the neural layer. The speed of learning η usually is decreased during the process. 1.2 The Winner-Relaxing Kohonen Algorithm. We now consider an energy function V first proposed in Ritter et al. (1992). If we have a discrete input space, the potential function for the expectation value of the learning step is given by V({w}) =


1
γ grs p(vµ ) · |vµ − wr |2 , 2 rs µ µ|v ∈F ({w})

(1.3)

s

where Fs ({w}) is the cell of the Voronoi tesselation (or Dirichlet tesselation) of input space defined by equation 1.1. For discrete input space, where p(v) is a sum over delta peaks δ(v − vµ ), the first derivative with regard to wr is not continuous at all weight vectors where the borders of the Voronoi tesselation are shifting over one of the input vectors (see Figure 1). However, equation 1.3 requires the assumption that none of the borders of the Voronoi tesselation is shifting over a pattern vector vµ , which may be fulfilled in the final convergence phase for discrete input spaces but becomes problematic if there are more receptor positions than neurons. If p(v) is continuous, the sum over µ becomes an integral, and with every stimulus vector update, the surrounding Voronoi borders slide over stimuli (which means they become represented by another weight vector), so that there is no global energy function for the general case. We remark that replacing the crisp  (or hard) winner selection, equation 1.1, by a soft-winner s = argminr r grr |wr − v|2 minimizes equation 1.3 even in the continuous case (Graepel, Burger, & Obermayer, 1997, Heskes, 1999). This is a formally elegant approach if one wants to ensure the existence of an energy function and accepts modifying the winner selection. However, to motivate the winner-relaxing learning, we return to the hard winner selection scheme, equation 1.1, and take up the learning rule given

998

J. Claussen

Figure 1: Shift of Voronoi borders as an effect of weight vector update.

by Kohonen (1991). Our use of this ansatz, however, is justified here only a posteriori by its use for adjusting the magnification. From the shift of the borders of the Voronoi tesselation Fs ({w}) (see Figure 1) in evaluation of the gradient with respect to a weight vector wr , Kohonen (1991) derived  for the (approximated) gradient descent in V the additive term − 12 ηδrs r =s gr s (v − wr ), extending equation 1.2 for the winning neuron. As it implied an additional elastic relaxation, it was straightforward to call it the “winner-relaxing” (WR) Kohonen algorithm (Claussen, 1992). In the remainder of this article, we study the (generalized) winner-relaxing Kohonen algorithm, or winner-relaxing self-organizing Map (WRSOM), introduced first in Claussen (1992) as  δwr = η (v −

γ wr )grs

− λδrs


r =s

 γ gr s (v

− w r ) ,

(1.4) γ

where s is the center of excitation for incoming stimulus v, and grs is a gaussian function of distance in the neural layer with characteristic length γ . Here, λ is a free parameter of the algorithm. The original algorithm (associated with the potential function) proposed by Kohonen in 1991 is obtained

Winner-Relaxing Self-Organizing Maps

999

Table 1: Magnification Laws for One-Dimensional Maps. Elastic Net 1 1+

κ P σ2 J

VQ, NG

WRK, λ = 12

SOM, λ=0

WRK, λ = −1

Linsker

P 1/3

P 4/7

P 2/3

P1

P1

for λ = +1/2, whereas the classical SOM algorithm is obtained for λ = 0. The influence of λ on the magnification behavior is the central issue of this article. 1.3 The Magnification Factor. The magnification factor is defined as the density of neurons r (i.e., the density of synaptic weight vectors wr ) per unit volume of input space, and therefore is given by the inverse Jacobi determinant of the mapping from input space to neuron layer: M = |J |−1 = | det(dw/dr)|−1 . We assume the input space to be continuous and of the same dimension as the neural layer, and the map to be noninverting (J > 0). The magnification factor quantifies the network’s response to a given probability density of stimuli P(v). To evaluate M in higher dimensions, one in general has to compute the equilibrium state of the entire network and therefore needs complete global knowledge on P(v), except for separable cases. For one-dimensional mappings, the magnification factor can ¯ follow a universal magnification law, that is, M(w(r)) is a function of the local probability density P only, independent of both the location r in the ¯ neural layer and the location w(r) in input space. It is nontrivial whether there exists a power law; the elastic net obeys a universal magnification law that remarkably is not a power law (Claussen & Schuster, 2002) due to a nonvanishing elastic tension in regions of small input density. For the classical Kohonen algorithm, the magnification law is given by a power ρ ¯ ¯ law M(w(r)) ∝ P(w(r)) with exponent ρ = 23 (Ritter & Schulten, 1986). (See Table 1 for an overview.) For a discrete neural layer and different neighborhood kernels, corrections apply (Ritter, 1991; Ritter et al., 1992; Dersch & Tavan, 1995). As the brain is assumed to be optimized by evolution for information processing, one could conjecture that maximal mutual information can define an extremal principle governing the setup of neural structures. For feedforward neural structures with lateral inhibition, an algorithm of maximal mutual information has been defined by Linsker (1989) using the gradient descent in mutual information. It requires computationally costly integrations and has a highly nonlocal learning rule; therefore, it is neither favorable as a model for biological maps nor feasible for technical applications. Due to realization constraints, both technical applications and cortical networks (Plumbley, 1999) are not necessarily capable of reaching this

1000

J. Claussen

optimum. Even if one had experimental data of the magnification behavior, the question from what self-organizing dynamics neural structures emerge remains. Overall it is desirable to find learning rules that minimize mutual information in a simpler way. An optimal map from the view of information theory would reproduce the input probability exactly (M ∼ P(v)ρ with ρ = 1), being equivalent to the condition that all neurons in the layer are firing with same probability. This defines an equiprobabilistic mapping (van Hulle, 2000). An exponent ρ = 0, on the other hand, corresponds to a uniform distribution of weight vectors, or no adaptation at all. So the magnification exponent is a direct indicator of how far an SOM algorithm is away from the optimum predicted by information theory. 2 Magnification Exponent of the Winner-Relaxing Kohonen Algorithm We now derive the magnification law of the winner-relaxing Kohonen algorithm, equation 1.4, for the case of a 1D→1D map. Note that for higher dimensions, analytical results can be obtained only for special degenerate cases of the input probability density and therefore lack generality. The necessary condition for the final state of the algorithm is that the expectation value of the learning step vanishes for all neurons r :  ∀r ∈R

0=

dv p(v)δ w¯ r (v).

(2.1)

Since this expectation value is equal to the learning step of the pattern parallel rule, equation 2.1 is the stationary state condition for both serial and parallel updating and also for batch updating. Thus, we can proceed with these variants simultaneously. (As synaptic plasticity is widely assumed to be based on integrative effects, one could claim that a parallel model is sufficient.) The update rule, equation 4.1, can be extended by an additional diagonal term controlled by µ:1  δwr = η (v − wr ) ·

grγs

+ µ(v − wr )δr s − λδr s


r  =s

 γ gr  s (v

−w ) . r

(2.2) 1 Whereas the extra term controlled by the parameter µ has been introduced in Claussen (1992) for pure generality and will be kept within the derivation, it does not contribute to the magnification. In general, the setting µ = 0 is recommended (and probably most stable), and the winner-relaxing Kohonen algorithm thus has only one relevant control parameter λ.

Winner-Relaxing Self-Organizing Maps

1001

By insertion of the update rule, equation 2.2, one obtains  γ 0 = ds P(w(s))J(s)g ¯ − w(r ¯ )) ¯ r s (w(s)  + (µ + λ) ·  −λ ·  =



ds P(w(s))J(s)δ ¯ ¯ − w(r ¯ )) r s (w(s)  ≡0

γ

ds dr  P(w(s))J(s)δ ¯ ¯ − w(r ¯  )) r s gr  s (w(s)

ds P(w(s))J ¯ (s)grγs (w(s) ¯ − w(r ¯ )) 

+ λ · P(w(r ¯ ))J (r ) ·

γ

dr  gr  r (w(r ¯  ) − w(r ¯ )).

(2.3)

The derivation can be performed analogous to Ritter et al. (1992). In the continuum limit, there is always an exactly matching winning weight vector w¯ s = v. Further, the integration variable is substituted, dv = dw ¯ s = J (s)ds, and we define the abbreviation P¯ := P(w(r ¯ )). In the first integrand, P¯ J has to be expanded in powers of q := s − r . Within the second integral, P¯ J is evaluated only at r . Thus, the integration yields in leading order in q :

 d2 w¯ ¯ d( P¯ J ) 1 2 dw ¯ 0=γ + PJ dr dr 2 dr 2    2 2 dw¯ q d w¯  γ   + λ · P¯ J · dq g0q   q dr + 2 dr 2   

0=γ2

contribution 0

d( P¯ J ) P¯ J dJ P¯ J dJ J + +λ dr 2 dr 2 dr

 .

(2.4)

¯ Further, we have to require γ = 0, P = 0, d P/dr = 0. Then the ansatz of a ¯ )) (that is, J depends only on universal local magnification law J(r ) = J( P(r the local value of P, which may be expected for the one-dimensional case only) requires J to fulfill the differential equation

 J 1 λ dJ 0= + 1+ + (2.5) P¯ 2 2 d P¯ or 2 J dJ . =− d P¯ 3 + λ P¯

(2.6)

1002

J. Claussen

Magnification exponent

1 0.9 0.8

2/3 4/7 0.5

-1

-0.5

0

0.5

1

λ Figure 2: Impact of parameter λ on the magnification exponent. The cases of λ = 1/2 (Kohonen, 1991), the SOM case λ = 0 (Kohonen, 1982) and the winnerenhancing choice λ = −1 are marked with filled circles.

It has a power law solution (provided that λ = −3), which verifies the ansatz made above, J being a function of the local density only, M=

2 1 ∼ P(v) 3+λ . J

(2.7)

2 and can be tuned from Thus, the magnification exponent is given by 3+λ 1/2 to 1 (see Figure 2) within the range of stability. For the λ = 1/2 choice of the winner-relaxing Kohonen algorithm, the magnification factor follows an exact power law with magnification exponent ρ = 4/7, which is smaller than ρ = 2/3 for the classical self-organizing feature map (Ritter & Schulten, 1986), but is still much larger than ρ = 1/3 for vector quantization and neural gas. In any case, the maps resulting from the choices λ = 1/2 and λ = 0 are not optimal in terms of information theory.

3 Enhancing the Magnification From this result, one would try to invert the relaxing effect by choosing negative values for λ, which means to “enforce” the winner. In fact, the choice of λ = −1 leads to the magnification exponent 1. The magnification law, equation 2.7, is verified numerically, as is shown in Table 2. Apart from the fact that the exponent can be varied by a priori parameter choice between 1/2 and 1, the simulations show that our

Winner-Relaxing Self-Organizing Maps

1003

Table 2: Magnification Exponent of the Winner-Relaxing Algorithms Determined Numerically from a Sample Setup with 200 Neurons and 2 · 107 Update Steps and a Learning Rate of 0.1. ↓γλ→ 0.1 0.5 1.0 2.0 5.0 Theory:

−1

−3/4

−1/2

−1/4

0

1/4

1/2

3/4

1

0.29 ±.04 0.49 ±.02 0.75 ±.04 0.93 ±.03 0.99 ±.05 1.00

0.29 ±.04 0.46 ±.01 0.77 ±.02 0.86 ±.02 0.88 ±.04 0.89

0.23 ±.04 0.43 ±.02 0.68 ±.02 0.77 ±.02 0.80 ±.03 0.80

0.29 ±.04 0.45 ±.01 0.67 ±.02 0.71 ±.01 0.72 ±.02 0.73

0.27 ±.05 0.43 ±.01 0.61 ±.01 0.65 ±.01 0.66 ±.02 0.67

0.25 ±.04 0.40 ±.01 0.58 ±.01 0.61 ±.01 0.61 ±.02 0.62

0.26 ±.04 0.39 ±.02 0.58 ±.01 0.57 ±.01 0.57 ±.02 0.57

0.27 ±.04 0.37 ±.01 0.53 ±.01 0.53 ±.01 0.53 ±.02 0.53

0.27 ±.05 0.34 ±.01 0.51 ±.01 0.50 ±.01 0.50 ±.02 0.50

Notes: The input space was the unit interval; the stimulus probability density was chosen exponentially as exp(−βw) with β = 4. After an adaptation period of 5 · 107 learning steps further, 10% of the learning steps were used to calculate average slope and its fluctuation of log J as a function of log P. (The first and last 10% of neurons were excluded to eliminate boundary effects.) The small numbers denote the fluctuation of the exponent through the final 10% of the experiment. For small γ , the neighborhood interaction becomes too weak. If the gaussian neighborhood extends over some neurons (γ = 5), the exponent follows the predicted dependence of γ given by 2/(3 + λ). For |λ| > 1, the system is unstable. This is the case where the additional update term of the winner is larger than the sum over all other update terms in the network. Tuning of the parameter µ did not seem to extend the region of stability. As the relaxing effect is inverted for λ < 0, fluctuations are larger than in the Kohonen case.

winner-relaxing algorithm is able to establish information theoretically optimal SOMs in the “winner-enforcing” case (λ < 0).

4 Ordering Time and Stability Region At least for the 2D → 2D case, the winner-relaxing Kohonen algorithm was reported as somewhat faster (Kohonen, 1991) in the initial ordering process. In a 1D → 1D sample setup (Claussen, 2003), a marginally quicker ordering was observed for negative λ, at least at a relatively high learning rate η = 1. As a lot of parameters and the input distribution itself influence the ordering time and decay of fluctuations, different results may be obtained; for example, a small fraction of input distributions containing topological kinks takes much longer to become orderered, so minimal, maximal, averaged, and inverse-averaged ordering time will deviate.

1004

J. Claussen

λ = −1.0 λ = −0.5 λ = 0.0 λ = 0.5 λ = 1.0

log (fluctuations)

-1

-2

-3

-4

10

100

1000

10000

1e+05

1e+06

Number of iterations Figure 3: Time dependence (every tenth iterate shown) of the log root mean square fluctuations for different λ. Here, the same setup of a single run with γ = 1.0, η = 0.1, and 10 neurons is being used. Each run starts with the same configuration and random initial values between 0 and 1. For λ > 0, a quicker ordering is observed.

log (fluctuations)

-1

-2

-3

-4

10

100

1000

10000

1e+05

1e+06

Number of iterations Figure 4: Fast learning using a simple switching strategy. Starting with λ = 1/2, ordering is acheived quickly. At iteration step 2000, λ is immediately changed to −1 (dotted). This speeds up the learning phase by two orders of magnitude compared to starting with λ = −1 and by a factor 4 compared to λ = 0 (dashed, shown for comparison). If the duration of the initial ordering phase is underestimated, a long learning phase results (solid line; switch at step 200).

Winner-Relaxing Self-Organizing Maps

1005

log (fluctuations)

15

10

5

0

-5 -2

-1

0

1

2

3

4

5

λ Figure 5: For µ ∈ [−1, +1], the common stability range is λ ∈ [−1, +1]. For λ < −1, the log root mean square of the weight vector differences wr − wr −1 diverges, but extremely long, quiet transients are observed there. In the upper range λ > +1, making use of the diagonal term by using µ = 0 extends the stability range. The plots correspond to 107 (straight), 106 (dash-dotted), and 105 (dashed), respectively, for µ = 0. For 107 iterations, the cases µ = −1 (thin dots) and µ = +1 (thick dots) are shown. Parameters are γ = 1.0, η = 0.1, and 10 neurons are initialized near an equidistant chain with noise of amplitude 0.01 added.

If one instead investigates the time dependence of the fluctuations, for positive values of λ, a considerably quicker decay is observed (see Figure 3), being consistent with the observation by Kohonen (1991) mentioned above. These simulations indicate that for obtaining optimal magnification, the price of a longer learning phase may have to be paid. However, this drawback can be circumvented by combining the advantages of both λ ranges— using λ > 1 in the initial phase to speed up ordering and switching to λ = −1 after a considerable decay of fluctuations (see Figure 4). No complicated time dependence of this parameter switch has been used, and neither learning rate nor neighborhood has been changed during the simulation. The last important issue to be addressed is the dependence of stability on the parameter λ, especially at the border −1. Fortunately, the algorithm appears to be stable (in the 1D→1D case) in the whole range −1 ≤ λ ≤ +1, as shown in Figure 5. On both borders, the winner-relaxing learning remains

1006

J. Claussen

Output Entropy

4.6

10 * 10 batch 5 * 20 batch 10 * 10 5 * 20

4.4

4.2

4

-1

-0.8

-0.4

-0.6

-0.2

0

λ Figure 6: Entropy enhancement for the 2D→2D case for network geometries of 10 ∗ 10 and 5 ∗ 20 neurons. The data density was sin(πv1 ) · sin(πv2 ) within the unit square, γ = 5.0, and η was decreased from 0.01 to 0.001 during 106 learning steps. Alternatively, batch learning (over 100 steps) has been used; here, η was decreased from 0.05 to 0.001, and γ = 2.0 in the first 2 · 105 steps when ordinary SOM learning was applied (γ = 5.0, λ = 0). In all cases, for λ = −1, the entropy is enlarged compared to the unmodified case λ = 0 and close to the optimum (ln 100 = 4.605).

stable. Thus, the full range of magnification exponents between 1/2 and 1 can be acheived. In higher dimensions, no universal magnification law is expected, but one can evaluate the output entropy for a given input distribution and network. As shown in Figure 6, the enhancement of output entropy by winner-relaxing learning is effective also in the 2D case, although parameters have to be chosen more carefully.

5 Discussion After our first study (Claussen, 1992), Herrmann, Bauer, and Der (1995) introduced annother modification of the learning process, which was also applied to the neural gas algorithm (Villmann & Herrmann, 1998). Their central idea is to use a learning rate η that is locally dependent on the

Winner-Relaxing Self-Organizing Maps

1007

input probability density and also an exponent 1 can be obtained. As the input probability density should not be available to a neural map that selforganizes from stimuli drawn from that distribution, it is estimated from the actual local reconstruction mismatch (being an estimate for the size of the Voronoi cell) and from the time elapsed since the last time being the winner. Both operations require additional memory and computation, and due to the estimating character, the learning rate has to be bounded in practical use. This localized learning was overall more easily applicable and overcame the stability problems of the early approach of conscience learning (DeSieno, 1988). Another systematic method, the extended maximum entropy learning rule, has been introduced by van Hulle (1997). It approximates a map of maximal output entropy for arbitrary dimension, although in higher dimensions, the handling of the quantization regions becomes less practial (van Hulle, 1998). A quite different approach that is also capable of generating equiprobabilistic maps is via kernel optimization (van Hulle 1998, 2000, 2002); that is, neighborhood kernel radii themselves become learning parameters, in addition to the weight vectors defining the kernel centers. Other approaches, also influencing magnification, consider the selection of the winner to be probabilistic, leading to elegant statistical approaches to potential functions, as given by Graepel et al. (1997) and Heskes (1999). As shown recently (Claussen & Villmann, in press), the winner-relaxing concept can also be transferred successfully to the neural gas, confirming the utility of this class of learning rules.

6 Conclusions The Linsker, elastic net, and winner-relaxing Kohonen algorithms can be derived from an extremal principle, given by information theory, physical motivations, and reconstruction error, respectively. In this article, we have chosen the magnification law to indicate how close the algorithm reaches the adaptation properties of a map of maximal mutual information. The magnification law is one quantitative property that is both accessible by neurobiological experiments and manifests as a quantitative control parameter of a neural map used as vector quantizer in applications. A map of maximal mutual information uses all neurons with same probability; that is, their firing rate will be equal. In this article, we have investigated the winner-relaxing approach to establish a new family of vector quantizers. The shift from Kohonen (ρ = 2/3) to winner-relaxing Kohonen algorithm (ρ = 4/7) seems to be marginal if the emphasis is laid on the existence of a potential function. If a large magnification exponent is desired, the winner-relaxing Kohonen algorithm (with λ = −1) combines simple computation with a magnification corresponding to maximal mutual information.

1008

J. Claussen

Acknowledgments I thank H.G. Schuster for calling attention to the topic and for stimulating discussions. References Claussen, J. C. (1992). Selbstorganisation Neuronaler Karten. Diploma thesis, Kiel, Germany. Claussen, J. C. (2003). Winner relaxing and winner-enhancing Kohonen maps: Maximal mutual information from enhancing the winner. Complexity, 8(4), 15–22. Claussen, J. C., & Schuster, H. G. (2002). Asymptotic level density of the elastic net self-organizing map. In J. R. Dorronsoro (Ed.), Artificial neural networks. Berlin: Springer. Claussen, J. C., & Villmann, T. (in press). Magnification control in winner relaxing neural gas. Neurocomputing. Available at http://dx.doi.org/j.neucom. 2004.01.191. Dersch, D. R., & Tavan, P. (1995). Asymptotic level density in topological feature maps. IEEE Transactions on Neural Networks, 6, 230–236. DeSieno, D. (1988). Adding a conscience to competitive learning. In Proc. ICNN’88, International Conference on Neural Networks (pp. 117–124). Piscataway, NJ: IEEE Service Center. Durbin, R., & Willshaw, D. (1987). An analogue approach to the travelling salesman problem using an elastic net method. Nature, 326, 689–691. Graepel, T., Burger, M., & Obermayer, K. (1997). Phase transitions in stochastic selforganizing maps. Phys. Rev. E, 56, 3876–3890. Herrmann, M., Bauer, H.-U., & Der, R. (1995). Optimal magnification factors in selforganizing maps. In F. Fogelman-Soulie, & P. Gallinari (Eds.), Proceedings of the International Conference on Artificial Networks (pp. 75–80). Paris: Ec2 et Cie. Heskes, T. (1999). Energy functions for self-organizing maps. In E. Oja & S. Kaski (Eds.), Kohonen maps. Amsterdam: Elsevier. Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological Cybernetics, 43, 59–69. Kohonen, T. (1991). Self-organizing maps: Optimization approaches. In T. Kohonen, K. Makisara, O. Simula, & J. Kangas (Eds.), Artificial neural networks. Amsterdam: North-Holland. Linsker, R. (1989). How to generate ordered maps by maximizing the mutual information between input and output signals. Neural Computation, 1, 402–411. Obermayer, K., Blasdel, G. G., & Schulten, K. (1992). Statistical-mechanical analysis of self-organization and pattern formation during the development of visual maps. Phys. Rev. A, 45, 7568–7589. Plumbley, M. D. (1999). Do cortical maps adapt to optimize information density? Network, 10, 41–58. Ritter, H. (1991). Asymptotic level density for a class of vector quantization processes. IEEE Trans. Neural Networks, 2, 173–175. Ritter, H., Martinetz, T., & Schulten, K. (1992). Neural computation and self-organizing maps: An introduction. Reading, MA: Addison-Wesley.

Winner-Relaxing Self-Organizing Maps

1009

Ritter, H., & Schulten, K. (1986). On the stationary state of Kohonen’s self-organizing sensory mapping. Biol. Cybernetics, 54, 99–106. Ritter, H., & Schulten, K. (1988). Convergence properties of Kohonen’s topology conserving maps: Fluctuations, stability and dimension selection. Biological Cybernetics, 60, 59–71. van Hulle, M. M. (1997). Nonparametric density estimation and regression achieved with topographic maps maximizing the information-theoretic entropy of their outputs. Biological Cybernetics, 77, 49–61. van Hulle, M. M. (1998). Kernel-based equiprobabilistic topographic map formation. Neural Computation. 10, 1847–1871. van Hulle, M. M. (2000). Faithful representations and topographic maps. New York: Wiley. van Hulle, M. M. (2002). Kernel-based topographic map formation by local density modeling. Neural Computation, 14, 1561–1573. Villmann, T., & Herrmann, M. (1998). Magnification control in neural maps. In Proc. European Symposium on Artificial Neural Networks, 1998 (pp. 191–196). Bruges: D-Facto.

Received August 20, 2002; accepted November 1, 2004.

LETTER

Communicated by DeLiang Wang

Image Segmentation by Networks of Spiking Neurons Joachim M. Buhmann [email protected]

Tilman Lange [email protected] Swiss Federal Institute of Technology, CH-8092 Zurich, Switzerland

Ulrich Ramacher [email protected] ¨ Infineon Technologies, D-81739 Munchen, Germany

A network of leaky integrate-and-fire (IAF) neurons is proposed to segment gray-scale images. The network architecture with local competition between neurons that encode segment assignments of image blocks is motivated by a histogram clustering approach to image segmentation. Lateral excitatory connections between neighboring image sites yield a local smoothing of segments. The mean firing rate of class membership neurons encodes the image segmentation. A weight modification scheme is proposed that estimates segment-specific prototypical histograms. The robustness properties of the network implementation make it amenable to an analog VLSI realization. Results on synthetic and real-world images demonstrate the effectiveness of the architecture. 1 Introduction The task of segmenting a given image into disjoint (homogeneous) areas or segments is often considered an important yet difficult task in computer vision. Image segmentation provides the common basis to solve many subsequent computer visions problems such as object recognition and scene analysis. Segmented images are significantly reduced in complexity, which might be measured by coding lengths, but at the same time they retain a large amount of information on objects in the image and their mutual relations. The most competitive models for image segmentation are currently formulated as high-level probabilistic models of pixel assignments to groups (Puzicha, Hofmann, & Buhmann, 1999; Tu, Zhu, & Shum, 2002). The inference essentially amounts to finding minima of the corresponding energy functional. This strategy is advantageous, since stable states correspond to (local) minima of the objective function. The optimization, however, might be infeasible for practical applications depending on the model. Furthermore, the biological plausibility of these models is tentative since the Neural Computation 17, 1010–1031 (2005)

© 2005 Massachusetts Institute of Technology

Image Segmentation by Networks of Spiking Neurons

1011

popular claim that biological neural networks are statistical inference machines involves a large amount of abstraction. Bridging the gap between biologically plausible models of low and intermediate vision and abstract probabilistic model for image analysis is required for two reasons: (1) a testable model in biological science has to be sufficiently close to physiological neurons so that theoretical predictions are amenable to experimental tests in living animals (Koch, 1999), and (2) biological neural networks might motivate new designs of hardware and software solutions for vision problems, for example, in analog VLSI. This article is motivated by both lines of thought, although our research effort was mainly driven by the second goal. Our proposal employs a network of spiking neurons for image segmentation. Although a similar implementation in a rate model is also possible, the use of spiking neurons promises high relevance for biological systems and, furthermore, might be more flexible for computer vision applications. Starting with a highly competitive probabilistic model for image segmentation called the histogram clustering model (HCM), we propose a neuronal version of this segmentation model that has been successfully applied to color and texture analysis of images (Puzicha et al., 1999). Section 2 introduces the model and shows how it can be applied to image segmentation and extends the model by a smoothness constraint based on a local neighborhood system. The introduction of these lateral relationships leads to significantly increased computational complexity compared to the original model. The basic architecture of the derived network implementation is presented in section 3, where the smoothness requirement is taken into account. Finally, section 4 provides segmentation results for artificial and real images and compares them to the deterministic annealing implementation of the unconstrained histogram clustering model. Alternative neural network architectures that follow a similar line of thought are discussed in von der Malsburg and Buhmann (1992) and Wang and Terman (1997). Locally excitatory, globally inhibitory oscillator networks (LEGION), which are introduced in Wang and Terman (1997), consist of locally coupled integrate-and-fire (IAF) neurons. Here, pairwise correlations are used in order to achieve grouping. From a conceptual point of view, this network design is close to pairwise Markov random fields (MRF) used in image grouping. The LEGION network has been successfully applied to image segmentation problems. However, it appears to be only generally applicable if an algorithmic abstraction as developed in Campbell, Wang, and Jayaprakash (1999) is employed. LEGION combined with an MRF-based preprocessing has been also applied for the purpose of texture segmentation in Cesmeli and Wang (2001). In Wersing, Steil, and Ritter (2001), the competitive-layer model (CLM) has been employed for perceptual grouping. Using different layers, each representing a different class, pairwise interactions encode whether an object does or does not belong to a certain group. This model is highly related to the approach presented here and—with

1012

J. Buhmann, T. Lange, and U. Ramacher

lateral connection weights chosen accordingly—to pairwise data clustering (Hofmann & Buhmann, 1997). 2 The Histogram Clustering Model The neural network for image segmentation (see section 3) is motivated by the histogram clustering model (HCM) as proposed in Pereira, Tishby, and Lee (1993) and Puzicha et al. (1999), which explains the observed data by prototypical distributions on the feature space X —one prototype distribution for each designated cluster. Experimental evaluations (e.g., in Puzicha et al., 1999) demonstrate the competitive performance of histogram clustering and its superiority to pure pairwise clustering approaches as well as other graph cut methods. This observation motivates the neural implementation of the model discussed next. For the HCM, the input data are composed of histograms h i, j that count the co-occurrence of object i ∈ {1, . . . , N} and feature j ∈ {1, . . . , M}. Hence, for each object (each image site for the segmentation), a histogram hi := (h i, j )1≤ j≤M is defined as a tuple hi ∈ N M . M denotes the number of features, and n
i denotes the number of observations belonging to object i, that is, ni := j h i, j . Then, by normalizing h i, j , one arrives at empirical conditional probabilities pˆ ( j|i) :=

h i, j , ni

(2.1)

so that pˆ i := ( pˆ ( j|i))1≤ j≤M estimates the probability to observe features j given object i. To segment an image, we decompose it in N not necessarily disjoint image patches (see Figure 1) of equal size where ni is the number of pixels in image patch i. The central pixel of a patch is called a site. The sites can be thought of as pixels in a subsampled version of the original image. These sites are the objects considered in the HCM. Histograms are formed from gray values of each image patch to provide information about each site where each histogram consists of M bins. These bins represent the features in the segmentation problem and reflect the composition of intensity values within an image patch: an intensity value x ∈ {0, 1, . . . , 255} increases the Mx count in the bth bin by one where b =  256  + 1. The goal of image segmentation is to partition the set of image sites based on the features obtained from histogramming into K disjoint classes, called segments, where the number of groups K is supposed to be known a priori. In the HCM, segments are represented by prototypical distributions qν = (q ( j|ν)) j∈{1,...,M} where ν ∈ {1, . . . , K }. Here, q ( j|ν) can be interpreted as the probability of observing feature j with an object of class (segment) ν and, thus, qν is a class-conditional distribution. The HCM can be formulated as

Image Segmentation by Networks of Spiking Neurons

1013

Figure 1: Schematic view of the image patches and the histogram generation process.

the result of a maximum a posteriori (MAP) estimation for a generative model, which is characterized by the following sampling procedure: 1. Select object i with (a priori) probability Pr({i}). 2. Suppose i belongs to class ν. Then select feature j with probability q ( j|ν). For image segmentation, the prior probabilities of sites are uniform, that is, Pr({i}) := 1/N. Hence, the probability of observing feature j when object i belongs to class ν, is q ( j|ν)/N when generated according to the above sampling procedure. Here, the sets of unknown parameters are (1) the assignments of image sites to segments, which are denoted by c: {1, . . . , N} → {1, . . . , K }, and (2) the prototypical distributions q1 , . . . , q K , which both have to be estimated from the data. Considering the log posterior of the unknown parameters given the data (in the form of the histograms pˆ i ), we derive the negative log likelihood −L(c, q1 , . . . , q K |pˆ 1 , . . . , pˆ N ), const −





1≤i≤N 1≤ j≤M

pˆ ( j|i) log(q ( j|c(i))),

(2.2)

1014

J. Buhmann, T. Lange, and U. Ramacher

which has to be minimized. Replacing the constant term with the negative

empirical entropy i Pr({i}) j pˆ ( j|i) log( pˆ ( j|i)), we derive the objective function 

H(c, {q}) :=

DKL (pˆ i qc(i) ).

(2.3)

1≤i≤N

The argument that minimizes the costs H(c, {q }) represents the desired segmentation of the N image sites. Here, DKL denotes the Kullback-Leibler divergence between (discrete) probability distributions with DKL (pq) :=



p j log

1≤ j≤M

pj qj

(2.4)

for p = ( p1 , . . . , p M ) and q = (q 1 , . . . , q M ). Note that this model can be extended to include a local smoothness constraint. Suppose there
is a
(local) neighborhood Ni defined for each site i. By adding the term λ i j∈Ni 1{c(i) = c( j)} to the cost function defined in equation 2.3, one can penalize segmentations with high local variability. The multiplier λ ∈ R+ , which can be considered as a Lagrange parameter in a constrained optimization problem, controls the amount of penalization. The modified cost function H H (c, {q}) := H(c, {q}) + λ

 i

1{c(i) = c( j)}

(2.5)

j∈Ni

represents the model that we attempt to implement with spiking neurons. It usually results in improved segmentations (due to the smoothness) in comparison to those obtained by minimizing H. In Puzicha et al. (1999), a deterministic annealing approach to minimizing the cost function H has been proposed that employs the Gibbs distribution for the inference of robust solutions in every iteration. Note that the Gibbs distribution factorizes for H, thereby enabling relatively efficient operation. However, for the modified model H , the Gibbs distribution has no longer a factorial form,1 which renders annealing approaches to exact optimization impractical, since the Gibbs distribution is hard to compute. The experimental section of Puzicha et al. (1999) provides experimental evidence for the increased computational complexity when couplings between assignments 1 To be precise, the Gibbs distribution is considered over the set of all possible clustering solutions {c}, that is, P(c) = Z1 exp(−β H(c))
for a cost function H. The crucial point here represents the normalization constant Z := c exp(−β H(c)) (the partition function), being a sum over exponentially many solutions. For the HCM, this sum has a simple factorial form. However, for the constrained problem, there is no simple factorization, thereby making the exact computation of the Gibbs distribution intractable.

Image Segmentation by Networks of Spiking Neurons

1015

are introduced. The network-based approach aims at overcoming this issue without resorting to independence assumptions as, for example, used in mean-field approximations (Opper & Saad, 2001). For the experiments in section 4, we have employed the deterministic annealing approach in conjunction with the unconstrained model in order to compare the results with those obtained with network implementation, described next. 3 A Neuronal Implementation of the Histogram Clustering Model This section describes the network of spiking neurons that is employed for the purpose of image segmentation. We start by introducing the underlying leaky integrate-and-fire neuron model (Koch, 1999; Gerstner, 1998, 2002). 3.1 The Neuron Model. At a fixed time t, each neuron k is characterized by its internal state sk (t), its constant threshold θk , and its output yk (t) ∈ R. The spike duration τk is assumed to be constant for all neurons (τk := τ := 1) and is measured in milliseconds. The time at which neuron k fires for the (f) (1) (n) f th time is denoted by tk . Let tk , . . . , tk be the firing times of neuron k if it has fired for n times. The state of neuron k before firing, a fictitious neuron potential, is determined by the first-order ordinary differential equation (ODE), dsk sk (t) (t) = − + Ak, dt 

(3.1)

(n)

for t > tk + τ (i.e., t larger than the last time neuron k stopped firing) and sk
(t) < θk (i.e., the internal state is still below the threshold θk ). Here, Ak = l∈k wlk (t)yl (t) denotes the activation of neuron k, where k is the set of all presynaptic neurons of k and wlk ∈ R is the weight or strength of the connection between the presynaptic neuron l and the postsynaptic neuron k. When the internal state sk (t) reaches the threshold, it is reset to sk (t) = 0 and the neuron fires yk (t) for t ∈ [t ( f ) , t ( f ) + τ ], 1 ≤ f ≤ n; it emits a pulse of width τ . Neurons do not accumulate incoming signals while they emit a pulse themselves in our model, and the internal state sk (t) of neuron k is reset to zero immediately after firing. Furthermore, the term −sk (t)/ in equation 3.1 forces the neuron to forget prior excitation if the incoming stimulus (the activation Ak ) is not strong enough. In the following, t measures the model time (in ms). 3.2 The Network Architecture. As in the HCM (see section 2), we consider image sites as the objects to be grouped. In the case of segmentation,2 2

We adopt here the usual clustering nomenclature. What we call objects here are often denoted as features in the neural networks literature where the clustering problem is also called feature binding problem.

1016

J. Buhmann, T. Lange, and U. Ramacher

Figure 2: The basic architecture for an image site.

each object i is characterized by ( pˆ ( j|i))1≤ j≤M —the empirical probabilities of observing a pixel of a certain gray value belonging to the jth bin in the image path around site i. For the purpose of neural segmentation, we use for each site i a building block consisting of K + M neurons (see Figure 2). The M input neurons fire in each simulation step with frequency proportional to pˆ ( j|i). Each of the M input neurons is connected to all K neurons. The idea is now that firing patterns should reflect the dissimilarity of the prototypical feature probabilities q ( j|1), . . . , q ( j|K ) to the empirical feature frequencies pˆ ( j|i) as measured by DKL (pˆ i qν )—without explicitly relying on the prototypical distributions qν . The νth of the latter neurons is supposed to be active if the image site i should belong to the νth segment. Suppose the weight between an input neuron j for site i and the ν-segment membership neuron is chosen approximately as (i)

w j,ν (t) ≈ wimax + log(q ( j|ν))

(3.2) (i)

for sufficiently large times t; then the weights w j,ν (t) will reflect the KL divergence. The weight learning (see section 3.3) adapts the weights in this direction. The weights are time dependent due to the weight learning that employs the current state of the network. It should be noted that the prototypical distributions are not necessary for computation: we only need weights and an adaption scheme. wimax denotes a suitable constant that ensures positivity. In fact, it is the maximal synaptic strength (since log q ( j|ν) ≤ 0), and it also determines the minimal interspike interval through the synaptic dynamics in equation 3.1. Technically, wimax can be

Image Segmentation by Networks of Spiking Neurons

1017

seen as constant input current fed into each ν-segment membership neuron. Clearly, wimax ≤ θi,ν , where θi,ν is the threshold of the ν-segment-membership neuron in site i, in order to avoid trivial activity at maximal rate. Hence, the closer the input stimulus and a prototypical value are, the larger will be the input to the corresponding class representing neuron. With an appropriate weight learning, the input to segment-membership neurons reflects the per object costs (i.e., the per site costs) if site i is assigned to segment ν. This relation can be seen from equations 2.2 and 3.2. In particular, for appropriately chosen weights, the total input will reflect wimax − DKL (pˆ i qν ), and hence it will be the larger the more similar pˆ i is to qν , that is, the smaller DKL (pˆ i qν ) is, which exactly models the desired adaptation behavior. In addition to the excitatory stimuli, the segment membership neurons are connected by inhibitory synapses that realize a (soft) winner-takes-all mechanism; the segment-membership neuron ν with the largest input over several epochs is most likely to fire, while other neurons are potentially kept from firing. Ideally, the winner neuron is the one for which the corresponding cluster prototype q ( j|ν) has the highest similarity to pˆ ( j|i). This WTA mechanism is related to the inhibition scheme employed in Wersing et al. (2001), where lateral inhibition is also used for segment-coding neurons. The corresponding weights in our implementation are denoted by wµ,ν (set to −2/3 for the experiments) and are time independent and not affected by the learning of weights. However, these weights can be made dependent on the current state of the network. Experiments indicate that this dependence leads to slightly improved segmentations. A similar weight (i) adaption scheme as that for w j,ν (see below) can be employed for this purpose. We refer to the local neuronal architecture of a single site i as a block. There exist N blocks, one for each site. In order to smooth the neuronal segmentation, the architecture favors local cooperation between neurons coding for the same segments and competition between neurons of different segments. The standard fourneighborhood Ni of image sites around a site i is employed where neurons of the same segment layer are excitatorily connected and neurons of different segment layers inhibit each other. Clearly, larger neighborhoods are possible, essentially enforcing smoother segmentations. The weights between segment-membership neurons in neighboring blocks are chosen (i,l) to be time independent and are denoted by wµ,ν (set to −1/3(1 − 2δµ,ν ) in the experiments) if lateral connections between neuron µ from block i and neuron ν from block l are concerned. It should be noted that these connection strengths implicitly regulate the amount of penalization of nonsmooth segmentations and thereby determine the Lagrange parameter λ in equation 2.5. Technically such connections are realized as follows: for each site, the neuron representing the membership to segment ν gets inhibitory stimuli from all membership neurons for segments µ = ν of each block from the four-neighborhood and excitatory stimuli from those for the same segment ν of each block from this four-neighborhood.

1018

J. Buhmann, T. Lange, and U. Ramacher

To sum up, the total activation exciting the ν-segment membership neuron for site i is described by A(i) ν =

 j

(i)

(i)

w j,ν (t) y j (t) +

 µ=ν

wν,µ yµ(i) (t) +

 l∈Ni

µ

(i,l) (l) wµ,ν yµ (t).

(3.3)

(i)

The(i) resulting ODE for the ν-segment membership neuron is dsdtν (t) = (i) − sρν + Aν . In equation 3.3, the first sum adds up the excitatory stimulus (i) (i) from the input layer y j (t), 1 ≤ j ≤ M, which is weighted by w j,ν (t) (see equation 3.2). The second sum in equation 3.3 represents the inhibitory stimuli from the other segment-membership neurons µ = ν from block i. Finally, the last double sum arises from the excitatory and inhibitory links from the neighboring blocks. The ν-segment-membership neuron in site i fires when the internal state reaches the threshold θi,ν . A (hard) assignment of image sites i to segments ν is finally obtained by considering the firing rates (i.e., the number of firing times normalized by the maximally possible number of firing times within the simulation time) of segment-membership neurons. Note that we determine the segment assignment by means of a rate code. At first glance, the timing of spikes does not play any role for the rate code that we employ. In fact, timing becomes crucial through the term −sk (t)/, which erases prior excitation by forgetting, and through the lateral connections that realize the soft WTA mechanism and the lateral smoothing. Furthermore, we observe synchrony of segmentmembership neurons coding for the same class. This is investigated in the experimental section. 3.3 Learning of Weights. For the HCM, K · (M − 1) parameters have to be learned—the prototypical probabilities q ( j|ν), ν ∈ {1, . . . , K }, j ∈ {1, . . . , M}. These probabilities can be constructed from a given assignment of image sites to image segments. In contrast to the explicit calculation of prototypical probabilities by Bayesian inference, we will define a network dynamics where these quantities are iteratively estimated on the basis of local input statistics. The network implementation has K · N · M free parameters—the weights between input neurons and segment-membership (i) neurons. The weights w j,ν are randomly initialized in most simulation experiments, and as a consequence, the initial assignments become random. The of the learning rule is to shift the weights in a direction such that
aim (i) (i) j w j,ν (t) y j (t) reflects how well the ith site fits into segment ν. Based on the current firing rate of each segment-membership neuron within a block (∼site), a winner neuron can be determined, yielding a temporary assignment of the site to the winner segment. Let mi,ν = 1 iff segment ν is the winner segment in the block corresponding to site i and mi,ν = 0 otherwise. Denote by h j,ν the arithmetic mean of the probabilities for feature j of all sites that are temporarily assigned to segment ν, that

Image Segmentation by Networks of Spiking Neurons

1019



is, h j,ν = i pˆ ( j|i) mi,ν / i mi,ν . h j,ν can be seen as the current estimate of q ( j|ν). It should be noted that weighted sums can be computed in a network of spiking neurons (Ruf, 1998). Thus, considering h j,ν does not pose a conceptual problem, as is also demonstrated in Wersing et al. (2001), where a similar averaging quantity is involved in the learning process. The idea underlying the proposed weight adaption scheme originates from the observation that the prototypical distributions q ( j|ν) in the HCM ideally represent the average probability to observe feature j given segment ν, which is encoded by the histogram h j,ν . In the space of distributions, one therefore obtains d q ( j|ν) = α( h j,ν − q ( j|ν)) dt

(3.4)

as an update rule, where α is a suitable learning rate. For updating the weights between input neurons j and segment-membership neurons ν for site i and feature j, we propose the following rule, which can be derived from equations 3.2 and 3.9,     d (i) (i) w = α exp wimax − w j,ν (t) h j,ν − 1 , dt j,ν

(3.5)

where, α ∈ [0, 1] is a time-independent learning rate that is usually small (i) (i.e., α ≈ 0.01). Note that exp(wimax − w j,ν (t)) ≈ 1/q ( j|ν) holds due to relation 3.2. It determines how quickly weights are adapted. The learning rule depends on the ratio of the pre- and the postsynaptic activity of all neurons coding for the same segment. The mechanism employed here is similar to the learning approaches used in competitive learning (Intrator, 2002) and supports the following statistical interpretation: when the likelihood ratio h j,ν /q ( j|ν) between the measurements and the inferred prototypical feature distribution exceeds one, that is, the learned feature probabilities (i) are too small to match the measurements, then the weights w j,ν should be increased to raise q ( j|ν). By using equation 3.5, the weight dynamics are regulated in such a way that the likelihood ratio fluctuates around one, which yields a self-stabilizing learning dynamics. The value h j,ν which stores a reference value for the average feature probability given segment ν, might raise questions on the biological plausibility of this quantity in the context of neural networks. In essence, it represents a population average (as it is an average over the output of the input, both in time and space). It is clear that globally distributed information has to be collected in the network to represent the segment statistics. Such information can be computed within the network (similar to those in Maass, 1998; see also Ruf, 1998) and thus can be read out, for example, by adding neurons for each segment ν and each feature j to the architecture.

1020

J. Buhmann, T. Lange, and U. Ramacher

3.4 Localization of Information Processing. The process of learning involves information gathered from all sites belonging to the same segment. This can be considered disadvantageous in terms of biological plausibility as well as for hardware implementability. We have translated the system to a localized scheme without resorting to pure pairwise interactions. The latter would imply a CLM-like architecture as proposed in Wersing et al. (2001) and thereby would essentially amount to a pairwise clustering setting (Hofmann & Buhmann, 1997). We sketch the localization scheme: the set of neighbors of site i mentioned above is used for local smoothing. For the localization, a second set of neighbors attached to each site, denoted by Mi , is considered. Regarding the energy functions of section 2, the additional neighborhood system affects only the HCM cost function H in equation 2.5. To be precise, we approximate H by 

DKL (pˆ i qc(i) ) ≈

i



  (M ) DKL pˆ i qc(i)i ,

(3.6)

i (M )

where qν i represents a local centroid that depends on only the elements of Mi and the given assignment c. Again, we want to stress that this provides (M ) only the intuition for the approach taken. In practice, qν i is not used in the network during propagation. In fact, the introduction of the additional neighborhood system Mi affects only the weight learning rule: instead of considering h j,ν as average over all sites assigned to segment ν, only those that are in the vicinity Mi of site i are taken into consideration. This leads (M ) to a local estimate h j,ν i . Clearly, if Mi = {1, . . . , N} for all i, we have the original cost function H . The neighborhood system therefore regulates the amount of localization. In Geman, Geman, Graffigne, and Dong (1990), few (randomly chosen) long-range connections are used in order to break permutation symmetries, that is, to make the labeling unique. For our purpose, the inclusion of such long-range interactions in Mi also ensures that weights for classes that potentially do not occur in a small neighborhood around the site of interest can be reasonably learned. Therefore, the neighborhood used (solely) for learning mainly contains local neighbors of the current site i, in particular Ni ⊂ Mi , and a few long-range connections. Experimentally, we have observed that the localized scheme leads to satisfying results, even for small neighborhoods Mi , as demonstrated in Figure 3. In this example, Mi consisted of the four-neighborhood Ni and randomly chosen connections to 10% of the remaining sites. A 32 × 32 grid of sites was chosen. The major drawback of the localized scheme now becomes apparent: the potentially decreased convergence speed. Note that this is a difficult example for the localized version since there might be information about the other segment available only for the sites at the segment borders. Nevertheless, the localized scheme renders possible a

Image Segmentation by Networks of Spiking Neurons

1021

Figure 3: Localized versus nonlocalized information processing: The localized scheme achieves similar results but needs much more time to reach a stable state. This is, however, a difficult example for the localization (see the text). The time t refers to the model time (not to the number of simulation steps).

simple, robust, and essentially local learning rule, which is desirable for an analog VLSI implementation. We conclude that localized information processing is possible in principle but might significantly decrease convergence speed. 4 Experimental Results We now provide some experimental results that shed light on the properties of the proposed architecture and its usefulness for image segmentation. We have implemented a discrete time simulation with time steps t := 0.05. Further parameters of the simulations are  := 3.5, τ := 1, θ := 3, and w max := 3. States have been randomly initialized, and state updates are subject to small additive noise. It should be noted that this randomness reflects noisy environments as found, for example, in analog VLSI implementations. Learning starts after t = 2 in order to allow the network architecture to stabilize itself in its start configuration. The number of segments was chosen to be K := 5 in all experiments. For the histogramming, M := 12 bins were used. The grid used to generate image patches and sites consists of 64 × 64 nonoverlapping cells (as schematically depicted in Figure 1). 4.1 Application to Artificial Images. We apply the segmentation architecture to two artificial images (see Figure 4): an image consisting of simple geometric objects (a square and two disks) and a textured image composed of five textures. The network activity is monitored by measuring the average firing rate of the segment neurons for a particular site i and assigning this site to the segment represented by the neuron with the highest activity. The segment labels are gray value coded in the following images. For the first image (see Figure 4A), one would expect that all (four) objects are detected as entities of its own. In fact, HCM recovers this structure from the data as expected and introduces a fifth segment defined as the border around the lower disk-like object. The line-like fifth segment became

1022

J. Buhmann, T. Lange, and U. Ramacher

Figure 4: Artificial (input) images used for the experiments.

possible since we have chosen five rather than four potential segments and HCM does not invoke a model order selection procedure. The spiking neuron network finds a segmentation that is close to the ground-truth knowledge (and thus to the HCM result; see Figure 5). For comparing results, we have plotted the HCM result together with the current network segmentation and a difference picture in which black codes for a difference in the segmentations. For this image, we have rerun the simulation with K = 2 and monitored the mean activity of segment membership neurons (see Figure 6A). One clearly recognizes class-specific activity patterns that are shifted in phase. Therefore, neurons coding for the same class appear to synchronize, while one can observe desynchrony up to a certain extent with class membership neurons coding for different classes. Figures 7A and 7B show the firing rate and the spikes during simulation of the class membership neurons in a specific site. Two observations are noteworthy: the winner class in fact suppresses the other class, supported by the learning of weights, and the firing rate can be considered as an assignment probability. The second example—the image composed of five different textures; see Figure 4B—represents a much harder problem since the features (the histograms in this case) do not necessarily capture the difference between two textures. The notion of texture refers to the occurrence of repeated patterns in an image. The ability of the method to identify distinct textures based on local histograms largely depends on the kind of texture and of the locality of the histograms. Other features, such as Gabor filter responses, are more suitable for texture segmentation since they are more discriminative (Hofmann, Puzicha, & Buhmann, 1998) than gray value histograms. For

Image Segmentation by Networks of Spiking Neurons

1023

Figure 5: Segmentation results for an artificial image. Assignments (left: HCM; middle: network; right: difference picture) are shown for several time steps. The HCM segmentation has been included for comparison. (A) At t ≈ 10, the segments are roughly recognized by the network. (B) At t ≈ 95, the network has reached a stable state. Here, the assignments do not change anymore.

Figure 6: (A) Mean activity of class membership neurons for the first toy image in Figure 4A. (B) The misclassification rate of the network with regard to the HCM segmentation for six simulation runs with the railway station image: The plot demonstrates how the weight learning adapts weights to capture the image statistics.

1024

J. Buhmann, T. Lange, and U. Ramacher

Figure 7: (A) Firing rates of class 1/class 2 membership neurons for a single site. (B) Spikes generated by the class 1/class 2 membership neurons for a single site. See the text.

this reason, neither HCM nor the network can be expected to perfectly find the five different textures, and, in fact, this is not the case (see Figure 8). Nevertheless, both segmentations come rather close to ground truth. 4.2 Segmentation of Real-World Images. The results of the last section have demonstrated that the network matches the quality of the segmentation results achieved by HCM when applied to artificial data. We use two real-world images (see Figure 9) to show the segmentation abilities of the network under realistic conditions of real-world imagery. On the first image, depicting a railway station, we observe again a segmentation similar to the one of HCM (see Figure 10). All major segments have been identified by both approaches. Figure 6B shows the misclassification rate with regard to the HCM segmentation for a simulation run with this image. 3 It demonstrates how the network approximates the HCM segmentation via weight adaption. After some fluctuations, the network starts learning the segments around t = 5–6, and after that, it quickly finds a segmentation that is already very close to the one generated by HCM. For the balloon image, the network segmentation and that of HCM differ significantly (consider Figure 11). We argue that the segmentation generated by the network is superior because the meadow in the foreground is split into two parts by HCM while the network recognizes and represents it as a homogeneous region. Similarly, the forest in the image background is divided into two segments in a stripe pattern by HCM. By reconsidering

3

Since segment labels are unique only up to a permutation of the label indices, we consider the misclassification rate with respect to the permutation that minimizes the total number of misclassification.

Image Segmentation by Networks of Spiking Neurons

1025

Figure 8: Segmentation results for a Mondrian image. Assignments (left: HCM; middle: network; right: difference picture) are shown for several time steps. The HCM segmentation has been included for comparison. (A) At t ≈ 4, the segments are roughly recognized by the network. (B) At t ≈ 50 the network has reached a stable state.

Figure 9: Real-world (input) images used for the experiments: (A) Railway station. (B) Balloon image.

1026

J. Buhmann, T. Lange, and U. Ramacher

Figure 10: Segmentation results for an image of a railway station. Assignments (left: HCM; middle: network; right: difference picture) are shown for several time steps. The HCM segmentation has been included for comparison. (A) At t ≈ 5, the segments are roughly recognized by the network. (B) At t ≈ 50, the network has reached a stable state.

the original image in Figure 9B, one can hardly detect this pattern except for one case where the network has also been able to detect two different segments. Note that this behavior was essentially reproducible over 20 different simulation runs. Hence, we conclude that the network proposed leads to a visually more plausible result for the balloon image, although effectively not all segments are used by the network. In the experiments, we observed waves of activation sweeping through segments. This synchronization effect is caused by the lateral inhibitory and excitatory stimuli between neighboring image sites. We want to shed some more light on the network dynamics using the balloon image example. Figure 12A shows the values of the HCM cost function as a function of the model time (in ms), while Figure 12B shows the costs of the constrained model. For comparison, the HCM costs as obtained by the deterministic annealing approach in Puzicha et al. (1999) are also depicted as a constant function. Clearly, the deterministic annealing solution is superior in terms of HCM costs, which is not particularly surprising since the learning mechanism does not optimize the exact HCM cost function. The network, for example, has built in local smoothness constraints by

Image Segmentation by Networks of Spiking Neurons

1027

Figure 11: Segmentation results for the balloon image. Activities (left: HCM; middle: network; right: difference picture) are shown for several time steps. The HCM segmentation has been included for comparison. (A) At t ≈ 5, the network starts recognizing segments. (B) At t ≈ 50, the network has reached a stable state.

its lateral excitatory and inhibitory connections. Nevertheless, the weight adaption scheme still minimizes the HCM costs H, although in an oscillatory rather than steepest-descent fashion. Furthermore, we observe that the network finds a better solution in terms of the constrained objective function H in comparison to the HCM solution. In Figure 12D, we have again plotted the mean activity per segment of segment membership neurons (i, ν) in a small time window. One can detect periodic different activity patterns (see also above)—one characteristic pattern for each segment. In the time window shown, that network has almost reached a stable state. In general, we have observed these segment-specific periodic firing patterns, that is, synchrony within classes can be observed. The dynamics of the network can be nicely visualized by considering the prototypical histograms q ( j|ν). Note that these are computed for diagnostic purposes from the current assignment of image sites to segments, that is, from the current network read-out. The time evolution of these prototypical histograms is shown in Figure 12C for the second segment in the segmentation of the balloon image. Each curve depicts q ( j|ν), 1 ≤ j ≤ M as a function of the simulation time. From the figure, the effects of the weight adaptation scheme can be traced since

1028

J. Buhmann, T. Lange, and U. Ramacher

Figure 12: Demonstration of the network dynamics and learned statistics. (A) HCM costs for the balloon image in comparison with the costs achieved by the deterministic annealing approach. (B) HCM costs with smoothness penalty for the balloon image in comparison with the costs achieved by the deterministic annealing approach. (C) q ( j|2) as computed from the network site-to-segment assignment as a function of the time (see the text). (D) Activity pattern of segment membership neurons in a simulation time window.

the weights essentially determine the assignment of sites to segments and the prototypes amount to a compact representation of the assignments here. One can thus see how the weight adaptation scheme modifies the weights (i) w j,ν and how the network reaches a stable state. In Figure 13 we take a look at the number of milliseconds t (model time) required until a stable state is reached. In this context, we talk about a stable state if less than 1% of the assignments have changed for the previous 10 iterations. We have used 49 restarts, each with random initializations, in order to get an idea of how many simulation steps are needed. For the railway station, a stable state is reached after 44 ± 14 ms and after 28 ± 14 ms in the case of the balloon image. Note furthermore that no weight adaption takes place in the first 2 ms simulation steps. We conclude that the proposed

Image Segmentation by Networks of Spiking Neurons

1029

Figure 13: Histogram for the milliseconds t (model time) until a stable state is reached for the (A) railway station and (B) balloon image.

architecture rather quickly leads to stable results despite the use of lateral smoothness constraints. 5 Conclusion We have presented a network of spiking neurons for the purpose of image segmentation. It is motivated by a statistical model for histogram clustering. A learning rule with fast synaptic adaptation is derived from the model that allows the inference of robust segmentations. Results on artificial as well as real-world images demonstrate the applicability of the proposed procedure. The implementation of the segmentation model with spiking neurons provides the advantage that computationally expensive Gibbs models can be efficiently simulated by the network dynamics rather than approximated by mean field or loopy belief propagation techniques. Alternatives to the network simulation often lead to the use of such Gibbs sampling schemes, which are qualitatively equivalent to our suggested network model but demand a larger amount of computation for obtaining comparable results. Future work should address the computation of histograms within the network (which is externally done in the presented simulations) and an RBF-like network architecture. The latter would ideally rely on the timing of spikes for the purpose segmentation. This modification of the network essentially amounts to a change of the coding scheme (e.g., time to first spike), since the present architecture neglects the exact timing of spikes. For Euclidean data, a successful temporal scheme has been proposed in Ruf (1998) for the purpose of clustering. At the far end, a general theory for mapping statistical models to networks of pulsing neurons would be desirable since the use of statistical models enables robust inference and networks of pulsed neurons can be

1030

J. Buhmann, T. Lange, and U. Ramacher

cheaply implemented in hardware. We consider the proposed architecture a promising step in this direction. Acknowledgments We thank the anonymous reviewers for their helpful comments and suggestions. References Campbell, S. R., Wang, D. L., & Jayaprakash, C. (1999). Synchrony and desynchrony in integrate-and-fire oscillators. Neural Computation, 11, 1595–1619. Cesmeli, E., & Wang, D. (2001). Texture segmentation using gaussian-Markov random fields and neural oscillator networks. IEEE Transaction on Neural Networks, 12(2), 394–404. Geman, D., Geman, S., Graffigne, C., & Dong, P. (1990). Boundary detection by constrained optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(7), 609–628. Gerstner, W. (1998). Spiking neurons. In W. Maass & C. M. Bishop (Eds.), Pulsed neural networks. Cambridge, MA: MIT Press. Gerstner, W. (2002). Integrate-and-fire neurons and networks. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (2nd ed.). Cambridge, MA: MIT Press. Hofmann, T., & Buhmann, J. M. (1997). Pairwise data clustering by deterministic annealing. IEEE PAMI, 19(1), 1–14. Hofmann, T., Puzicha, J., & Buhmann, J. (1998). Unsupervised texture segmentation in a deterministic annealing framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8), 803–818. Intrator, N. (2002). Competitive learning. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (2nd ed., pp. 238–241) Cambridge, MA: MIT Press. Koch, C. (1999). Biophysics of computation—Information processing in single neurons. New York: Oxford University Press. Maass, W. (1998). Computing with spiking neurons. In W. Maass & C. M. Bishop (Eds.), Pulsed neural networks. Cambridge, MA: MIT Press. Opper, M., & Saad, D. (2001). Advanced mean field methods. Cambridge, MA: MIT Press. Pereira, F. C. N., Tishby, N., & Lee, L. (1993). Distributional clustering of English words. In Proceedings of the 31st Meeting of the Association for Computational Linguistics (pp. 183–190). Columbus, OH. Puzicha, J., Hofmann, T., & Buhmann, J. (1999). Histogram clustering for unsupervised segmentation and image retrieval. Pattern Recognition Letters, 20, 899–909. Ruf, B. (1998). Computing and learning with spiking neurons—theory and simulations. Unpublished doctoral dissertation, TU Graz. Tu, Z., Zhu, S. C., & Shum, H. (2002). Image segmentation by data driven Markov chain Monte Carlo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5), 657–673.

Image Segmentation by Networks of Spiking Neurons

1031

von der Malsburg, C., & Buhmann, J. (1992). Sensory segmentation with coupled neural oscillators. Biological Cybernetics, 67, 233–242. Wang, D., & Terman, D. (1997). Image segmentation based on oscillatory correlation. Neural Computation, 9, 805–836. Wersing, H., Steil, J., & Ritter, H. (2001). A competitive-layer model for feature binding and sensory segmentation. Neural Computation, 13, 357–387.

Received January 7, 2004; accepted October 1, 2004.

LETTER

Communicated by Bard Ermentrout

Neural Modeling of an Internal Clock Tadashi Yamazaki [email protected]

Shigeru Tanaka [email protected] Laboratory for Visual Neurocomputing, RIKEN Brain Science Institute. Wako, Saitama 351-0198, Japan

We studied a simple random recurrent inhibitory network. Despite its simplicity, the dynamics was so rich that activity patterns of neurons evolved with time without recurrence due to random recurrent connections among neurons. The sequence of activity patterns was generated by the trigger of an external signal, and the generation was stable against noise. Moreover, the same sequence was reproducible using a strong transient signal, that is, the sequence generation could be reset. Therefore, a time passage from the trigger of an external signal could be represented by the sequence of activity patterns, suggesting that this model could work as an internal clock. The model could generate different sequences of activity patterns by providing different external signals; thus, spatiotemporal information could be represented by this model. Moreover, it was possible to speed up and slow down the sequence generation.

1 Introduction An internal clock, which explicitly represents temporal information among cognitive and behavioral events, is widely believed to exist in the brain. In particular, many lines of evidence support the idea that a time passage from the onset of an input signal is represented. Several hypotheses of how a time passage is represented have been proposed (see Ivry, 1996, for review), and one possible way of the representation is by assigning one population of active neurons to one time interval from the stimulus onset. If active neuronal populations are sequentially generated in the order of interval lengths and one population exclusively corresponds to only one time interval, then we regard the sequence of these populations as the time passage from the stimulus onset and the stimulus onset as the trigger of starting the internal clock. Moreover, if another sequence of populations of active neurons is generated when another input signal is presented, this mechanism can represent not merely temporal information but also spatiotemporal information. Yet what type of neuronal circuit enables such a population coding of time? Neural Computation 17, 1032–1058 (2005)

© 2005 Massachusetts Institute of Technology

Neural Modeling of an Internal Clock

1033

To address this question, we studied a simple recurrent inhibitory network in which neurons were connected with random synaptic weights. We conducted computer simulations and demonstrated that this model generates a sequence of activity patterns of neurons dynamically changing from the stimulus onset and these activity patterns did not recur. Thus, this model could represent a time passage from the stimulus onset in the rate-coding scheme. We confirmed the nonrecurrence of activity patterns by calculating correlation between patterns generated at two given time steps. We then examined the parameter dependency of dynamics both numerically and analytically. We found that the total number of active neurons remained constant, suggesting that the maximum length of a time passage may be exponential to the number of neurons. The total activity of neurons was also estimated and was constant. Moreover, we found a parameter that controls the accuracy of a time passage representation. We further demonstrated some abilities of the model. The model can generate different activity patterns of neurons when different input stimuli are presented. This implies the ability to generate different time passages for different input patterns, which suggests that spatiotemporal information is represented. The stability of sequence generation against noise was examined. An important issue regarding an internal clock is how to reset it, that is, when an identical stimulus is presented, the same population of active neurons must always be generated. This was possible by adding a strong transient signal to the input signal. Finally, the clock speed could be modified by changing the neuronal time constant or the synaptic strength of recurrent connections. 2 Model Description The model consists of N neurons, each with two types of input: excitatory afferent inputs and recurrent inhibitory inputs from other neurons. Let zi (t) be the activity of neuron i at time t. This is calculated as zi (t) = [ui (t)]+ , where [x]+ = x if x > 0 and 0 otherwise, and ui (t) is the internal state of neuron i at time t, which is calculated as ui (t) = I −


j

wij

t


exp (−(t − s)/τ ) z j (s − 1).

(2.1)

s=1

I and wij denote the afferent input signal to neuron i and the synaptic weight of recurrent inhibition from neuron j to neuron i, respectively. We assume the temporal integration of activities of neurons over a long time, which is represented by the summation with respect to s, where τ determines the

1034

T. Yamazaki and S. Tanaka

integration range. Since z can take a real value, the model is represented in the rate-coding scheme. Our goal is to show that the activity patterns of neurons generated using equation 2.1 can encode a passage of time. Specifically, we hypothesized that the activity patterns do not recur, that is, the activity pattern at one time step is dissimilar to the pattern at a different time step when the interval between the two steps is large. Therefore, we use the following correlation function as the similarity index: N zi (t1 )zi (t2 ) C(t1 , t2 ) =  i=1  . N N 2 2 z (t ) z (t ) 1 2 i=1 i i=1 i

(2.2)

The numerator represents the inner product of vectors of activity patterns at times t1 and t2 , and the denominator normalizes the vector lengths. Since zi (t) can take only positive values, the index takes the value between 0 and 1. It takes 1 when theactivities of  neurons at t1 and t2 are identical. We N N defined C(t1 , t2 ) = 0 if i=1 zi2 (t1 ) or i=1 zi2 (t2 ) = 0, which occurs when the inhibition is strong and no neurons are active. Furthermore, we considered the similarity between two activity patterns (1) (2) under different parameter settings. Let zi (t) and zi (t) be the activities of neuron i at time t under different parameter settings. In this case, we used the following cross-correlation function as the similarity index: N C(t1 , t2 ) = 

N

(1) (2) i=1 zi (t1 )zi (t2 )



(1)2 i=1 zi (t1 )

N

(2)2 i=1 zi (t2 )

.

(2.3)

This similarity index also takes the value between 0 and 1. Its value is 1 when the two activity patterns are identical. We also defined C(t1 , t2 ) = 0 if  N (1)2  N (2)2 i=1 zi (t1 ) or i=1 zi (t2 ) = 0. We conducted computer simulations in T steps and evaluated similarity indices between t1 and t2 . We used the following parameters as the basic parameter setting: N = 1000, T = 1000, I = 1, τ = 100, and wij ’s were given under the binomial distribution Pr(wij = 0) = Pr(wij = 2κ/N) = 0.5, where κ = 1. We tested the cases of uniform distribution and normal distribution with average κ/N and variance (κ/N)2 , and found no significant difference in results. 3 Results 3.1 Representation of a Time Passage. The left panel in Figure 1 shows the plot of active states of the first 100 neurons out of 1000 neurons composing the recurrent network during T steps under the basic parameter setting. Once a neuron started to become active, its activity continued for several

Neural Modeling of an Internal Clock

1035

t2 80 60 40 20 0

0

200 400 600 800 1000

t

t1

similarity index

1

neuron index

100

0.8 0.6 0.4

C(200,t) C(500,t) C(800,t)

0.2 0

0

200 400 600 800 1000

t

Figure 1: Raster plot of active states (zi (t) > 0) of the first 100 neurons (left), similarity index (C(t1 , t2 )) (middle), and plots of C(200, t), C(500, t), and C(800, t) (right).

hundreds of steps, and then the neuron became inactive. Some neurons were reactivated after the inactive period. Thus, the active and inactive periods appeared alternately. We examined how the number of active neurons changed with respect to time steps and found that the number rapidly converged to become constant; only 196.0 ± 7.5 neurons were activated on average in the last 500 steps. This indicates that a constant number of neurons are active at each time step, that is, a time passage is represented based on the sparse coding. This also implies the number of possible active states of  N  that 5 exp(0.5N) √ neurons is approximately ≈ . Here we used Stirling’s formula N/5 √ 2 2π N (n! ≈ 2πnnn e −n ). Hence, the model may be able to represent a sufficiently long time passage. We calculated the total activity of neurons and found that the quantity also rapidly converged to become constant—10.8 ± 0.07 on average for the last 500 steps. The middle panel in Figure 1 shows the similarity index calculated using equation 2.2. Since equation 2.2 takes two arguments of t1 and t2 , we obtained a T × T matrix, where the row and the column were specified by t1 and t2 , respectively. Similarity indices were plotted in a gray scale in which black indicated 0 and white 1. A broad white band appeared diagonally. Since the similarity index at the identical step (t2 = t1 ) takes 1, the diagonal elements of the similarity index appeared white. The similarity index decreased monotonically as the interval between t1 and t2 became longer. However, it remained high when the interval was short. This could be confirmed as shown in the right panel of Figure 1, which shows the plots of C(200, t), C(500, t), and C(800, t), namely, the 200th, 500th, and 800th rows in the similarity index. This result indicates that the activity pattern of neurons changed gradually with time and did not recur. We tested the case of 100,000 steps (i.e., T = 100,000) and confirmed that the similarity index displayed a white diagonal band and dark off-diagonal domains, indicating that the representation of a time passage was stable at least up to 100,000 steps. Thus, our model can represent a time passage by a dynamically changing population of active neurons in the rate-coding scheme.

1036

T. Yamazaki and S. Tanaka

(a) N = 100

(b) N = 500

(c) N= 1000

(d) N= 2000

(e) τ = 10

(f) τ = 50

(g) τ = 100

(h) τ = 200

(i) κ = 0.5

(j) κ = 1.0

(k) κ = 2.0

(l) κ = 3.0

t1t 2 Figure 2: Similarity indices at various parameter settings.

3.2 Parameter Dependency. We studied how the dynamics changed when the value of a parameter was modified. First, we varied N and calculated similarity indices. Other parameters were fixed under the basic parameter setting. For N = 100 (see Figure 2a), the width of the white diagonal band became narrower and jagged, and white off-diagonal domains indicating the recurrence of an activity pattern of neurons appeared. As N increased, the white off-diagonal domain disappeared, although the diagonal band became broad and vague (see Figures 2b–2d). This suggests that the number of neurons should be large for the unique representation of a time passage by activity patterns of neurons. On the other hand, for N = 2000 (see Figure 2d), off-diagonal domains tended to be totally white. Next, we varied τ . For τ = 10, the similarity index became totally white so that the diagonal band did not appear (see Figure 2e). This was because

Neural Modeling of an Internal Clock

1037

for τ = 10, no neurons exhibited the repetition of transition between active and inactive states so that the activity pattern did not change. For τ = 50, the diagonal band appeared but expanded as t1 or t2 increased (see Figure 2f). For larger τ , the diagonal band appeared clearly, and the off-diagonal domains became darker (see Figures 2g and 2h). This suggests that the range of temporal integration should be long for the stable representation of a time passage. Figures 2i to 2l plot similarity indices for various κ values. The diagonal band became narrower and the off-diagonal domains became darker as κ increased, suggesting that a large κ enables the model to stably represent a time passage. As κ increased, black domains appeared on the top and the left sides of the similarity index. The emergence of these domains indicates that all neurons became inactive for the first tens of time steps because of the strong recurrent inhibition due to a large κ. Hence, κ should be within an appropriate range. Varying I did not change the similarity index. Let zic (t) be the activity of neuron i at time t under I = c. For any i and t, we obtained the following relationship: zic (t) = czi1 (t), where {zi1 (t)} is the activities of neurons under the basic setting. This indicates that neuronal activity is scaled by the input strength. Consequently, the similarity index does not depend on I . This scaling property depends on the choice of the nonlinearity. In the model, the scaling property holds because the half-rectification zi (t) = [ui (t)]+ does not saturate the (positive) value of ui (t). In summary, N and τ should be large while κ should be within an appropriate range to generate a sequence of activity patterns of neurons uniquely representing a time passage. 3.3 Analysis of Parameter Dependency. We also analyzed the parameter dependency of the dynamics of our model. We analytically estimated the total activity of neurons, which is given by lim

t→∞


i

zi (t) =

NI . 1 + κτ

(3.1)

The derivation of this formulais found in the appendix. The left panel in Figure 3 shows the plots of i zi (T) obtained by computer simulations and equation 3.1 against τ , and the middle panel the plots of i zi (T) and equation 3.1 against κ. In computer simulations, we confirmed that the value of i zi (t) had already reached the steady state and became constant at t = T. As can be seen, there is excellent agreement between the analytical and simulation results.

1038

T. Yamazaki and S. Tanaka 100

Simulation Theory

80

60

60

40

40

20

0 0

50

100 150 200

0 0

τ

80 40

20

0

Simulation Theory

120

∑izi(t)

∑izi(t)

80

160

Simulation Theory

tinactive

100

0.5

1

1.5

κ

2

0

1

2

3

4

5

κ

 Figure 3:Plots of i zi (T) obtained by computer simulations against τ (left), plots of i zi (T) against κ (middle), and the simulated duration of inactive periods in simulations against κ (right). The solid curves in the left and middle panels indicate the analytical estimation of the total activity given by equation 3.1, and the solid curve in the right panel indicates the analytically estimated duration of inactive periods given by equation 3.2.

Figures 2k and 2l show that all neurons became inactive in the first tens to hundreds of steps when κ was large. We theoretically estimated the duration of the initial inactive period. Let tinactive be the time step when neurons recovered from the initial inhibition and started to become active, that is, the duration of the initial inactive period. When κ < 1, tinactive = 0. On the other hand, when κ ≥ 1, we obtained   tinactive = τ log κ + 1 . (3.2) This derivation is found in appendix. The right panel in Figure 3 plots equation 3.2 and the duration of inactive periods in simulations with different κ’s. The simulation result agrees with the theoretical one in general; however, there is a slight difference when κ is large. The difference could be due to an approximation in the derivation of equation 3.2. We omitted fluctuation terms in the derivation, which might shorten the inactive period when κ was large. Then we studied how the width of the diagonal band is determined. We defined width(t), the width of the diagonal band at time t, by width(t) = arg max (t  > t, C(t, t  ) > θ ),  t

(3.3)

where θ is the threshold of similarity and was set at 0.9. We simplified equation 2.1 and derived the following equation (the derivation is found in appendix):  t I 1 − √κτN
κ
 + ui (t) ≈ δ

wij exp (−(t − s)/τ ) z j (s − 1), N j 1 + κτ 1 − √1 s=1 N

(3.4)

Neural Modeling of an Internal Clock

1039

800

800

width(t)

1000

width(t)

1000

600

N=1000 N=2000 N=3000 N=4000 N=5000

400 200 0 0

20

40

60

N=1000 N=2000 N=3000 N=4000 N=5000

600 400 200 0

80

100

0

200 400 600 800 1000

t

t

1000

N τ

and various N values (left) and the same

κ=0.40 κ=0.45 κ=0.50 κ=0.55 κ=0.60 κ=0.65 κ=1.00

800

width(t)

√

600 400 200 0 0

200 400 600 800 1000

t

4000 3000

width(t)

Figure 4: Plots of width(t) at κ = √ plots at κ = 2 τ N (right).

2000 1000 0 0.3

0.4

κ

0.5

0.6

Figure 5: Plots of width(t) (left) and width(T) against κ (right).

where Pr(δ

wij = 1) = Pr(δ

wij = −1) = 0.5. This equation shows that if √κτN is less than 1, then ui (t) for all i’s tends to be a positive constant due to the first term in equation 3.4, and thus the fluctuation term (the second term in equation 3.4) may not contribute. Hence, width(t) increases as t increases. The left panel of Figure 4 plots width(t) under different N’s, τ = 100, and √ κ = τN . As can be seen, width(t) increases exponentially as t increases. Note that the change in width(t) is invariant regardless of N, suggesting that √κτN controls the dynamics. On the other hand, as √ shown in the right panel of Figure 4, when κ was twice as large (i.e., κ = 2 τ N ), width(t) became constant. Moreover, the change in width(t) is invariant, again regardless of N. These results show that there exists a critical κ value denoted by κ∗ such that if κ < κ∗ then width(t) increases rapidly, whereas if κ > κ∗ , then width(t) remains constant. We varied κ and plotted width(t) with different κ’s, which is shown in the left panel of Figure 5. As can be seen, κ∗ can be estimated at approximately 0.5 when N = 1000 and τ = 100. If κ < 0.5, then width(t) increases rapidly, whereas if κ > 0.5, then width(t) remains constant, and when κ = 0.5, the increase in width(t) becomes linear. Then we estimated κ∗ more accurately. We conducted computer simulations in 5T steps and plotted width(T) against κ in the right panel of Figure 5. At

1040

T. Yamazaki and S. Tanaka

κ < 0.496, width(T) became 4T. Since the total simulation steps were 5T, the upper bound of width(T) was 4T, concluding that the diagonal band reached the maximum. At κ ≈ 0.496, width(T) sharply decreased, and as κ further increased, width(T) became constant. Hence, we could observe phase transition at κ∗ = 0.496. The control parameter √κτN also implies that width(t) increases when N is large, τ is small, or κ is small. This is consistent with the simulation results shown in Figure 2d (for N), Figures 2e and 2f (for τ ), and Figure 2i (for κ). Therefore, although the plots of similarity indices show substantial differences in their shapes, this control parameter provides a unified view of how the shapes differ under different parameter settings. 3.4 Analysis of the Sparseness. These dependency results may imply that much of the variance among similarity indices is accounted for by the level of sparseness of the active neuron population. We examined how much the sparseness could account for the variance. Let r (t) be the ratio of the number of active neurons to N at time t defined as r (t) =

# of active neurons at time t . N

The sparseness is the reciprocal of r (t), that is, smaller r (t) values indicate more sparseness. We conducted simulations to clarify the relationship between width(t) and r (t). Parameters N, τ , and κ were chosen from N ∈ {500, 1000, 2000}, τ ∈ {50, 100, 150, 200}, and κ ∈ {0.5, 1.0, 2.0, 3.0}, and all combinations of N,τ and κ were tested. Thus, we conducted 3 × 4 × 4 = 48 simulations. We did not test the case of N = 100, because the fluctuation of the diagonal band observed in Figure 2a is clearly due to the size effect. Hence, we considered only larger N values: N = 500, 1000, and 2000. We also ignored the case of τ = 10 and tested τ = 150 instead. Then we plotted width(t) against r (t) at t = T in Figure 6. The left panel of Figure 6 shows the plot of width(T) against r (T) for 0 ≤ r (T) ≤ 1. As can be seen, width(T) values for r (T) < 0.4 are small but scattered, while width(T) values for r (T) > 0.4 reach the maximum value of 1000. Moreover, this plot implies that there may be a positive correlation between width(T) and r (T). To verify this, we fitted a linear function to the set of data points for r (T) < 0.4 using a least-square method and drew the regression line. The line has a positive slope, indicating a positive correlation. Next, we examined how the scattering of data points appeared. For each τ ∈ {50, 100, 150, 200}, we selected corresponding data points, fitted a linear function to these points, and drew the line in the right panel of Figure 6. We used the data points for r (T) < 0.4 only. This figure indicates that the scattering of width(T) values is dependent on τ values, that is, width(T) increases with τ . Therefore, the sparseness can be a good measure

Neural Modeling of an Internal Clock

1041

800

800

width(T)

1000

width(T)

1000

600 400

τ=50 τ=100 τ=150 τ=200

600 400 200

200 0

0 0

0.2

0.4

0.6

r(T)

0.8

1

0

0.1

0.2

0.3

r(T)

0.4

0.5

Figure 6: Plots of width(T) against r (T) for 0 ≤ r (T) ≤ 1 with the linear regression line calculated using data points for r (T) ≤ 0.4 (left) and that for 0 ≤ r (T) ≤ 0.4 with regression lines determined using data points corresponding to each τ value (right).

that accounts for the variation among similarity indices, even though τ also plays a critical role in determining the exact value of width(T). 3.5 Representation of Different Time Passages for Different Input Signals. The model can represent different time passages for different input signals by generating different sequences of activity patterns of neurons. To demonstrate this, we extended the input I to Ii such that individual neurons could receive arbitrary signals independently, and we conducted two simulations. In each simulation, Ii ’s were set to 0 or 1 randomly with the probability 0.5. Then we calculated the similarity index between activity patterns in the two simulations. The left and middle panels in Figure 7 show the similarity indices obtained from each simulation, and the right panel represents the similarity index between two sequences of activity patterns generated at different settings of {Ii }. As can be observed, the similarity index for each simulation shows a clear diagonal band, suggesting that a sequence of activity patterns is generated without recurrence. The similarity index between these two sequences, however, is totally black. This indicates that different time passages are generated by different input signals. 3.6 Stability Against Noise in Input Signals. We examined to what extent a sequence of activity patterns of neurons generated by our model was robust against temporal noise. We extended input signals as follows, Ii (t) = I · (1 + η · U[−1, 1]),

(3.5)

where U[−1, 1] denotes a uniform random number in [−1, 1] considered as noise and η represents the noise intensity. Note that different noise was incorporated at each time step.

1042

T. Yamazaki and S. Tanaka

t1t 2 Figure 7: Similarity indices at different settings of {Ii }’s (left and middle) and the similarity index between two sequences (right).

t1t 2 Figure 8: Similarity indices at various η values. From left to right, the η values are 0.01, 0.05, and 0.10, respectively.

The simulation procedure was the same as that in the previous simulation. We tested the cases of η = 0.01, 0.05, and 0.10. For each η value, we (1) (2) generated two sequences of input signal patterns {Ii (t)} and {Ii (t)} and conducted simulations. Then we calculated the similarity indices between the simulations, which are shown in Figure 8. Even with injection of 5% noise, the diagonal band still remained. The diagonal band disappeared at 10% noise injection. 3.7 Stability Against Noise in Connection Matrix. We also examined the robustness of sequence generation against noise in the connection matrix wij . We extended the synaptic weights as follows,  wij (t) =

(1 + η · U[−1, 1]) · wij

wij =

(η · U[0, 1]) ·

otherwise,

1 N

2κ , N

(3.6)

where U[−1, 1] and U[0, 1], respectively, denote uniform random numbers in [−1, 1] and [0, 1] considered as noise and η represents the noise intensity. Note that different noise was incorporated at each time step. The simulation procedure was the same as that in the previous section. Briefly, we varied

Neural Modeling of an Internal Clock

1043

t1t 2 Figure 9: Similarity indices at various η values. From left to right, the η values are 0.3, 0.5, and 0.7, respectively.

t1t 2 Figure 10: Similarity indices for {zi (t)} (left) and for {zi (t + T)} (middle), and the similarity index between {zi (t)} and {zi (t + T)} (right).

η as 0.3, 0.5, and 0.7. For each η value, we conducted two simulations under different random seeds, and then the similarity index between the two sequences generated by the two simulations was calculated. The calculated similarity indices are shown in Figure 9. Even with injection of 50% noise, the diagonal band still remained. The diagonal band disappeared at 70% noise injection. 3.8 Resetting of an Internal Clock. Can we reset the clock in the model? We conducted simulation in 2T steps, applying an input signal in which strong transient input signals I = 3.0 at t = 0 and t = T were added to the sustained signal I . The left and middle panels in Figure 10 are similarity indices for sequences {zi (t)} and {zi (t + T)} for 0 ≤ t ≤ T − 1, respectively. Immediately after the transient injection of a strong signal, all neurons became active once; then they became inactive for the subsequent tens of steps because they received strong inhibitory feedback from other neurons. Hence, similarity indices represent black domains on the top and the left side. After the release from inhibition, neurons started to become active, and a sequence of populations was generated, which is indicated by the white diagonal band. The right panel in Figure 10 is the similarity index between {zi (t)} and {zi (t + T)}, which also shows a white diagonal band.

1044

T. Yamazaki and S. Tanaka

100

I(0)=I+∆I I(0)=I

∑izi(t)

80 60 40 20 0 0

200

400

t

600

800 1000

 Figure 11: Plots of i zi (t) when the transient signal was added (I (0) = I + I ) and not added (I (0) = I ).

This index shows that the sequence of activity patterns generated at t ≥ T was almost identical to that generated at t ≥ 0. This suggests that a transient component of the input signal can reset the internal  clock. Does the transient signal affect the total activity i zi (t)? If the activity markedly decreases after receiving the signal and does not return to the normal calculation using equation 2.4, the model may not be adequate for the internal clock. We plotted in Figure 11 the total activity of neurons i zi (t) when the transient signal was added and not added. As seen, during the inactive period, the total activity was 0, and the activity returned to normal immediately after the end of the inactive period, indicating that the transient signal does not affect the level of the total activity. 3.9 Speeding Up and Slowing Down an Internal Clock. We examined the similarity between sequences of activity patterns of neurons generated using different values of τ or κ. For each case, we conducted two computer simulations: in one simulation, the value of either τ or κ was altered, and in the other, the basic setting was used. Then we calculated the similarity (1) index between the sequence {zi (t)} in the altered case and the sequence (2) {zi (t)} in the basic case. The left panel in Figure 12 shows the similarity index for the case in which τ was set at 90. As can be seen, the white band representing high similarity slanted slightly above the diagonal line. The degree of the slant depended on the difference of values of τ . If we set τ at 80, the slant increased, as shown in the middle panel of Figure 12. However, the length of the white band became shorter, which suggests that a long time passage could not be represented. The right panel in Figure 12 shows the similarity index in the second case, in which κ was at 1.5. We can see that the white band also slanted less than the diagonal line, and the band became shorter. The black

Neural Modeling of an Internal Clock

1045

t1t 2 Figure 12: Similarity indices in cases of decreasing τ to 90 (left) and 80 (middle), and increasing κ to 1.5 (right).

region on the top edge represents the period in which all neurons became inactive due to the strong inhibition by the large κ. These results indicate that the clock speed increased with a change in parameter τ or κ.

4 Discussion 4.1 The Model. We studied a random recurrent inhibitory network of an internal clock. Individual neurons exhibited the random repetition of transition between active and inactive states following the stimulus onset due to the randomness incorporated in the recurrent connections, and the active and inactive periods were sustained for several tens to hundreds of steps. Hence, the population of active neurons changed gradually with time and did not recur. This property was confirmed by calculating the similarity index. Both the number of active neurons and the total activity of neurons remained constant. This is an important property for a stable time representation because if the number of active neurons varies at each time step, there may exist a time step at which no neurons become active. Moreover, if a population of active neurons at a time step is a subset of another population of active neurons at another time step, then it is impossible to distinguish these two time steps from each other in the learning of a timing discussed in section 4.2. This also indicates that a time passage is represented based on the sparse coding, suggesting that the model may be able to generate a sequence of neuronal activity patterns in which the length is the exponential of the number of neurons. Noise stability was also discussed, and sufficient stability for up to 5% uniform noise in the input signals and 50% uniform noise in the connection matrix were demonstrated. Our internal clock could be reset by a strong transient input added to the sustained signal. Accuracy is an important issue for the representation of time. Accuracy in our framework may be estimated by the length of successive time steps represented by one single activity pattern of neurons, because when neurons reveal an activity pattern, time steps represented by this activity pattern

1046

T. Yamazaki and S. Tanaka

cannot be distinguished from each other. If successive time steps are represented by a single activity pattern, the similarity index between two time steps within the range is high. Thus, the range is equivalent to the width of the white diagonal band in the similarity index. Therefore, width(t) defined in equation 3.2 represents the accuracy of time representation in our framework. As discussed in section 3.3, the change in width(t) depends on the parameter √κτN . If this is small, then width(t) increases exponentially, hence, accuracy deteriorates as t increases, whereas if this is large, width(t) remains constant and accuracy is preserved. Therefore, depending on this parameter, the model can represent an accurate internal clock as well as an internal clock in which the accuracy decreases with time. Psychophysical experiments have shown a relationship between time interval and accuracy known as Weber’s law (Gibbon, Malapani, Dale, & Gallistel, 1997), which states that as time interval t increases, the standard deviation of reported intervals over trials increases linearly, while the average of reported intervals is equal to t. That is, the accuracy of the representation of time t decreases linearly as t increases. The accuracy in our model linearly decreases only at the critical point κ = κ∗ . Hence, strictly speaking, this model does not satisfy Weber’s law. Nevertheless, the left panel in Figure 5 demonstrates that at κ = 0.5, width(t) seems increasing linearly with respect to t, implying that Weber’s law is satisfied in the vicinity of the critical point for finite time steps even in this model. Furthermore, we tested whether the model could change the clock speed. It has been shown that the basal ganglia is involved in time estimation, specifically, for longer time events (>3 sec) in motor planning (Ivry, 1996; Lalonde & Hannequin, 1999). Some studies have shown that in the basal ganglia, the facilitation of dopamine secretion from substantia nigra pars compacta speeds up the internal clock, while the suppression or blockade of dopamine secretion decreases the speed (Meck, 1996). Since modulatory neurotransmitters change neuronal excitability (Missale, Nash, Robinson, Jaber, & Caron, 1998; Kandel, Schwartz, & Jessel, 2000), we considered varying τ or κ. Then we obtained similarity indices in which the white band deviated from the diagonal line. These results suggest that the change in neuronal activity patterns is speeded up or slowed down, hence the speed of the internal clock is also. 4.2 Learning of a Timing in the Cerebellum. Pavlovian eyelid conditioning is a clear and experimentally traceable example of the representation of a time passage (Mauk & Donegan, 1997). This experiment involves repeated presentations of a conditional stimulus (CS) paired with an unconditional stimulus (US) delayed for a certain time after the CS onset. Typically, a neutral stimulus such as a tone serves as the CS and an air puff directed to one eye as the US. With this training, the subject not only learns to close its eye (conditioned response, CR) in response to the CS but also learns to elicit the CR at the same time as the interstimulus interval (ISI)

Neural Modeling of an Internal Clock

1047

GOLGI

LTD

PARALLEL FIBERS

PURKINJE

GRANULE

MOSSY FIBERS

SYNAPSES EXCITATORY INHIBITORY

CS PONTINE NUCLEUS

CR

CEREBELLAR NUCLEUS CLIMBING FIBER

US INFERIOR OLIVE

Figure 13: Schematic of known cell types and synaptic connections in the cerebellum. Shown in bold letters and solid lines are the components corresponding to our model.

between the CS and US onsets during the conditioning, indicating that the time passage from the CS onset to the US onset is learned. Evidence from electrophysiological and behavioral studies demonstrates that the Pavlovian eyelid conditioning is mediated by the cerebellum working as an internal clock (Ivry, 1996). Figure 13 represents the schematic of known cell types and synaptic connections in the cerebellum (Eccles, Ito, & Szent´agothai, 1967; Ito, 1984). The neural signal of the CS comes from the pontine nucleus through mossy fibers to the cerebellar nucleus, which attempts to elicit spike activity. The CS signal is also sent to Purkinje cells relayed by granule cells, and then Purkinje cells inhibit the cerebellar nucleus. Thus, the cerebellar nucleus receives both excitatory inputs directly through mossy fibers and inhibitory inputs from Purkinje cells. Golgi cells are inhibitory neurons; they receive excitatory inputs from granule cells through parallel fibers and inhibit granule cells. The connections between granule and Golgi cells are recurrent and bipartite; there is no direct connection from a granule cell to another granule cell and from a Golgi cell to another Golgi cell. The neural signal of the US comes from the inferior olive through climbing fibers to Purkinje cells. The ISIs between the CS and US onsets are learned by long-term depression (LTD) of parallel fiber terminals at Purkinje cells (Ito, 1989). Before LTD is induced, the cerebellar nucleus receives the CS signal through mossy fibers and attempts to elicit the spike activity. The nucleus, however, is inhibited by Purkinje cells that are excited by granule cells; hence, the nucleus cannot elicit spikes. When the US signal arrives at a Purkinje cell through a climbing fiber, with conjunctive stimulation through parallel fibers, the

1048

T. Yamazaki and S. Tanaka 1

1

1

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0

0 0

200

400

t

600

800

1000

0 0

200

400

t

600

800

1000

0

200

400

t

600

800

1000

Figure 14: Plots of 0.1Net input (t) (dashed line) and C(t1 , t) (solid line) for t1 = 200, 500, 800 (left to right).

LTD occurs at the parallel fiber terminals at the Purkinje cell. Thereby, the net input from granule cells to the Purkinje cell decreases. After the LTD is established, the nucleus is released from inhibition by the Purkinje cell and can elicit the CR. Our model described in equation 2.1 can be derived from dynamical equations of granule and Golgi cells; the derivation is in the appendix. Hence, the model may be able to play a role of the cerebellar internal clock. Let us examine whether our model can be used for the learning of ISI between the onsets of the CS and US on the basis of the above scenario. Let wiPF be the synaptic weight between granule cell i and the target Purkinje cell, which takes 1 for any i initially. We assumed that the US signal arrived at the Purkinje cell at t = 200 and set wiPF = 0 for i ∈ {i|zi (200) > 0} (neurons that were active at t = 200) due to the LTD of parallel fiber terminals. The net input to the Purkinje cell at time t was calculated according to
Net input(t) = wiPF zi (t). i

We also tested the cases of t = 500 and 800. The left panel in Figure 14 shows the plot of Net input(t) and C(200, t). The value of Net input(t) was rescaled by multiplying 0.1 so as to be comparable to [0, 1]. At t = 1, this value reached 0.1N because inhibition was not effective at this step (out of range in this plot). Then it decreased gradually as t increased until t = 200 and became 0 at t = 200, after which the value slowly increased and became constant. Similar results were obtained in cases of t = 500 and 800 (the middle and right panels in Figure 14). These results indicate that our internal clock works for the learning of ISI. Note that the shape of Net input(t) became the inverse of the similarity index at time t, suggesting that the success of learning coincides with the appearance of a clear white band in the similarity index. The model can address several aspects in Pavlovian eyelid conditioning. Mauk & Ruiz (1992) demonstrated that a differential conditioning procedure in which discriminable CSs are individually paired with a US at different ISIs can promote the concurrent acquisition of differently timed

Neural Modeling of an Internal Clock

1049

CRs in an identical animal. Discriminable CSs used by Mauk & Ruiz (1992) may correspond to the different patterns of input signals in our model, and the model could represent different time passages for the different patterns as shown in Figure 7. Aitkin & Boyd (1978) reported that neurons in the lateral pontine nuclei, which project mossy fiber inputs to the cerebellum and is a source of the CS signal, show strong transient responses at the onset of a pure tone presentation followed by sustained activities during the presentation. They found that three-fourths of the mossy fibers showed transient activity. We simulated the 3:1 ratio of transient:sustained activity strength of the afferent input signal and successfully reproduced the identical sequence of activity patterns of neurons. The cerebellum is modulated by dopaminergic inputs. Specifically, dopaminergic fibers from the ventral tegmental area to the cerebellar cortex terminate in the granular layer (in which granule and Golgi cells are located) of the crus I and crus II ansiform lobules (Ikai, Takada, Shinonaga, & Mizuno, 1992). It was also reported that these areas are essential for the normal learning of Pavlovian eyelid conditioning (Hardiman & Yao, 1992). Although there has been no report of whether the cerebellar internal clock is speeded up or slowed down by these neurotransmitters, they might modulate the speed of the cerebellar internal clock. 4.3 Relationships with Other Models. While the model utilized random recurrent inhibitory connections corresponding to recurrent connections between granule and Golgi cells for the generation of a cerebellar internal clock, several models based on different mechanisms have been proposed to date. Moore, Desmond, and Berthier (1989) proposed a tappeddelay-line model. They assumed an anatomical structure of mossy fibers such that they fire one after another with a certain delay after the onset of mossy fiber signals. Due to such an explicit delay mechanism, they proposed that a firing event of each granule cell represents a time passage after the signal onset. However, no such anatomical structure among mossy fibers has been found. Gluck, Reifsnider, and Thompson (1990) assumed oscillating granule cells with different frequencies. Their model is analogous to the Fourier series expansion, on the point that any timing can be represented by a population of neurons. However, they had to assume synaptic weights of complex values to calculate the Fourier series expansion. Hence, their model seems unrealistic. Chapeau-Blondeau and Chauvet (1991) proposed a model in which parallel fibers play the role of an explicit delay mechanism so that the transmission of granule cell activities is delayed. However, as demonstrated by Vos, Maex, Volny-Luraghi, and De Schutter (1999), Golgi cells receiving the same parallel fiber input fire synchronously, suggesting that parallel fibers do not generate delay. Bullock, Fiala, and Grossberg (1994) have proposed a spectral timing model using an intracellular mechanism of delay generation. However, to generate such delay, they needed to assume that the number of Golgi cells is equal to that of granule cells so

1050

T. Yamazaki and S. Tanaka

that there is one-to-one correspondence between granule and Golgi cells. Biologically, a granule/Golgi cell ratio is estimated to be about 1000 on the basis of a granule/Purkinje cell ratio of 274 (Harvey & Napper, 1991) and a Purkinje/Golgi ratio of 3.22 (Palkovits, Magyar, & Szent´agothai, 1971). Later Fiala, Grossberg, and Bullock (1996) proposed another mechanism of delay generation in Purkinje cells on the basis of the slow process of intracellular transduction by the metabotropic glutamate receptor (mGluR), by which their model can generate delay up to 4 secs. Moreover, such slow processes have been reported (Batchelor, Madge, & Garthwaite, 1994; Dzubay & Otis, 2002) and glutamate uptake controls the length of delay (Reichelt & Knopfel, ¨ 2002). However, delay reported so far is at most 700 msec, which is insufficient to account for Pavlovian eyelid conditioning because the conditioning is successful during the ISI is up to 2 to 3 secs (Mauk & Donegan, 1997). Also, the 4 sec delay generated by their model is dependent on a parameter that does not have biological counterparts. Rather, they chose parameters so as to obtain 4 sec delay. Buonomano and Mauk (1994) were the first to focus on the recurrent loop between granule and Golgi cells as the device of time coding. To demonstrate how the recurrent circuit works, they modeled the circuit based on many biological properties, such as neural circuitry, cell ratio, the synaptic convergence-divergence ratio, realistic neuron models with leaky membranes, and realistic cell parameters. Then the model generated a temporal code based on the population of active neurons. Later the model was extended to the complete cerebellar circuit and could reproduce the experimental results of Pavlovian eyelid conditioning (Medina, Garcia, Nores, Taylor, & Mauk, 2000; Medina & Mauk 2000). However, despite the completeness of the realistic modeling, the mechanism of generating such a temporal code was still unclear due to the complexity of the realistic model. In contrast to their realistic modeling, our approach focused on the simplicity of the model. This simplicity made possible a degree of mathematical and numerical analysis of how and why this model could generate a temporal code. We could clarify the mechanism of time coding; that is, randomness in recurrent connections plays the key role. Moreover, we could investigate the dynamics of the model in depth and could demonstrate some properties, such as parameter dependency, noise stability, resetability, and ability, to produce different patterns for different input signals. Our simplified modeling approach had the advantage of enabling mathematical and numerical analyses, but potentially at the cost of realism. We have started a realistic modeling of the cerebellar circuit based on conductance-based leaky integrate-and-fire neuron models with large decay constants of excitatory postsynaptic potentials (EPSP) at synapses from granule to Golgi cells and inhibitory postsynaptic potentials (IPSP) at synapses from Golgi to granule cells. Such large time constants are evident; the NMDA component of EPSP can be fitted by two exponential functions in

Neural Modeling of an Internal Clock

1051

which the time constants are 31 msec (33%) and 170 msec (67%) (Dieudonn´e, 1998), and the GABAA component of IPSP by three exponential functions in which the time constants are 8.0 msec (22%), 33.5 msec (58%), and 102.9 msec (20%) (Brickley, Cull-Candy, & Farrant, 1999). To date, we have confirmed that the dynamics of the realistic model is quite similar to that of the simple model. It is not surprising, because Ermentrout (1994) reported the equivalence of conductance-based models to rate models when the conductance change caused by incoming spikes is sufficiently slow. Rabinovich et al. (2001) proposed a theoretical framework of a temporal coding called winnerless competition in which the dynamics of neurons is described by the Lotka-Volterra equation. This model slightly differs from ours on some points, although the temporal changes of active neurons are commonly caused by asymmetric recurrent inhibitory connections. In their approach, the response profile of one neuron, which is temporally complex, is assumed to carry information of a stimulus. In our approach, the temporal profile of one neuronal response does not carry this information. Rather, the temporal change of active neuronal populations correlates with a time passage. Hence, whereas in their model, the number of active neurons can vary with time because the response profiles are important, in our model the number should not vary for stable representation of a time passage as discussed in section 4.1. In their seminal papers, Marr (1969) and Albus (1971) had paid attention to the recurrent circuit between granule and Golgi cells as the device for the sparse representation of spatial information called codon representation (Marr, 1969) or expansion recoding (Albus, 1971). They did not, however, study the ability for time coding in the cerebellum. We extended their idea of sparse representation to spatiotemporal information processing. Li and Hopfield (1989) studied the dynamics of the network composed of mitral and granule cells in the olfactory bulb. Mitral cells are excitatory cells that receive odor signals and excite granule cells, which inhibit mitral cells in turn. In this way, the mitral-granule cells network is a recurrent inhibitory circuit like the cerebellar granule–Golgi cells network. Hence, their dynamical model becomes similar to ours. However, they focused not on the time coding but the oscillatory responses of mitral cells. Bugmann (1998) and Okamoto and Fukai (2001) proposed models representing a single time interval and satisfying Weber’s law. These models have symmetric recurrent connections and hence a steady state. A single time interval is represented by the length of the relaxation process required to attain the steady state. On the contrary, these models do not work as an internal clock because they cannot represent multiple time intervals. Our internal clock model is simple and derivable from the well-known structure of the cerebellum. Its simplicity helps us to clarify the mechanism and the function of the internal clock, while its realistic design may justify the biological reality.

1052

T. Yamazaki and S. Tanaka

Appendix A.1 Derivation of Equation 3.1 From equation 2.1, it follows that ui (t) ≈




I 1 wij z j (t − 1), + 1− ui (t − 1) − κ τ τ j

and by considering the continuum limit, we obtain τ


dui (t) wij z j (t). = −ui (t) + I − κτ dt j

Let ui and zi be ui (t) and zi (t) at the steady state, respectively. Since at the steady state, then ui = I − κτ




dui (t) dt

=0

wij z j ,

j

and hence  zi = F

I − κτ




 wij z j ,

j

where F (x) = max(x, 0). Since {wij } has the same number of 0’s and 2/Ns on average, if this equation can be linearized at the steady state, the mean field solution is I 1
zi = N i 1 + κτ and equation 3.1 is derived. def 1 N

A.2 Derivation of Equation 3.2. We calculate q (t) = steady state when N → ∞, which is q (t) = I − κ

 i



t−1−s exp − q (s). τ s=0

t−1


From the definition, it follows that q (0) = 0 and q (1) = I . Then,



1 1 q (2) = I − κ exp − I = I 1 − κ exp − . τ τ

zi (t) at the

Neural Modeling of an Internal Clock

1053

If q (2) > 0, then the inactive period stops. On the other hand, if q (2) ≤ 0, then we calculate q (3) as

2 q (3) = I 1 − κ exp − . τ Inductively, tinactive can be estimated as  tinactive =

    >0 arg mint 1 − κ exp − t−1 τ

κ ≥ 1,

0

otherwise.

When κ ≥ 1, it follows that



t−1 tinactive = arg min 1 − κ exp − t τ



>0

= arg min(t > τ log κ + 1) t

= τ log κ + 1, and equation 3.2 is derived.  def A.3 Derivation of Equation 3.4. We recalculate q (t) = N1 i zi (t) when N is finite but sufficiently large. A similar calculation carried out in the derivation of equation. 3.2 yields

t−1−s exp − q (s) τ s=0

t−1
1

t−1−s + κ δwij exp − z j (s). N i j τ s=0

q (t) ≈ I − κ

t−1


(A.1)

 2 , Since N1 i δwij is a gaussian variable with zero mean and variance (1/N) N an upper bound of this term may be N√1 N . Then, instead of equation A.1, we consider the following equation:

t−1−s exp − q ∗ (s) τ s=0

t−1 t−1−s κ
exp − q ∗ (s) +√ τ N s=0




t−1 1 t−1−s = I −κ 1− √ exp − q ∗ (s). τ N s=0

q ∗ (t) = I − κ

t−1


1054

T. Yamazaki and S. Tanaka

Thus, q ∗ (t) may be regarded as an upper bound of q (t). A similar calculation carried out in the derivation of equation 3.1 yields q ∗ (t) =

I  1 + κτ 1 −

√1 N

.

Notice that q ∗ (t) → q (t) when N → ∞. We assume that |q ∗ (t) − q (t)| is sufficiently small and hence q ∗ (t) ≈ q (t). Then, by incorporating this into equation 2.1, we have

t−1−s exp − z j (s) τ j s=0

t−1

t−1−s δwij exp − z j (s) +κ τ j s=0

t−1

t−1−s τκI  +κ δwij exp − z j (s) ≈I− τ 1 + κτ 1 − √1N j s=0 

t−1 I 1 − √κτN

t−1−s  +κ = δwij exp − z j (s). τ 1 + κτ 1 − √1N j s=0

ui (t) = I − wκ

t−1



A.4 Derivation of Equation 2.1 from the Cerebellar Circuit. We consider N granule cells and K Golgi cells. Each granule cell receives mossy fiber signal and sends its output signal to Golgi cells with excitatory synapses. In turn, each Golgi cell sends its output signal to granule cells with inhibitory synapses. Let ziGr (t) and zkGo (t) be activities of granule cell i and Golgi cell k at time t, respectively. They are given as  + ziGr (t) = uiGr (t)  + zkGo (t) = uGo k (t) , where uiGr (t) and uGo k (t) are internal states of granule cell i and Golgi cell i at time t, respectively, and [x]+ = x if x > 0 and 0 otherwise. uiGr (t) and uGo k (t) are calculated using the following equations, τ Gr u˙ iGr = −uiGr + I − Go τ Go u˙ Go k = −uk +


j




Gr←Go Go wi,k zk (t),

k Go←Gr Gr wk, z j (t), j

Neural Modeling of an Internal Clock

1055

where τ Gr and τ Go are decay time constants, respectively, and I represents Gr←Go the afferent input signal conveyed through mossy fibers. wi,k denotes the Go←Gr synaptic weight from Golgi cell k to granule cell i, and wk, j the synaptic weight from granule cell j to Golgi cell k. They are solved as

t uiGr (t) = I 1 − exp − Gr τ −

uGo k (t) =

1
τ Gr

j

 Go←Gr wk, j

t

0

k

1
τ Go

 Gr←Go wi,k

t

0

t−r exp − Gr zkGo (r )dr, τ



t−s exp − Go zGr j (s)ds. τ

(A.2)

(A.3)

Since Golgi cells receive only excitatory inputs, uGo k (t) is always positive, that is,  + = uGo zkGo (t) = uGo k (t) k (t). Using this relationship and incorporating equation A.3 into equation A.2, we obtain



1 t Gr←Go Go←Gr uiGr (t) = I 1 − exp − Gr − Gr Go wi,k wk, j τ τ τ j k



 t r r − s Gr t−r × exp − Gr exp − Go z j (s)dsdr τ τ 0 0

t = I 1 − exp − Gr τ



 t r
1 t−r r −s − Gr wij exp − Gr exp − Go zGr j (s)dsdr, τ τ τ 0 0 j (A.4) where def

wij =

1
τ Go

Gr←Go Go←Gr wi,k wk, j .

k

We assume the temporal integration of activities of neurons over a long time, which is represented by the summation with respect to s, where τ Go determines the integration range. This assumption is based on biological

1056

T. Yamazaki and S. Tanaka

findings. Some in situ experiments showed that upon the injection of depolarizing current pulses of the same amplitude, Golgi cells fire much less frequently than granule cells: 0.07 HzpA−1 for Golgi cells (Dieudonn´e, 1998) and 2.94 HzpA−1 for granule cells (D’Angelo, Filippi, Rossi, & Taglietti, 1995). Since current pulses are injected continuously, this decreased firing frequency of Golgi cells may indicate that input signals are integrated in the time interval between two successive spikes. The large difference in firing rate per input current suggests τ Gr τ Go . Specifically, we assume τ Gr 1 τ Go . Then we regard τ 1Gr exp − t−r as the delta function δ(t − r ) τ Gr   and exp − τ tGr ≈ 0. It follows that uiGr (t) = I −






t

wij 0

j

t−s exp − Go zGr j (s)ds. τ

Finally, by replacing uiGr (t),ziGr (t),τ Go simply with ui (t), zi (t), τ , and discretizing the time step, we obtain ui (t + 1) = I −


j



t−s wij exp − z j (s), τ s=0 t


and equation 2.1 is derived. References Aitkin, L. M., & Boyd, J. (1978). Acoustic input to the lateral pontine nuclei. Hearing Research, 1, 67–77. Albus, J. S. (1971). A theory of cerebellar function. Mathematical Biosciences, 10, 25–61. Batchelor, A. M., Madge, D. J., & Garthwaite, J. (1994). Synaptic activation of metabotropic glutamate receptors in the parallel fibre–Purkinje cell pathway in rat cerebellar slices. Neuroscience, 63(4), 911–915. Brickley, S. G., Cull-Candy, S. G., & Farrant, M. (1999). Single-channel properties of synaptic and extrasynaptic GABAa receptors suggest differential targeting of receptor subtypes. Journal of Neuroscience, 19, 2960–2973. Bugmann, G. (1998). Towards a neural model of timing. BioSystems, 48, 11–19. Bullock, D., Fiala, J. C., & Grossberg, S. (1994). A neural model of timed response learning in the cerebellum. Neural Networks, 7(6/7), 1101–1114. Buonomano, D. V., & Mauk, M. D. (1994). Neural network model of the cerebellum: Temporal discrimination and the timing of motor responses. Neural Computation, 6, 38–55. Chapeau-Blondeau, F., & Chauvet, G. (1991). A neural network model of the cerebellar cortex performing dynamic associations. Biological Cybernetics, 65, 267–279. D’Angelo, E., Filippi, G. D., Rossi, P., & Taglietti, V. (1995). Synaptic excitation of individual rat cerebellar granule cells in situ: Evidence for the role of NMDA receptors. Journal of Physiology, 484(2), 397–413.

Neural Modeling of an Internal Clock

1057

Dieudonn´e, S. (1998). Submillisecond kinetics and low efficacy of parallel fibre-Golgi cell synaptic currents in the rat cerebellum. Journal of Physiology, 510(3), 845– 866. Dzubay, J. A., & Otis, T. S. (2002). Climbing fiber activation of metabotropic glutamate receptors on cerebellar Purkinje neurons. Neuron, 36, 1159–1167. Eccles, J. C., Ito, M., & Szent´agothai, J. (1967). The cerebellum as a neuronal machine. New York: Springer-Verlag. Ermentrout, B. (1994). Reduction of conductance-based models with slow synapses to neural nets. Neural Computation, 6, 679–695. Fiala, J. C., Grossberg, S., & Bullock, D. (1996). Metabotropic glutamate receptor activation in cerebellar Purkinje cells as substrate for adaptive timing of the classically conditioned eye-blink response. Journal of Neuroscience, 16(11), 3760–3774. Gibbon, J., Malapani, C., Dale, C. L., & Gallistel, C. (1997). Toward a neurobiology of temporal cognition: Advances and challenges. Current Opinion in Neurobiology, 7, 170–184. Gluck, M. A., Reifsnider, E. S., & Thompson, R. F. (1990). Adaptive signal processing and the cerebellum: Models of classical conditioning and VOR adaptation. In M. A. Gluck & D. E. Rumelhart (Eds.), Neuroscience and connectionist theory (pp. 131–186). Hillsdale, NJ: Erlbaum. Hardiman, M. J., & Yao, C. H. (1992). The effect of kainic acid lesions of the cerebellar cortex on the conditioned nictitating membrane response in the rabbit. European Journal of Neuroscience, 4, 966–980. Harvey, R. J., & Napper, R. M. A. (1991). Quantitative studies of the mammalian cerebellum. Progress in Neurobiology, 36, 437–463. Ikai, Y., Takada, M., Shinonaga, Y., & Mizuno, N. (1992). Dopaminergic and nondopaminergic neurons in the ventral tegmental area of the rat project, respectively, to the cerebellar cortex and deep cerebellar nuclei. Neuroscience, 51(3), 719– 728. Ito, M. (1984). The cerebellum and neuronal control. New York: Raven Press. Ito, M. (1989). Long-term depression. Annual Review of Neuroscience, 12, 85–102. Ivry, R. B. (1996). The representation of temporal information in perception and motor control. Current Opinion in Neurobiology, 6, 851–857. Kandel, E. R., Schwartz, J. H., & Jessel, T. M. (2000). Principles of neural science (4th ed.). New York: McGraw-Hill. Lalonde, R., & Hannequin, D. (1999). The neurobiological basis of time estimation and temporal order. Reviews in the Neurosciences, 10, 151–173. Li, Z., & Hopfield, J. J. (1989). Modeling the olfactory bulb and its neural oscillatory processing. Biological Cybernetics, 61, 379–392. Marr, D. (1969). A theory of cerebellar cortex. Journal of Physiology, 202, 437–470. Mauk, M. D., & Donegan, N. H. (1997). A model of Pavlovian eyelid conditioning based on the synaptic organization of the cerebellum. Learning and Memory, 3, 130–158. Mauk, M. D., & Ruiz, B. P. (1992). Learning-dependent timing of Pavlovian eyelid responses: Differential conditioning using multiple interstimulus intervals. Behavioral Neuroscience, 106(4), 666–681. Meck, W. H. (1996). Neuropharmacology of timing and time perception. Cognitive Brain Research, 3, 227–242.

1058

T. Yamazaki and S. Tanaka

Medina, J. F., Garcia, K. S., Nores, W. L., Taylor, N. M., & Mauk, M. D. (2000). Timing mechanisms in the cerebellum: Testing predictions of a large-scale computer simulation. Journal of Neuroscience, 20(14), 5516–5525. Medina, J. F., & Mauk, M. D. (2000). Computer simulation of cerebellar information processing. Nature Neuroscience, 3, 1205–1211. Missale, C., Nash, S. R., Robinson, S. W., Jaber, M., & Caron, M. G. (1998). Dopamine receptors: From structure to function. Physiological Reviews, 78, 189–225. Moore, J. W., Desmond, J. E., & Berthier, N. E. (1989). Adaptively timed conditioned responses and the cerebellum: A neural network approach. Biological Cybernetics, 62, 17–28. Okamoto, H., & Fukai, T. (2001). Neural mechanism for a cognitive timer. Physical Review Letters, 86(17), 3919–3922. Palkovits, M., Magyar, P., & Szent´agothai, J. (1971). Quantitative histological analysis of the cerebellar cortex in the cat. II. Cell numbers and densities in the granular layer. Brain Research, 32, 13–32. Rabinovich, M., Volkovskii, A., Lecanda, P., Huerta, R., Abarbanel, H. D. I., & Laurent, G. (2001). Dynamicsl encoding by networks of competing neuron groups: Winnerless competition. Physical Review Letter, 87(6), 068102. Reichelt, W., & Knopfel, ¨ T. (2002). Glutamate uptake controls expression of a slow postsynaptic current mediated by mGluRs in cerebellar Purkinje cells. Journal of Neurophysiology, 87, 1974–1980. Vos, B. P., Maex, R., Volny-Luraghi, A., & De Schutter, E. (1999). Parallel fibers synchronize spontaneous activity in cerebellar Golgi cells. Journal of Neuroscience, 19(RC6), 1–5.

Received March 17, 2004; accepted October 1, 2004.

LETTER

Communicated by Klaus Obermayer

Mirror Symmetric Topographic Maps Can Arise from Activity-Dependent Synaptic Changes Reiner Schulz [email protected]

James A. Reggia [email protected] Departments of Computer Science and Neurology, UMIACS, University of Maryland, College Park, MD 20742, U.S.A.

Multiple adjacent, roughly mirror-image topographic maps are commonly observed in the sensory neocortex of many species. The cortical regions occupied by these maps are generally believed to be determined initially by genetically controlled chemical markers during development, with thalamocortical afferent activity subsequently exerting a progressively increasing influence over time. Here we use a computational model to show that adjacent topographic maps with mirror-image symmetry can arise from activity-dependent synaptic changes whenever the distribution radius of afferents sufficiently exceeds that of horizontal intracortical interactions. Which map edges become adjacent is strongly influenced by the probability distribution of input stimuli during map formation. Our results suggest that activity-dependent synaptic changes may play a role in influencing how adjacent maps become oriented following the initial establishment of cortical areas via genetically determined chemical markers. Further, the model unexpectedly predicts the occasional occurrence of adjacent maps with a different rotational symmetry. We speculate that such atypically oriented maps, in the context of otherwise normally interconnected cortical regions, might contribute to abnormal cortical information processing in some neurodevelopmental disorders. 1 Introduction Multiple adjacent, roughly mirror-image topographic maps are commonly observed experimentally in the sensory neocortex of many species.1 Familiar examples of adjacent mirror image cortical maps include multiple representations of the body surface in primary somatosensory cortex of monkeys as illustrated in Figure 1A (Merzenich, Kaas, Sur, & Lin, 1978; Sur, Nelson, & 1

We are concerned solely with topographic map formation, in which a roughly pointto-point “image” of a receptive surface forms on the cortical surface, and not with “feature maps,” such as those involving orientation sensitivity. Neural Computation 17, 1059–1083 (2005)

© 2005 Massachusetts Institute of Technology

1060

R. Schulz and J. Reggia

Figure 1: (A) A cortical map in the somatosensory region SI of squirrel monkey, composed of adjoining areas 3b and 1, based on multiunit microelectrode readings. The arrow pairs indicate that the map consists of two complete, nearly topographic representations of the body surface that are roughly mirror images of each other, where the axis of reflection corresponds to the border between areas 3b and 1. From (Sur et al., 1982, with permission). (B) Architecture of the model cortical region used in this and previous studies. An input pattern x
encoding the stimulation of a point on the sensory surface is modulated by afferent synaptic strengths W to produce an activation pattern y
over a region of cortical elements representing the neocortical surface. During learning of W, a map of the input patterns and hence, assuming a suitable encoding is used, of the underlying sensory surface forms in the cortical region.

Kaas, 1982) and several mirror-image tonotopic maps in primary auditory cortex (Heschl’s gyrus) in humans (Engelien et al., 2002; Formisano, Kim, & Salle, 2003). If one considers not only primary but also secondary sensory cortex (which also receives thalamocortical projections), numerous other mirror-image maps have been found in somatosensory (Beck, Pospichal, & Kaas, 1996; Krubitzer & Calford, 1992; Krubitzer, Clarey, Tweedale, Elston, & Calford, 1995; Nelson, Sur, Felleman, & Kaas, 1980), visual (Drager, 1975; Newsome, Maunsell, & van Essen, 1986; Sereno et al., 1995; Tiao &

Symmetric Maps

1061

Blakemore, 1976), and auditory (Imig, Real, & Brugge, 1986; Pantev et al., 1995; Talavage, Ledden, & Benson, 2000) cortex in a variety of species. In addition, mirror-image movement representations have been found in the motor cortex of the macaque monkey (Gentilucci et al., 1989). At present, the initial parcellation of cortex into multiple regions or areas is generally believed to be due to genetically determined chemical markers and independent of thalamocortical afferent activity (Sur & Leamey, 2001). However, it remains less clear as to why partially redundant cortical maps occur, why they are so often oriented with reflection symmetry, and what role thalamocortical activity plays in their formation. Multiple adjacent maps are often hypothesized to arise during development due to genetically mediated chemical gradients (Grove & Tomomi, 2003; Levitt, 2000; Zhou & Black, 2000). They are sometimes conjectured to have evolved due to genetic mutations (Allman & Kaas, 1971; Krubitzer, 1995), and it has been suggested that they may provide advantages due to separation of spatial and temporal processing, parallel processing of different sensory attributes, minimization of connection distances, and other factors (Kaas, 1988; Cowey, 1981; Jones, 1990). There has been less speculation as to why such maps often exhibit reflection symmetry, and the relative contributions during map formation of activity-dependent versus activity-independent mechanisms remain the source of some debate, even for individual maps (Grove & Tomomi, 2003; Cohen-Cory, 2002). Past computational models of self-organizing neocortical topographic maps (Kohonen, 2001; Ritter, Martinetz, & Schulten, 1992; Sutton, Reggia, Armentrout, & D’Autrechy, 1994; Sirosh & Miikkulainen, 1994) have generally been limited to single maps and thus do not shed substantial light on the issue of forming multiple mirror-image maps. Here we use a computational model to show that, in principle, multiple adjacent and mirror symmetric topographic maps can arise from activitydependent synaptic changes alone, assuming that the distribution radius of already established cortical afferents sufficiently exceeds that of horizontal intracortical interactions during map development (Brown, Keynes, & Lumsden, 2001). This result raises the possibility that activity-dependent synaptic changes, which are generally assumed to be a central force in individual topographic map self-organization during development (Kohonen, 2001; Ritter et al., 1992; Sutton et al., 1994), may play a more important role in forming the orientations of adjacent cortical maps than is currently recognized. At the least, our results indicate the existence of an influence on the relative orientations of adjacent maps that may have affected the evolution of their genetically guided afferent connectivity. 2 Methods Our computer simulations adopt the widely used Kohonen model of map formation (Kohonen, 2001; Ritter et al., 1992) in a straightforward fashion.

1062

R. Schulz and J. Reggia

Our goal is not to provide a detailed, veridical model of neocortical circuitry and its dynamics, but simply to show that principles widely used in past selforganizing map models are sufficient (with simple modifications) to create mirror-image map formation. While the standard Kohonen model is a substantial simplification of biological reality, it can produce single maps that, at least qualitatively, are strikingly similar to neocortical maps (Bauer, 1995; Kohonen, 1989; Martinetz, Ritter, & Schulten, 1989; Obermayer, Blasdel, & Schulten, 1992; Obermayer, Ritter, & Schulten, 1990; Obermayer, Schulten, & Blasdel, 1992; Palakal, Murthy, Chittajallu, & Wong, 1995; Ritter & Schulten, 1986), suggesting that it captures some fundamental principles of cortical self-organization. We extend the single-winner Kohonen model in a natural way to allow for the occurrence of multiple “winner nodes” across the emerging map’s surface. Such an extension is consistent with the multifocal activity peaks often found in neocortical regions (Donoghue, Leibovic, & Sanes, 1992; Georgopoulos, Kettner, & Schwartz, 1988; Pei, Vidyasagar, Volgushev, & Creutzfeldt, 1994). Multiwinner self-organizing maps (SOMs) have been used in past— more biologically realistic and more complex and computationally expensive cortical models (von der Malsburg, 1973; Sutton et al., 1994; Sirosh & Miikkulainen, 1994). However, these previous multiwinner SOMs typically do not compute their output using Kohonen’s algorithm (or a generalization thereof, as we do here). Instead, a system of differential equations governs the SOM’s activation dynamics in these cases, and the system’s iterative simulation usually leads to multiple, spatially separated “Mexican hat” patterns of activity. So, past multiwinner SOMs differ from Kohonen SOMs algorithmically and also with respect to their application domain (Schulz & Reggia, 2004). Here, we address the issue of how generalizing Kohonen’s algorithm to multiple winners influences map formation, an issue that to our knowledge has not been investigated before. 2.1 Model Architecture and Dynamics. The basic architecture of the multiwinner SOM, illustrated in Figure 1B, is identical to that of a standard SOM. While many variations are possible, we elected for the cortical nodes to be arranged in a regular, rectangular lattice of R rows by C columns. The distance between two cortical nodes i and i  at positions (r, c) and (r  , c  ) in the lattice is measured using the box-distance metric, that is, dlattice (i, i  ) = max(|r − r  |, |c − c  |). Each cortical node i receives an afferent connection from each of the P nodes in the input layer. Every afferent connection carries a nonnegative, real-valued weight wi j on the connection from the jth input P to the ith cortical node, and w
i ∈ R+ represents the afferent weight vector to the ith cortical node. The level of activation of an input or cortical node ranges between 0 (inactive) and 1 (fully active). The activation levels of all P input nodes make up the input pattern, a vector x
∈ [0, 1] P of unit length.

Symmetric Maps

1063

Similarly, the activation levels of all cortical nodes form the output pattern, a vector y
∈ [0, 1] RC . In general, an input pattern x
encodes the stimulation of a point on a two-dimensional sensory surface. To avoid biases due to unequal length input vectors, the planar sensory surface inputs were normalized in length by their projection onto the surface of the unit sphere. Specifically, given a point p = unit square, its image on the unit sphere is
[ px , p y ] on the  p point q = q x = pax , q y = ay , q z = ab , where a = ( px2 + p 2y + b 2 )1/2 and b = √ 2 − ( px2 + p 2y )1/2 . The images on the unit sphere of the 441 points at the intersections in a regular grid of 21 rows by 21 columns covering the unit square were used for training. We used coordinate encoding of input patterns in most of our experiments because, especially during the search for suitable training parameters, computational efficiency was critical. Real sensory stimuli during biological map formation are, of course, more complex, but such simple local stimuli have often been used in past models of topographic map formation and have repeatedly been found to be adequate to support formation of single topographic maps. However, to demonstrate that coordinate encoding does not bias the model in favor of our hypothesis about orientations of adjacent maps, we repeated some of the experiments using bell-shaped sensory activation patterns (described later). Given an input pattern x
, the output pattern is determined by the same computationally efficient process employed by the standard SOM (Kohonen, 2001), except that it is generalized in a natural way to allow the simultaneous existence of multiple “winners.” First, the net input h to each cortical node i is computed as h i = w
iT x
where T indicates the transpose of the column vector w
i . We approximate the computationally expensive, iterative competitive activation dynamics (Mexican hat pattern) that is often implemented via the numerical solution of differential equations and iteratively transforms h
into y
(Cho & Reggia, 1994; Pearson, Finkel, & Edelman, 1987; Reggia, D’Autrechy, Sutton, & Weinrich, 1992; Sutton et al, 1994; von der Malsburg, 1973) by a one-step selection of winners. However, unlike in the standard SOM, multiple winners occur where each cortical node i that receives a net input greater than that to each of the N neighboring cortical nodes closest to i (ties resolved arbitrarily) is taken to be a winner. For all cortical nodes, including those near or on the edges of the SOM’s lattice, N is taken to be the number of other cortical nodes within a fixed radius of competition rcomp from the cortical node at location (R/2, C/2) in the center of the lattice. Note how having the same N for cortical nodes along the lattice’s edges is different from letting each cortical node compete with all other cortical nodes within a fixed radius from its position, which would introduce a bias favoring cortical nodes located near the lattice boundary. Since parameter rcomp is usually chosen to be small relative to the size of the lattice, typically multiple winner cortical nodes occur throughout the lattice in response to each input pattern. Each winner is made the central ”peak” of an ”island” of activation. The distribution of activation on a single island

1064

R. Schulz and J. Reggia

is such that the winner at the center of the island (cortical node i) is maximally active (yi = 1), and the activation level of each cortical node j that competed with i decreases exponentially with increasing distance between j and i. Specifically, if the set V of winners is V = {i | ∀ j = i : j competes with i ⇒ h j (t) < h i (t)},

(2.1)

then the activation of cortical node j is y j = γ d(i, j)

with i ∈ V, and

∀k ∈ V, d(k, j) ≥ d(i, j),

(2.2)

where γ ∈ [0, 1] determines the shape of each island of activation (lower γ means faster drop-off from the peak). If two or more islands of activation partially overlap, the activation level of a cortical node j in the region of overlap is determined by the island whose peak is closest to j. Unless stated otherwise, the parameter values used in the experiments reported here are C = 11 and rcomp = 7. Before training, each weight is independently initialized with a random value from the interval [0, 1], and each weight vector is then normalized to unit length. During training, the SOM learns by adjusting the weights on the incoming connections in response to each input of a vector from the training set, presented in a random order that is different for each epoch. The number of training epochs is 2000 unless noted otherwise. For each cortical node i, the learning rule is w
i = w
i + µyi x


(2.3)


i /||w
i ||2 . w
i = w

(2.4)

Equation 2.3 implements typical Hebbian learning where µ ∈ (0, 1] is the learning rate. Normalization in equation 2.4 restricts w
i to move across the surface of the unit hypersphere and may decrease a connection’s efficacy due to competition with the other connections terminating at cortical node i. We refer to this learning as competitive Hebbian learning, as it incorporates competition between cortical nodes for activation, and competition between weights due to vector normalization, as well as Hebbian learning. Typically during the training of a standard single-winner SOM, the values of certain parameters in the learning rule depend on how far training has progressed (Kohonen, 2001). For example, training is often divided into two phases: a rough ordering phase corresponding to large values for γ and µ in our model and a convergent phase corresponding to small values for γ and µ. Analogously in our model, parameters γ and µ monotonically decrease in a nonlinear fashion from some initial value to a smaller final value. For example, γ (t) = γfin + (γinit − γfin )/(1 + e (t−γinfl )/γσ ), where t is the

Symmetric Maps

1065

fraction of completed training epochs, γinit = 0.9 (γfin = 0.0) determines γ ’s initial (final) value, γinfl = 0.33 is the point of inflection, and γσ = 0.1 determines the rate of decline. A similar function is used for µ, where µinit = 0.5, µfin = 0.0, µinfl = 0.5, and µσ = 0.1. 2.2 Measures of Map Formation. Measures of the goodness of map formation, such as the topographic product (Bauer & Pawelzik, 1992) or the topographic function (Villmann, Der, Herrmann, & Martinetz, 1997), include in their calculations the extent to which adjacent locations in the input sensory space are represented by nonadjacent cortical nodes. Thus, these measures give inaccurate scores when multiple maps are involved because two cortical elements in different cortical maps representing nearby sensory surface locations can be widely separated from each other. For this reason, we devised and used a measure M ≥ 0 of map formation computed as the mean of the smallest 2% of all pairwise dot products between the weight vectors of adjacent cortical nodes. Roughly, M is inversely proportional to the average distance between receptive field centers of adjacent cortical elements in those parts of the cortex where these distances are greatest. Larger values of M indicate better map formation. Due to the normalization of the weight vectors to unit length, M = 1 is maximal. Solely for the purpose of visualizing map formation, each input location is associated with a label that is either the character @ or the blank character. When these labels are viewed on the sensory surface at their associated positions as in Figure 2A, they can be seen to form an inward clockwise spiral, with the size of the @ labels decreasing toward the center of the spiral (the size of an @ label has no meaning other than to indicate visually its location). The blank spaces in between the arms of the spiral aid the efficient visual judgment of whether a roughly topographic map of the sensory surface has formed, and if so, what its orientation is. The spiral labeling pattern is asymmetric so that the relative orientations of any two adjacent maps can be determined unambiguously (without, for example, mistakenly identifying them as a single large map). Each cortical element i is labeled by the input pattern x
j to which that element is most sensitive, that is, j = argmaxk (w
iT x
k ). A topology-preserving or well-formed map of the sensory surface thus shows up on the cortical surface as a projected image of the input surface’s spiral pattern. The image may be rotated or slightly distorted or may show a reversal of the spiral’s direction from clockwise to counterclockwise, since these transformations do not violate the topology of the sensory surface. 3 Results Given the radius of competition rcomp of 7, multiwinner SOMs of 15 by 15 or fewer cortical nodes (15 = 2rcomp + 1) are equivalent to the standard single-winner SOM since each cortical node competes with all other cortical

1066

R. Schulz and J. Reggia

nodes for activation and learning, and hence there is always only a single winner. Figure 2B shows a typical example of the initial disorganized state of an 11 by 11 cortical region’s representation of the sensory surface prior to training that is due to the random initialization of the afferent weights. Figure 2C shows, for the same region, the ordered map representation that was formed by training the network. As expected, when 11 by 11 and 11 by 15 cortical regions were trained, each self-organized into a single topologypreserving map of the sensory surface that covered the entire cortical region (see Figure 2C). Fourteen separate experiments were conducted, each corresponding to a specific cortical region size (R = 11, 15, 20, . . . , 75; C = 11), and for each size cortical region, 20 independent runs were done (independent in the sense that in each run, the network was initialized with different random weights and a different random order was used for the presentation of the input patterns during training). In all simulations prior to learning, the points of the sensory surface were found to be randomly represented over the cortical surface. As can be seen from the left half of Table 1, for a sufficiently small R ≤ 20, our model was essentially equivalent to a standard single-winner SOM, and consequently, only a single map of the sensory surface formed, covering the entire lattice of cortical nodes (as in Figure 2C). In contrast, with R ≥ 25, multiple well-formed maps invariably appeared. Examples of

Figure 2: (A) The input patterns encode representative points (the points of intersection in a grid) on a planar 2D sensory surface. For illustrative purposes, parts of this surface are labeled with an inward clockwise spiral of @ characters of progressively decreasing size. By showing how this spiral is projected onto the cortical surface, one can reliably judge by visual inspection whether a rough topology-preserving map of the sensory surface has formed, and, if so, its orientation. (B) The disorganized pretraining representation of the sensory surface by an 11 by 11 cortical surface (one square cell per cortical element), showing that the label positions (which indicate the locations of the cortical elements’ receptive field centers on the sensory surface) are not arranged in any order due to the initially random weights. A cell’s brightness indicates how similar that cortical node’s weight vector is to the weight vectors of its immediate neighbors. Disorganization here is thus also indicated by the very dark shading and an associated M = .35. (C) As expected with a small cortical region, the posttraining representation is like that seen with a standard single-winner SOM: a single topology-preserving map of the sensory surface that covers the entire region. This is indicated by @ signs showing how the spiral on the sensory surface has become topographically projected onto the cortex, the relatively light shading of all cortical elements (which also shows that the topographic organization indeed extends to the unlabeled regions in between the spiral arms), and a much larger M = .98.

Symmetric Maps

1067 A 1.0 .9 .8 .7 .6 .5 .4 .3 .2 .1 0.0

@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@ @@@ @@@ @@@ @@@ @@@ @@@ @ @ @ @ @ @ @ @ @ @@@ @@@ @ @ @ @ @ @ @ @ @ @@@ @@@ @ @ @ @ @ @ @ @ @ @@@ @@@

@@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ 0.0 .1

@@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@

@ @ @ @ @ @

@ @ @ @ @ @

@ @ @ @ @ @

@ @ @ @ @ @

@ @ @ @ @ @

@ @ @ @ @ @ @@ @ @@@@@@@ @ @ @ @@ @ @@@@@@@ @ @ @ @@ @ @@@@@@@

.2

.3

@

@

.4

@ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @

.5

.6

.7

.8

.9

@

@

1.0

B 11

@

10

@

@

9 8

@

@

@ @

@

@

@

@ @

@

@ @

@ @

@

7

@

@

@

@ @

@

@

@

@

6

@ @ @

@

@

@

@

@

@

@

@

@

@

@

@

@

@

5 4

@

@

@ @

@

@

3

@

@

@

2

@

@

@

1

@

@

2

3

1

@

@

@

4

@

@

5

6

7

8

@

@

@

@

9

10 11

C 11 10

@ @ @ @ @ @ @ @ @ @ @ @ @

9

@

8

@

@

@

@

@

@

@

@

@

@

@

@

@

7

@

@

@

@

@

@

@

6

@

@

@

@

5

@

@

@

4

@

@

@

3

@

2

@

1

@ 1

@

@

@

@

@

@ @

@ @

@ @

2

@

@

@ @

@

@

@

@

@

@

@

@

3

4

5

6

7

8

9

10 11

1068

R. Schulz and J. Reggia

Table 1: Number and Pairwise Symmetries of Learned Maps (20 Runs Each). Pairwise Symmetriesa

Number of Maps R

Mean

Minimum

Maximum

m

g

r

11 15 20 25 30 35 40 45 50 55 60 65 70 75

1.00 1.00 1.00 2.00 2.00 2.12 2.95 3.00 3.40 3.83 4.00 4.50 5.07 5.18

1 1 1 2 2 2 2 3 3 3 4 2+ 2+ 3+

1 1 1 2 2 3 3 3 4 4 4 6 7 6

1.00 .90 .63 .92 .78 .73 .81 .93 .77 .82 .77

.00 .10 .37 .03 .10 .19 .15 .02 .20 .05 .12

.00 .00 .00 .05 .13 .08 .04 .05 .03 .14 .11

m = mirror reflection. g = glide reflection. r = 180◦ rotation. + = unorganized areas also present.

a

multiple map formation are illustrated in Figure 3. In general, the number of maps formed increased proportional to R: approximately one additional map was formed for each additional increment of R by 15 rows. This suggests that in general, the number of additional rows required to accommodate an additional map roughly equals the “diameter” of competition, that is, the number of rows 2rcomp + 1 (for R ≥ 2rcomp + 1) of other cortical nodes with which each cortical node has to compete for activation and learning (e.g., 15 for rcomp = 7). All of the maps in any one instance where multiple maps occurred were generally of the same size. Two adjacent maps were usually immediately adjacent; that is, there were no lattice parts in between them that were not part of the two adjacent maps, regardless of R. In all simulations producing multiple maps, only three types of symmetries were observed between any two adjacent maps, regardless of the total number of maps present. In the overwhelming majority (82%) of cases, the two adjacent maps were mirror images of each other (see Figure 3A). The second type of symmetry observed, found in 11% of the cases, was again essentially a mirror reflection, but now the axis of reflection was tilted so that the boundary between the two maps was no longer of minimal length (see Figure 3B). In addition, the maps were translated in opposite directions along their common tilted boundary so that the resultant transformation is better characterized as a glide reflection. Thus, in 93% of the cases, adjacent maps exhibited mirror symmetry or distorted mirror symmetry. In the remaining 7% of adjacent map pairs, each individual map was characterized as a rotation relative to the other of 180 degrees around a symmetry

Symmetric Maps

A

35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

1069

@@@@@@@@@@@ @@@@@@@@@@@ @@@ @@ @@@@ @@ @@@@@ @@ @ @ @ @@@ @ @@ @ @ @ @@ @ @ @@ @ @ @ @@ @@ @@ @@ @@ @@

@@@ @

@@

@@@@ @@@@ @@@

@ @ @ @

@@ @@ @@ @@@@@@@@@@@ @@@@@@@@@@@ @@@@@@@@@@@

@@@

@ @@ @@ @@ @@ @@ @@ @@

@@ @@@@ @@ @ @ @ @ @@ @ @ @ @ @ @ @ @@

@@ @@ @@ @ @@@@@@@

@ @ @ @@@ @ @@

@@ @@@@@@ @@ @@@ @@ @@@ @@@@@@@@ @ @@@@@@@@@@@ 1 2 3 4 5 6 7 8 9 10 11

B35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

@@@@@@@@@@@ @@@@@@@@@@@ @@ @@ @@ @ @@ @ @@

@ @ @ @@@ @ @@@@

@ @ @@

@@ @@ @ @ @@ @ @ @@@ @ @@ @

@@ @@ @@@ @@

@ @ @ @ @@ @@ @@

@@ @@ @@ @@ @@@ @@@@@ @ @ @ @@@@@@ @@@ @@@ @ @@ @ @ @ @@ @@ @ @ @@ @ @ @@ @ @ @ @@

@@ @@

@@ @@ @@

@@ @@ @@

@@

@@ @@ @@

@@@ @@@ @@@

@@ @ @ @ @ @@ @ @ @@ @ @ @ @@

@ @ @@ @@ @@ @

@@ @@ @@@@@@@@@@@ @@@@@@@@@@@ 1 2 3 4 5 6 7 8 9 10 11

C

40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

@@@@@@@ @ @ @@@@ @@ @ @@ @ @@@@@@ @@ @@@ @@ @@@ @@@ @@ @@@@ @@@ @@ @@@@ @ @ @@ @@ @@ @@ @@ @@ @@ @@

@@@ @@ @@ @ @@ @@@ @ @ @@@ @ @ @@ @ @ @@ @ @

@

@ @ @

@ @@ @@ @@ @@ @@ @@@ @@ @@@@ @@ @@@@ @@@@@ @@@@ @@@@@@ @@@@ @@@@@ @@@ @@@ @@ @@ @@ @@ @@ @@ @@@ @@ @ @ @@@ @ @ @ @ @@@ @ @ @ @ @@ @ @ @@ @ @ @ @ @

@@ @@ @@ @@ @@ @@@ @@ @@ @@@@ @ @@ @@ @@@@@ @@ @@@@ @ @

@@@ @@@@ @@@@@@@ @@@@@@

1 2 3 4 5 6 7 8 9 1011

Figure 3: Representative instances where two maps formed in cortical regions during learning (top). A corresponding schematic representation is given (bottom) for each of the three observed types of symmetry between adjacent maps. The spatial organization of the maps is indicated by how a single spiral painted on the sensory surface (see Figure 2A) is replicated and oriented in the map. (A) Mirror symmetry. (B) Glide reflection symmetry (distorted mirror symmetry). (C) Rotational symmetry. In the schematic representations, the thin lines in A and B and the point in C indicate the symmetry axis and the center of rotation, respectively.

point at the center in between the two maps (see Figure 3C). Rotational symmetry was easily identifiable since it was the only type of symmetry that caused the projected spiral patterns in two adjacent maps to follow the same (clockwise or anticlockwise) direction (see Figure 3). The right-most

1070

R. Schulz and J. Reggia

three columns of Table 1 show the fractions of mirror (m), glide (g), and rotation (r ) symmetries between adjacent maps for different lattice sizes R, averaged over 20 independent runs, respectively. Map visualizations like those in Figure 3 also revealed that the three symmetry types exhibited distinct patterns of similarity among the cortical elements along an intermap boundary. Cortical nodes along the boundaries between mirror-image maps typically were similar to their neighbors in the lattice, that is, their afferent weight vectors and, thus, their receptive field centers were close to one another. In Figure 3A, this is manifested by the lightly shaded cells along the intermap boundary. In contrast, for glide reflections and rotationally symmetric adjacent maps, dissimilar cortical elements (dark shaded cells) were typically prominent along the intermap boundary. Such dissimilar cortical elements were present all along the boundary between glide reflection symmetric maps (dark shaded boundary elements of rows 13–25 in Figure 3B), while their presence was limited to the outer reaches of the intermap boundary in the rotationally symmetric case (very dark shaded elements on the extreme left and right of rows 18–20 in Figure 3C). There was a weak tendency for the largest fractions of mirror symmetric maps (>80%) to occur when R was a multiple of 15 or slightly smaller (R = 25, 30, 40, 55, 60, 70 in Table 1). Under these conditions, the height of a single map was roughly 15. However, we found several cortical regions on which exclusively mirror symmetric map pairs formed and the number of maps exceeded the expected value because they were smaller. The single cortical region in Figure 4 provides an example of this where six somewhat compressed maps formed on the 75 by 11 cortical region. In a small minority of cases, the network did not completely self-organize, and parts of the cortical region remained disorganized after learning. For example, the entry 2+ for R = 65 in Table 1 indicates that in 1 of the 20 simulations with networks of this size, only two representations of the sensory surface were found, with the rest of the cortical region being disorganized (all other 19 simulations in this case exhibited at least four maps and no disorganized regions). 3.1 Measuring Map Formation and Types of Symmetries. For the 220 simulations with cortical regions sufficiently large for multiple maps to appear (R ≥ 25), the mean initial value of M prior to any learning was 0.31 (SD 0.02, minimum 0.23, maximum 0.38). Following learning, this increased to 0.97 (SD 0.02, minimum 0.87, maximum 0.98). Each cortical region that was in principle large enough for multiple well-formed maps to appear (all 220 runs in Table 1 for which R ≥ 25) was assigned to one of three categories after training. First, if any disorganized region was present, the cortical region was placed in the “?” category (20 runs, or 9%), even if well-formed maps were also present. The remaining fully organized cortical regions were divided into those in category m, where exclusively mirror

Symmetric Maps

75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38

1071

@@@@@@@ @@ @ @@@@@@@@ @ @ @@ @ @@ @ @ @ @@ @ @ @@ @ @ @ @ @ @ @@ @@ @ @ @ @@ @@ @ @ @ @@ @

@

@@ @@ @ @@ @@ @@@@@@@@@@@ @@@@@@@@@@@ @@@@@@@@@@@ @@@@@ @ @@@@@

@@ @@@ @@ @@@ @ @@ @ @ @ @ @@@@ @@ @ @ @@ @ @ @ @ @ @ @ @ @@ @@

@

@@

@@ @@ @@@@ @@@@ @@@@@@@@ @@@@@@@@ @@

@ @ @ @ @ @ @@ @@ @ @@ @ @@ @ @@ @ @@ @ @@ @ @ @ @@ @@@@@ @

@ @

@

@@ @@@@@ @ @@ @ @@ @@@@@@@@@@@ 1 2 3 4 5 6 7 8 9 1011

37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

@@@@@@@@@@@ @ @@ @ @@ @@ @

@ @ @@ @ @@ @@ @@ @

@@ @@ @@

@@

@ @

@@ @@ @@ @@ @@ @@ @@ @@

@@ @@ @@@@@@@@@ @@@@@@@@ @ @@ @@

@ @ @@ @ @@ @ @ @@ @@ @ @ @ @@ @ @ @@ @@ @ @@ @ @@@@ @ @@

@ @@ @ @ @ @ @@ @@ @@ @@ @@@@@@@@@@@ @@@@@@@@@@@ @@@@@@ @@ @@ @@ @ @@ @@ @ @@ @@

@ @ @ @ @@ @ @

@@ @ @@ @ @@ @ @ @@ @

@@ @@ @@ @@ @@ @@ @

@@

@ @@ @@ @@@@@@ @ @ @@ @@@@@@@@ 1 2 3 4 5 6 7 8 9 1011

Figure 4: A single cortical region on which six maps of the sensory surface appeared; every two adjacent maps are mirror images of each other. For illustrative purposes, the region has been split in the middle, with the top half shown on the left (schematic representation on the right).

symmetric map pairs had formed (138, 62%), and those in category g|r (62, 29%), where at least one glide reflection or rotationally symmetric map pair occurred. Figure 5 shows the distribution of the M values for these three categories of cortical regions. The mean M value was 0.980 (SD 0.002) for category m simulations and a significantly different 0.965 (SD 0.018) for category g|r simulations ( p < 10−3 on t-test). On average, the M values were significantly greater for category m (g|r ) than for category g|r (“?”) (Jonckheere trend test; p .0001), and the spread of the values was smaller for category m (g|r )

1072

R. Schulz and J. Reggia 80

Number of Pairs of Adjacent Maps

70

m g|r ?

60

50

40

30

20

10

0 0.87 0.88 0.89 0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99

M Figure 5: A histogram of the M values for each of the three symmetry categories. The values have been grouped into 16 consecutive intervals of width 0.01. To be comparable across the different-sized categories, each histogram shows, for each interval, the relative within-category frequency with which M fell within the limits of the interval. The histograms suggest that the means µ and standard deviations σ of the actual distributions of M values are ordered so that µm > µg|r > µ? and σm < σg|r < σ? .

than for category g|r (“?”). Since M primarily measures the organization along map boundaries when multiple maps are present, these results indicate that the same synaptic modifications responsible for individual map formation also tend to maximally preserve similarity of adjacent cortical element receptive fields along map boundaries by producing adjacent maps that are mirror symmetric. In contrast, other symmetry relationships (glide, rotational) are local maxima of M in which the map formation process becomes trapped during learning. Since category g|r lattices also exhibited mirror symmetric map pairs, the differences observed between categories m and g|r most likely would have been even more pronounced if each pair of adjacent individual maps had been manually categorized individually and if M had been measured separately for each pair (this was impractical to do).

Symmetric Maps

1073

3.2 Nonuniform Density of Sensory Stimuli. In the simulations described above where each sensory surface location was stimulated exactly once per epoch, when mirror-image maps formed, there was nothing to bias which of their four edges became adjacent. To see if the selection of which edge pair becomes adjacent during formation of mirror-image maps could be biased, we altered the previously uniform probability distribution of stimuli over the sensory surface during learning. This was done by partitioning the regular 21 by 21 grid of the sensory surface points into three consecutive 21 by 7 subgrids (I, II, and III). Each point in subgrid I was stimulated only once during a single epoch of training, while each point in subgrid II (III) was stimulated twice (three times) during the same period. During training, we used a fixed lattice size of 25 by 11 cortical nodes and coordinate-encoding input patterns, a combination that earlier had produced a pair of mirror-image maps in 19 of 20 runs when uniformly probable input stimuli were used. With uniform sensory stimuli previously, the absolute frequencies of the four relative orientations among these 19 mirror-image map pairs (6, 3, 6 and 4) were consistent with equiprobable adjacent edge selection (χ 2 = 1.42, p > 0.05). In contrast, of the same 20 independent runs of training that were performed using the nonuniformly distributed sensory stimuli, 19 again produced mirror-image map pairs. However, now 17 of the adjacent map edges corresponded to the single sensory surface edge where stimuli were most probable (absolute frequencies: 2, 0, 0, and 17). This was inconsistent with equiprobable edge selection (χ 2 = 42.26, p < 0.05). At this preferred orientation, the most frequently stimulated region of the sensory surface usually becomes represented in both maps along the intermap boundary, with the least frequently stimulated region most removed from the intermap boundary. Under these conditions, we also examined whether magnification effects occurred. In biological systems, sensory surface regions with higher innervation density or that are more frequently stimulated are typically magnified in cortical maps: their cortical representation occupies a disproportionately large area of cortical surface (Dykes & Ruest, 1984). Past computational models of single maps have reproduced this magnification effect, given inputs having a nonuniform distribution of sensory stimuli (Grajski & Merzenich, 1990). With the nonuniform sensory stimuli described above, we found that the images of the two more frequently stimulated sensory surface regions came to occupy a relatively larger area of the cortical region (at the expense of the third least stimulated sensory region) compared to when the three sensory regions were stimulated equally frequently during training. With the uniformly distributed sensory stimuli (the baseline), the map representations of the sensory regions corresponding to subgrids I, II, and III occupied, on average, 79.0, 104.0, and 91.9 cortical elements (standard deviations 13.0, 3.6, and 13.6, respectively). The somewhat smaller size of regions I and III here is due to edge effects. In comparison, with the nonuniformly distributed sensory stimuli, the same averages for regions

1074

R. Schulz and J. Reggia

I, II, and III were 57.6, 112.0, and 105.4 (standard deviations 4.9, 3.3, and 7.7, respectively). This is a significantly different result (U-test; p < 10−6 , p < 10−5 and p < .005, respectively, for each region). 3.3 Sensitivity to Model Changes. We examined a number of variations in the parameters and other aspects of our model to determine its robustness, that is, whether qualitatively similar results would be obtained after these variations. For a radius of competition rcomp = 7, we observed the largest fraction of mirror symmetric map pairs for a lattice width of C = 11 (0.82m, 0.11g, 0.07r ). Experiments with C < 11 resulted in a relatively larger number of glide reflections (e.g., 0.69m, 0.27g, 0.04r for C = 9). For C > 11, the relative number of rotational symmetries increased (e.g., 0.78m, 0.04g, 0.18r for C = 13, and 0.68m, 0.02g, 0.30r for C = 15). This suggests that for a given radius of competition rcomp , a particular width C of the lattice may be optimal for the formation of mirror symmetric map pairs. The initial values of γ , γinit , and γinfl were also important. For γinit = .8 (rather than .9, as in the experiments above), the fraction of mirror symmetric map pairs dropped to typically 60%. Rotation and glide reflection symmetry became more frequent, with each occurring in roughly 20% of the cases. Delaying γ ’s descent by increasing γinfl to 0.5 increased the number of cases in which self-organization failed partially or completely so that no wellformed maps were discernible in (parts of) the SOM’s lattice. The effects of parameter changes pertaining to γ especially depend on how the learning rate µ changes over time during training. We made the above observations on the effects of changes to γinit and γinfl while µinit = 0.5, µinfl = 0.5 and µsigma = 0.1 were held fixed. Two variations of how islands of activity are determined (see equation 2.2) were also tested. The first variant determines the activity of a cortical element by taking into account all winners (as opposed to just the closest one) and adding their activity contributions. Equation 2.2 was replaced by yj =



γ d(i, j) ,

(3.1)

i∈V

which, in general, increases the activity of cortical nodes that are located in an area of the cortical region where several islands of activation overlap. Given equation 3.1, it is possible that a cortical element in an area of overlap becomes more active than a winner. The second variant prevents this from happening by capping each cortical element’s activation if it exceeds 1. Both variants were somewhat less conducive to the formation of mirror symmetric map pairs than the original equation 2.2. In all of the simulations described, we used a computationally efficient coordinate encoding of sensory stimuli, as is often done with Kohonen maps. To verify that this did not bias our results, we repeated the first

Symmetric Maps

1075

Table 2: Number and Pairwise Symmetries of Learned Maps (10 Runs Each, Full Encoding). Pairwise Symmetriesa

Number of Maps S

Mean

Minimum

Maximum

m

g

r

11 15 20 25 30 35 40 45 50 55 60 65 70 75

1.00 1.00 1.00 2.00 2.00 2.22 3.00 3.00 3.40 4.00 4.00 4.78 5.20 5.50

1 1 1 2 2 2 3 3 3 2+ 3+ 3+ 5 5

1 1 1 2 2 3 3 3 4 4 4 5 6 6

1.00 1.00 1.00 1.00 .95 .92 .94 .79 .94 .93 .82

.00 .00 .00 .00 .00 .08 .03 .21 .00 .07 .09

.00 .00 .00 .00 .05 .00 .03 .00 .06 .00 .09

m = mirror reflection. g = glide reflection. r = 180◦ rotation. + = unorganized areas also present.

a

10 runs of each experiment, training the networks with a full encoding of the sensory input patterns. With the latter, each sensory activation pattern is a vector with as many components as the number of sample sensory surface points (441 in our model), making the model computationally much more expensive. With this full encoding, stimulation of a point on the sensory surface was represented as a bell-shaped activation pattern centered on the stimulus location. Specifically, if the stimulation of point p = ( px , p y , pz ) on the sensory surface is encoded by x
( p) then ( p) xq , the component of x
( p) corresponding to the activation level at point p q +p q +p q q = (q x , q y , q z ), equals (  ( pxx qx + pyy qy + pzz qz )2 )1/2 . This implies that the sensory x y z q activation patterns evoked by two separate and independent point stimuli are more similar the smaller the distance on the sensory surface between the two points (this is true also if coordinate encoding is used). The overall results, given in Table 2, were 91% mirror symmetric, 6% glide reflection symmetric, and 3% rotationally symmetric map pairs, indicating a significant increase in the fraction of (distorted or undistorted) mirror symmetric map pairs from 93% to 97% (one-sided χ 2 test; χ 2 = 6.71, p < .005). The average number of maps appearing on the cortical lattice was not significantly different at the 0.05 significance level, regardless of the specific lattice size R (U statistical test). These results indicate that using coordinateencoding input patterns did not substantially influence the average number of well-formed maps per cortical region and, more important, did not bias our results in favor of mirror symmetric adjacent maps.

1076

R. Schulz and J. Reggia

4 Discussion In this work we have explored the extent to which activity-dependent synaptic changes may play a role in the self-organization and relative orientation of adjacent topographic maps. Our computational model, when trained with input patterns that represent the stimulation of points on a sensory surface, formed multiple topologically correct maps of the sensory surface whenever the distribution radius of cortical afferents sufficiently exceeded that of horizontal intracortical interactions, conditions that may be present during development when thalamocortical afferent projections are more widespread than in adults (Brown et al., 2001; Mountcastle, 1998). Further, the adjacent maps produced in this fashion were largely mirror symmetric with respect to their common boundary, a finding that is consistent with observations of topographic maps and their relative orientations in biological cortex. These results persisted in the face of parameter variations and even a different representation of sensory stimuli as long as basic map self-organization was not disrupted. In all of our experiments, the topology of the sensory surface (the 2D unit square, projected onto the unit sphere to normalize input vectors to unit length) matched the topology of the modeled cortical surface (2D lattice of cortical elements). If the topology of the sensory surface had been as complex as, for example, the surface of the human body, and thus, more complex than the topology of the SOM’s 2D lattice, then the latter would have been unable to form perfectly topology-preserving maps of the entire sensory surface. A standard Kohonen SOM tends to minimize the degree to which the map of the sensory surface violates the topology of the sensory surface (Kohonen, 2001). However, it is an open question how a mismatch in topology would influence multiple map formation and their relative orientations in the multiwinner case. Mirror-image and distorted mirror-image maps in our model arose due to the natural tendency of competitive Hebbian learning to make adjacent cortical elements represent neighboring regions of the input surface while ensuring that the entire sensory surface is represented. A single topographic map of the sensory surface achieves both of these ends, but its formation requires a single global competition. In contrast, which biological cortical element responds most to a particular input is presumably known only locally. In our model, the multiple “winner elements” that arise due to simulating this locality result in the formation of multiple individually topology-preserving maps plus the optimization of the transitions between them. As illustrated in Figure 6A, which is pictured in the input space, this optimal transition corresponds to a fold in the cortical region’s lattice when projected onto the input space, the fold being perpendicular to the map edges and leading to mirror symmetric adjacent maps. The other two types of symmetry that were observed constitute suboptimal transitions from one

Symmetric Maps

A

1077

B

C

Figure 6: Various ways in which the model cortical region became embedded in the input space, a two-dimensional sensory surface that is shown as a light gray planar area. Cortical elements (small black rectangles) are connected by a solid line iff they are adjacent in the cortical region. Each case involves two roughly topographic maps of the sensory surface that have been spatially separated along the vertical axis to illustrate the projection of the cortical region in the input space. For illustrative purposes, the cortical regions shown here are much smaller than those used in the actual experiments. (A) Mirror symmetric maps (compare to Figure 3A). The cortical region becomes folded in the process of self-organization with the fold perpendicular to the longer sides of the cortical region, keeping connected cortical elements along the intermap boundary close in the input space. (B) Glide reflection symmetric maps (compare to Figure 3B). The cortical region contains a diagonal fold, and the cortical elements along the fold line (separated from the two maps along the z-axis) have been forced to move counterclockwise (clockwise) in the input space around the top (bottom) map. Many neighboring cortical elements along the fold line become widely separated in the input space (such nodes are gray and connected by thick gray lines). (C) Rotationally symmetric maps (compare to Figure 3C). The cortical region has been twisted in addition to being folded in the same manner as in A. In this case, adjacent cortical elements along the fold line and close to the edges of the cortical region become very widely separated in the input space.

map to the next. A glide reflection symmetry is the visible expression of a diagonal fold in the lattice combined with a shearing motion along the fold (see Figure 6B), which causes the entire diagonal intermap boundary to be mildly nonoptimal. Rotation symmetry indicates a combination of a perpendicular fold and a twist of the lattice (see Figure 6C) so that only the central region of the intermap boundary retains its optimality; near the edges of the cortical region, the boundary is highly nonoptimal. The key

1078

R. Schulz and J. Reggia

point illustrated by Figure 6 is that in the latter two cases, cortical elements near the map boundary become removed from one another in the input space, rendering these two transitions suboptimal and hence significantly less likely to occur. We also observed in our simulations that when the sensory surface was subdivided into regions, some being stimulated more often during training, two adjacent maps, in addition to being mirror symmetric, were almost always oriented in such a way that their representations of the most often stimulated region were located next to each other at the intermap boundary. The other regions were represented farther away from the boundary, following the gradient in the frequency of stimulation. The more often stimulated regions were represented by a relatively larger area of modeled cortical surface in each of the individual maps. Our finding that two adjacent maps produced by activity-dependent synaptic changes will usually be mirror images of one another is complementary to and consistent with the prevalent notion that activityindependent genetic factors initially determine cortical arealization and affect targeting of thalamocortical afferents. It raises the question of how genetic and activity-dependent synaptic changes might interact during development and even during evolution, as it seems improbable that evolutionary processes would hard-wire adjacent cortical maps to be mirror images so often unless there was some advantage to this arrangement (such as consistency with local synaptic plasticity). Our model also makes two specific, testable predictions that may or may not relate to biological cortical maps. First, when adjacent mirror image topographic maps occur in neocortex, their common edge should represent the region of sensory surface that develops and innervates first (i.e., that has the most frequent stimuli initially during map development). This is consistent with, for example, the otherwise surprising location of fingers and toes in biological neocortex far from the symmetry axis in mirror image hand and foot representations in S1 (see Figure 1A), as these distal digits appear late during development (Gilbert, 1994; Lonai, 1996). Second, adjacent maps may occasionally exhibit a very different rotational symmetry. If such previously unreported rotationally symmetric maps are ever observed experimentally in a small percentage of currently known cortical map regions, they would provide very strong support for our model. Such atypically oriented adjacent maps, in the context of normal connectivity between cortical regions, would be expected to cause abnormal cortical information processing, and we speculate that they might account for some of the cognitive deficits and functional imaging changes observed in neurodevelopmental disorders such as dyslexia or autism (Frank & Pavlakis, 2001; Papanicolaou et al., 2003; Temple et al., 2003). We believe that the rarity of such atypically oriented adjacent maps and the very limited experimental data on human maps may explain why they have not been reported experimentally.

Symmetric Maps

1079

In our model, each “competition” covers roughly as much cortical area as a single topographic map. This raises the question of whether lateral interactions in biological primary sensory cortex can span comparable distances. While most direct lateral interactions in adults are more localized than this, long-range lateral interactions have been reported in cortex. For example, in primary visual cortex (V1), specifically in the context of understanding the mechanisms that underlie the contextual modulation of a neuron’s receptive field, a range of interaction exceeding 13 degrees of the visual field has been described (Angelucci et al., 2002; Bringuier, Chavane, Glaeser, & Fr´egnac, 1999; Grinvald, Lieke, Frostig, & Hildesheim, 1994; Kitano, Niiyama, Kasamatsu, Sutter, & Norcia, 1994; Levitt & Lund, 2002). In adult macaque monkeys, this roughly corresponds to a distance in V1 of 20 mm. While this figure falls short of a lateral interaction range spanning all of V1 in the adult (macaque V1 is roughly an ellipsoid of 25 mm by 50 mm when flattened: Felleman and van Essen, 1991), the relative range may be longer early during development when map formation is occurring (Brown et al., 2001). In addition, the measurements often constitute lower bounds on the maximum range, and they take into account only specific types of lateral interaction (facilitation and surround inhibition). Thus, the range of competition in our model is not necessarily inconsistent with biological reality. Finally, we note that our one-shot model of map formation is computationally efficient, like Kohonen’s widely used model, yet at the same time exhibits more properties that are biologically plausible (e.g., multiwinner representations, multiple maps, mirror-image adjacent maps). This combination of properties may be of interest to future builders of very large-scale neural models where computational cost is a critical issue and therefore the use of iterative multiwinner SOMs becomes problematic. Some of our model’s computational properties may also prove valuable in traditional application domains of the single-winner Kohonen SOM. The property of redundant map formation could be exploited to increase the fault tolerance and, thus, the robustness of an existing single-winner SOM-based system. Data visualization, one of the main uses for single-winner SOMs, may benefit from the property that the relative orientation of adjacent mirror maps depends on the distribution of inputs, adding a visual clue that can help identify, for example, gradients in the input distribution. Exploration of these ideas is an important possible direction for future research.

Acknowledgments This work was supported by NIH award NS35466. We thank C. Lynne D’Autrechy, Mary Howard, and Scott Weems for constructive comments.

1080

R. Schulz and J. Reggia

References Allman, J., & Kaas, J. (1971). A representation of the visual field in the caudal third of the middle temporal gyrus of the owl monkey. Brain Research, 31, 85–105. Angelucci, A., Levitt, J., Walton, E., Hupe, J., Bullier, J., & Lund, J. (2002). Circuits for local and global signal integration in primary visual cortex. J. Neuroscience, 22(19), 8633–8646. Bauer, H.-U. (1995). Development of oriented ocular dominance bands as a consequence of areal geometry. Neural Computation, 7(1), 36–50. Bauer, H.-U., & Pawelzik, K. (1992). Quantifying the neighborhood preservation of self-organizing feature maps. IEEE Transactions on Neural Networks, 3(4), 570–579. Beck, P., Pospichal, M., & Kaas, J. (1996). Topography, architecture, and connections of somatosensory cortex in opossums: Evidence for five somatosensory areas. J. Comparative Neurology, 1(366), 109–133. Bringuier, V., Chavane, F., Glaeser, L., & Fregnac, ´ Y. (1999). Horizontal propagation of visual activity in the synaptic integration field of area 17 neurons. Science, 283, 695–699. Brown, M., Keynes, R., & Lumsden, A. (2001). The developing brain. New York: Oxford University Press. Cho, S., & Reggia, J. (1994). Map formation in proprioceptive cortex. Int. J. Neural Systems, 5(2), 87–101. Cohen-Cory, S. (2002). The developing synapse: Construction and modulation of synaptic structures and circuits. Science, 298, 770–776. Cowey, A. (1981). Why are there so many visual areas? In F. Schmitt (Ed.), The organization of the cerebral cortex (pp. 395–413). Cambridge, MA: MIT Press. Donoghue, J., Leibovic, S., & Sanes, J. (1992). Organization of the forelimb area in squirrel monkey motor cortex. Experimental Brain Research, 89, 1–19. Drager, U. (1975). Receptive fields of single cells and topography in mouse visual cortex. J. Comparative Neurology, 160, 269–290. Dykes, R., & Ruest, A. (1984). What makes a map in somatosensory cortex? In E. Jones & A. Peters (Eds.), Cerebral cortex (Vol. 5, pp. 1–29). New York: Plenum Press. Engelien, A., Yang, Y., Engelien, W., Zonana, J., Stern, E., & Silbersweig, D. (2002). Physiological mapping of human auditory cortices with a silent event-related fMRI technique. Neuroimage, 4(16), 944–953. Felleman, D., & van Essen, D. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1(1), 1–47. Formisano, E., Kim, D., & Salle, F. (2003). Mirror-symmetric tonotopic maps in human primary auditory cortex. Neuron, 40, 859–869. Frank, Y., & Pavlakis, S. (2001). Brain imaging in neurobehavioral disorders. Pediatr. Neurol., 25, 278–287. Gentilucci, M., Fogassi, L., Luppino, G., Matelli, M., Camarda, R., & Rizzolatti, G. (1989). Somatotopic representation in inferior area 6 of the macaque monkey. Brain, Behavior and Evolution, 2–3(33), 118–121. Georgopoulos, A., Kettner, R., & Schwartz, A. (1988). Primate motor cortex and free arm movements to visual targets in three-dimensional space. II. Coding of the directions of movement by a neural population. J. Neuroscience, 8, 2928–2937.

Symmetric Maps

1081

Gilbert, S. (1994). Developmental biology. Sunderland, MA: Sinauer. Grajski, K., & Merzenich, M. (1990). Neural network simulation of somatosensory representational plasticity. In D. Touretzky (Ed.), Advances in neural information processing systems, 2 (pp. 52–59). San Mateo, CA: Morgan Kaufmann. Grinvald, A., Lieke, E., Frostig, R., & Hildesheim, R. (1994). Cortical point-spread function and long-range lateral interactions revealed by real-time optical imaging of macaque monkey primary visual cortex. J. Neuroscience, 14(5), 2545–2568. Grove, E., & Tomomi, F. (2003). Generating the cerebral cortical area map. Annual Review Neuroscience, 26, 355–380. Imig, T., Reale, R., & Brugge, J. (1986). Topography of cortico-cortical connections related to tonotopic and binaural maps. In F. Lepore (Ed.), Two hemispheres—One brain (pp. 103–115). New York: Alan Liss. Jones, E. (1990). Modulatory events in the development and evolution of primate neocortex. In A. Peters (Ed.), Cerebral cortex (pp. 311–351). New York: Plenum. Kaas, J. (1988). Why does the brain have so many cortical areas? J. Cognitive Neuroscience, 1, 121–134. Kitano, M., Niiyama, K., Kasamatsu, T., Sutter, E., & Norcia, A. (1994). Retinotopic and nonretinotopic field potentials in cat visual cortex. Visual Neuroscience, 11, 953–977. Kohonen, T. (1989). Self-organization and associative memory (3rd ed.). Berlin: Springer. Kohonen, T. (2001). Self-organizing maps (3rd ed.). Berlin: Springer. Krubitzer, L. (1995). The organization of neocortex in mammals. Trends in Neuroscience, 18, 408–417. Krubitzer, L., & Calford, M. (1992). Five topographically organized fields in the somatosensory cortex of the flying fox: Microelectrode maps, myeloarchitecture, and cortical modules. J. Comparative Neurology, 1(317), 1–30. Krubitzer, L., Clarey, J., Tweedale, R., Elston, G., & Calford, M. (1995). A redefinition of somatosensory areas in the lateral sulcus of macaque monkeys. J. Neuroscience, 5(15), 3821–3839. Levitt, J., & Lund, J. (2002). The spatial extent over which neurons in macaque striate cortex pool visual signals. Visual Neuroscience, 19(4), 439–452. Levitt, P. (2000). Molecular determinants of regionalization of the forebrain and cerebral cortex. In M. Gazzaniga (Ed.), The new cognitive neurosciences (pp. 23–43). Cambridge, MA: MIT Press. Lonai, P. (1996). Mammalian development. London: Harwood. Martinetz, T., Ritter, H., & Schulten, K. (1989). Kohonen’s self-organizing map for modeling the formation of the auditory cortex of a bat. In R. Pfeifer, Z. Schreter, F. Fogelman-Souli´e, & L. Steels (Eds.), Connectionism in perspective (pp. 403–412). Amsterdam: North-Holland. Merzenich, M., Kaas, J., Sur, M., & Lin, C. (1978). Double representation of the body surface within cytoarchitectonic areas 3b and 1 in “S1” in the owl monkey (aotus trivirgatus). J. Comparative Neurology, 181(1), 41–73. Mountcastle, V. (1998). The cerebral cortex. Cambridge, MA: Harvard University Press. Nelson, R., Sur, M., Felleman, D., & Kaas, J. (1980). Representations of the body surface in postcentral parietal cortex of macaca fascicularis. J. Comparative Neurology, 4(192), 611–643.

1082

R. Schulz and J. Reggia

Newsome, W., Maunsell, J., & van Essen, D. (1986). Ventral posterior visual area of the macaque: Visual topography and areal boundaries. J. Comparative Neurology, 2(252), 139–153. Obermayer, K., Blasdel, G., & Schulten, K. (1992). A statistical mechanical analysis of self-organization and pattern formation during the development of visual maps. Physical Review A, 45(10), 7568–7589. Obermayer, K., Ritter, H., & Schulten, K. (1990). A principle for the formation of the spatial structure of cortical feature maps. Proc. National Academy of Sciences USA, 87, 8345–8349. Obermayer, K., Schulten, K., & Blasdel, G. (1992). A comparison of a neural network model for the formation of brain maps with experimental data. In J. Moody, S. Hanson, & R. Lippmann (Eds.), Advances in neural information processing systems, 4 (pp. 83–90). San Mateo, CA: Morgan Kaufmann. Palakal, M., Murthy, U., Chittajallu, S., & Wong, D. (1995). Tonotopic representation of auditory responses using self-organizing maps. Mathematical and Computer Modelling, 22(2), 7–21. Pantev, C., Bertrand, O., Eulitz, C., Verkindt, C., Hampson, S., Schuierer, G., & Elbert, T. (1995). Specific tonotopic organizations of different areas of the human auditory cortex revealed by simultaneous magnetic and electric recordings. Electroencephalography and Clinical Neurophysiology, 1(94), 26–40. Papanicolaou, A., Simos, P., Breier, J., Fletcher, J., Foorman, B., Francis, D., Costillo, E., & Davis, R. (2003). Brain mechanisms for reading in children with and without dyslexia. Dev. Neuropsychol., 24, 593–612. Pearson, J., Finkel, L., & Edelman, G. (1987). Plasticity in the organization of adult cerebral cortical maps: A computer simulation based on neuronal group selection. J. Neuroscience, 7, 4209–4223. Pei, X., Vidyasagar, T., Volgushev, M., & Creutzfeldt, O. (1994). Receptive field analysis and orientation selectivity of postsynaptic potentials of simple cells in car visual cortex. J. Neuroscience, 11(14), 7130–7140. Reggia, J., D’Autrechy, C., Sutton, G., & Weinrich, M. (1992). A competitive distribution theory of neo-cortical dynamics. Neural Computation, 4, 287–317. Ritter, H., Martinetz, T., & Schulten, K. (Eds.). (1992). Neural computation and selforganizing maps. Reading, MA: Addison-Wesley. Ritter, H., & Schulten, K. (1986). On the stationary state of Kohonen’s self-organizing sensory mapping. Biological Cybernetics, 54, 99–106. Schulz, R., & Reggia, J. (2004). Temporally asymmetric learning supports sequence processing in multi-winner self-organizing maps. Neural Computation, 16(3), 535– 561. Sereno, M., Dale, A., Reppas, J., Kwong, K., Belliveau, J., Brady, T., Rosen, B., & Tootell, R. (1995). Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science, 268(5212), 889–893. Sirosh, J., & Miikkulainen, R. (1994). Cooperative self-organization of afferent and lateral connections in cortical maps. Biological Cybernetics, 71, 65–78. Sur, M., & Leamey, C. (2001). Development and plasticity of cortical areas and networks. Nature Reviews Neuroscience, 2, 251–262. Sur, M., Nelson, R., & Kaas, J. (1982). Representations of the body surface in cortical areas 3b and 1 of squirrel monkeys: Comparisons with other primates. J. Comparative Neurology, 2(211), 177–192.

Symmetric Maps

1083

Sutton, G., Reggia, J., Armentrout, S., & D’Autrechy, C. (1994). Cortical map reorganization as a competitive process. Neural Computation, 6, 1–13. Talavage, T., Ledden, P., & Benson, R. (2000). Frequency-dependent responses exhibited by multiple regions in human auditory cortex. Hearing Research, 150, 225–244. Temple, E., Deutsch, G., Poldrack, R., Miller, S., Tallal, P., Merzenich, M., & Gabrieli, J. (2003). Neural deficits in children with dyslexia ameliorated by behavioral remediation. Proc. Nat. Acad. Sci., 100, 2860–2865. Tiao, Y., & Blakemore, C. (1976). Functional organization in the visual cortex of the golden hamster. J. Comparative Neurology, 4(168), 459–481. Villmann, T., Der, R., Herrmann, M., & Martinetz, T. (1997). Topology preservation in self-organizing feature maps: Exact definition and measurement. IEEE Transactions on Neural Networks, 8(2), 256–266. von der Malsburg, C. (1973). Self-organization of orientation sensitive cells in the striate cortex. Kybernetik, 14, 85–100. Zhou, R., & Black, I. (2000). Development of neural maps. In M. Gazzaniga (Ed.), The new cognitive neurosciences (pp. 213–236), Cambridge, MA: MIT Press.

Received June 11, 2004; accepted October 14, 2004.

LETTER

Communicated by Tamar Flash

Stochastic Optimal Control and Estimation Methods Adapted to the Noise Characteristics of the Sensorimotor System Emanuel Todorov [email protected] Department of Cognitive Science, University of California San Diego, La Jolla CA 92093-0515.

Optimality principles of biological movement are conceptually appealing and straightforward to formulate. Testing them empirically, however, requires the solution to stochastic optimal control and estimation problems for reasonably realistic models of the motor task and the sensorimotor periphery. Recent studies have highlighted the importance of incorporating biologically plausible noise into such models. Here we extend the linear-quadratic-gaussian framework—currently the only framework where such problems can be solved efficiently—to include controldependent, state-dependent, and internal noise. Under this extended noise model, we derive a coordinate-descent algorithm guaranteed to converge to a feedback control law and a nonadaptive linear estimator optimal with respect to each other. Numerical simulations indicate that convergence is exponential, local minima do not exist, and the restriction to nonadaptive linear estimators has negligible effects in the control problems of interest. The application of the algorithm is illustrated in the context of reaching movements. A Matlab implementation is available at www.cogsci.ucsd.edu/∼todorov. 1 Introduction Many theories in the physical sciences are expressed in terms of optimality principles, which often provide the most compact description of the laws governing a system’s behavior. Such principles play an important role in the field of sensorimotor control as well (Todorov, 2004). A quantitative theory of sensorimotor control requires a precise definition of success in the form of a scalar cost function. By combining top-down reasoning with intuitions derived from empirical observations, researchers have proposed a number of hypothetical cost functions for biological movement. While such hypotheses are not difficult to formulate, comparing their predictions to experimental data is complicated by the fact that the predictions have to be derived in the first place—that is, the hypothetical optimal control and estimation problems have to be solved. The most popular approach has been to optimize, in an open loop, the sequence of control signals (Chow & Jacobson, Neural Computation 17, 1084–1108 (2005)

© 2005 Massachusetts Institute of Technology

Methods for Optimal Sensorimotor Control

1085

1971; Hatze & Buys, 1977; Anderson & Pandy, 2001) or limb states (Nelson, 1983; Flash & Hogan, 1985; Uno, Kawato, & Suzuki, 1989; Harris & Wolpert, 1998). For stochastic partially observable plants such as the musculoskeletal system, however, open-loop approaches yield suboptimal performance (Todorov & Jordan, 2002b; Todorov, 2004). Optimal performance can be achieved only by a feedback control law, which uses all sensory data available online to compute the most appropriate muscle activations under the circumstances. Optimization in the space of feedback control laws is studied in the related fields of stochastic optimal control, dynamic programming, and reinforcement learning. Despite many advances, the general-purpose methods that are guaranteed to converge in a reasonable amount of time to a reasonable answer remain limited to discrete state and action spaces (Bertsekas & Tsitsiklis, 1997; Sutton & Barto, 1998; Kushner & Dupuis, 2001). Discretization methods are well suited for higher-level control problems, such as the problem faced by a rat that has to choose which way to turn in a twodimensional maze. But the main focus in sensorimotor control is on a different level of analysis: on how the rat chooses a hundred or so graded muscle activations at each point in time, in a way that causes its body to move toward the reward without falling or hitting walls. Even when the musculoskeletal system is idealized and simplified, the state and action spaces of interest remain continuous and high-dimensional, and the curse of dimensionality prevents the use of discretization methods. Generalizations of these methods to continuous high-dimensional spaces typically involve function approximations whose properties are not yet well understood. Such approximations can produce good enough solutions, which is often acceptable in engineering applications. However, the success of a theory of sensorimotor control ultimately depends on its ability to explain data in a principled manner. Unless the theory’s predictions are close to the globally optimal solution of the hypothetical control problem, it is difficult to determine whether the (mis)match to experimental data is due to the general (in)applicability of optimality ideas to biological movement, or the (in)appropriateness of the specific cost function, or the specific approximations—in both the plant model and the controller design—used to derive the predictions. Accelerated progress will require efficient and well-understood methods for optimal feedback control of stochastic, partially observable, continuous, nonstationary, and high-dimensional systems. The only framework that currently provides such methods is linear-quadratic-gaussian (LQG) control, which has been used to model biological systems subject to sensory and motor uncertainty (Loeb, Levine, & He, 1990; Hoff, 1992; Kuo, 1995). While optimal solutions can be obtained efficiently within the LQG setting (via Riccati equations), this computational efficiency comes at the price of reduced biological realism, because (1) musculoskeletal dynamics are generally nonlinear, (2) behaviorally relevant performance criteria are

1086

E. Todorov

unlikely to be globally quadratic (Kording & Wolpert, 2004), and (3) noise in the sensorimotor apparatus is not additive but signal-dependent. The third limitation is particularly problematic because it is becoming increasingly clear that many robust and extensively studied phenomena—such as trajectory smoothness, speed-accuracy trade-offs, task-dependent impedance, structured motor variability and synergistic control, and cosine tuning— are linked to the signal-dependent nature of sensorimotor noise (Harris & Wolpert, 1998; Todorov, 2002; Todorov & Jordan, 2002b). It is thus desirable to extend the LQG setting as much as possible and adapt it to the online control and estimation problems that the nervous system faces. Indeed, extensions are possible in each of the three directions listed above: 1. Nonlinear dynamics (and nonquadratic costs) can be approximated in the vicinity of the expected trajectory generated by an existing controller. One can then apply modified LQG methodology to the approximate problem and use it to improve the existing controller iteratively. Differential dynamic programming (Jacobson & Mayne, 1970), as well as iterative LQG methods (Li & Todorov, 2004; Todorov & Li, 2004), are based on this general idea. In their present form, most such methods assume deterministic dynamics, but stochastic extensions are possible (Todorov & Li, 2004). 2. Quadratic costs can be replaced with a parametric family of exponential-of-quadratic costs, for which optimal LQG-like solutions can be obtained efficiently (Whittle, 1990; Bensoussan, 1992). The controllers that are optimal for such costs range from risk averse (i.e., robust), through classic LQG, to risk seeking. This extended family of cost functions has not yet been explored in the context of biological movement. 3. Additive gaussian noise in the plant dynamics can be replaced with multiplicative noise, which is still gaussian but has standard deviation proportional to the magnitude of the control signals or state variables. When the state of the plant is fully observable, optimal LQG-like solutions can be computed efficiently, as shown by several authors (Kleinman, 1969; McLane, 1971; Willems & Willems, 1976; Bensoussan, 1992; El Ghaoui, 1995; Beghi & D’Alessandro, 1998; Rami, Chen, & Moore, 2001). Such methodology has also been used to model reaching movements (Hoff, 1992). Most relevant to the study of sensorimotor control, however, is the partially observable case, which remains an open problem. While some work along these lines has been done (Pakshin, 1978; Phillis, 1985), it has not produced reliable algorithms that one can use off the shelf in building biologically relevant models (see section 9). Our goal here is to address that problem, and provide the model-building methodology that is needed.

Methods for Optimal Sensorimotor Control

1087

Table 1: List of Notation. xt ∈ Rm ut ∈ R p yt ∈ Rk n A, B, H ξ t , ω t , εt , t , η t
ξ ,
ω ,
ε ,
 ,
η C1 , . . . , Cc D1 , . . . , Dd Qt , R
xt et t
x xe te , 
t , t vt Stx , Ste , st Kt Lt

state vector at time step t control signal sensory observation total number of time steps system dynamics and observation matrices zero-mean noise terms covariances of noise terms scaling matrices for control-dependent system noise scaling matrices for state-dependent observation noise matrices defining state- and control-dependent costs state estimate estimation error conditional estimation error covariance unconditional covariances optimal cost-to-go function parameters of the optimal cost-to-go function filter gain matrices control gain matrices

In this letter, we define an extended noise model that reflects the properties of the sensorimotor system; derive an efficient algorithm for solving the stochastic optimal control and estimation problems under that noise model; illustrate the application of this extended LQG methodology in the context of reaching movements; and study the properties of the new algorithm through extensive numerical simulations. A special case of the algorithm derived here has already allowed us (Todorov & Jordan, 2002b) to construct models of a wider range of empirical results than previously possible. In section 2 we motivate our extended noise model, which includes control-dependent, state-dependent, and internal estimation noise. In section 3 we formalize the problem and restrict the feedback control laws under consideration to functions of state estimates that are obtained by unbiased nonadaptive linear filters. In section 4 we compute the optimal feedback control law for any nonadaptive linear filter and show that it is linear in the state estimate. In section 5 we derive the optimal nonadaptive linear filter for any linear control law. The two results together provide an iterative coordinate-descent algorithm (equations 4.2 and 5.2), which is guaranteed to converge to a filter and a control law optimal with respect to each other. In section 6 we illustrate the application of our method to the analysis of reaching movements. In section 7 we explore numerically the convergence properties of the algorithm and observe exponential convergence with no local minima. In section 8 we assess the effects of assuming a nonadaptive linear filter and find them to be negligible for the control problems of interest. Table 1 shows the notation used in this letter.

1088

E. Todorov

2 Noise Characteristics of the Sensorimotor System Noise in the motor output is not additive but instead increases with the magnitude of the control signals. This is intuitively obvious: if you rest your arm on the table, it does not bounce around (i.e., the passive plant dynamics have little noise), but when you make a movement (i.e., generate control signals), the outcome is not always as desired. Quantitatively, the relationship between motor noise and control magnitude is surprisingly simple. Such noise has been found to be multiplicative: the standard deviation of muscle force is well fit with a linear function of the mean force, in both static (Sutton & Sykes, 1967; Todorov, 2002) and dynamic (Schmidt, Zelaznick, Hawkins, Frank, & Quinn, 1979) isometric force tasks. The exact reasons for this dependence are not entirely clear, although it can be explained at least in part with Poisson noise on the neural level combined with Henneman’s size principle of motoneuron recruitment (Jones, Hamilton, & Wolpert, 2002). To formalize the empirically established dependence, let u be a vector of control signals (corresponding to the muscle activation levels that the nervous system attempts to set) and ε be a vector of zero-mean random numbers. A general multiplicative noise model takes the form C(u)ε, where C(u) is a matrix whose elements depend linearly on u. To express a linear relationship between a vector u and a matrix C, we make the ith column of C equal to Ci u, where Ci are constant scaling matrices. Then we have C(u)ε = i Ci uεi , where εi is the ith component of the random vector ε. Online movement control relies on feedback from a variety of sensory modalities, with vision and proprioception typically playing the dominant role. Visual noise obviously depends on the retinal position of the objects of interest and increases with distance away from the fovea (i.e., eccentricity). The accuracy of visual positional estimates is again surprisingly well modeled with multiplicative noise, whose standard deviation is proportional to eccentricity. This is an instantiation of Weber’s law and has been found to be quite robust in a variety of interval discrimination experiments (Burbeck & Yap, 1990; Whitaker & Latham, 1997). We have also confirmed this scaling law in a visuomotor setting, where subjects pointed to memorized targets presented in the visual periphery (Todorov, 1998). Such results motivate the use of a multiplicative observation noise model of the form D (x) = i Di x i , where x is the state of the plant and environment, including the current fixation point and the positions and velocities of relevant objects. Incorporating state-dependent noise in analyses of sensorimotor control can allow more accurate modeling of the effects of feedback and various experimental perturbations; it also can effectively induce a cost function over eye movement patterns and allow us to predict the eye movements that would result in optimal hand performance (Todorov, 1998). Note that if other forms of state-dependent sensory noise are found, the model can still be useful as a linear approximation.

Methods for Optimal Sensorimotor Control

1089

Intelligent control of a partially observable stochastic plant requires a feedback control law, which is typically a function of a state estimate that is computed recursively over time. In engineering applications, the estimation-control loop is implemented in a noiseless digital computer, and so all noise is external. In models of biological movement, we usually make the same assumption, treating all noise as being a property of the musculoskeletal plant or the sensory apparatus. This is in principle unrealistic, because neural representations are likely subject to internal fluctuations that do not arise in the periphery. It is also unrealistic in modeling practice. An ideal observer model predicts that the estimation error covariance of any stationary feature of the environment will asymptote to 0. In particular, such models predict that if we view a stationary object in the visual periphery long enough, we should eventually know exactly where it is and be able to reach for it as accurately as if it were at the center of fixation. This contradicts our intuition as well as experimental data. Both interval discrimination experiments and reaching to remembered peripheral targets experiments indicate that estimation errors asymptote rather quickly, but not to 0. Instead, the asymptote level depends linearly on eccentricity. The simplest way to model this is to assume another noise process, which we call internal noise, acting directly on whatever state estimate the nervous system chooses to compute. 3 Problem Statement and Assumptions Consider a linear dynamical system with state xt ∈ Rm , control ut ∈ R p , feedback yt ∈ Rk , in discrete time t: Dynamics

xt+1 = Axt + But + ξ t +

Feedback

yt = Hxt + ω t +

Cost per step

xtT Qt xt + utT Rut

d 

c 

εti Ci ut

i=1

ti Di xt

(3.1)

i=1

The feedback signal yt is received after the control signal ut has been generated. The initial state has known mean
x1 and covariance 1 . All matrices are known and have compatible dimensions; making them time varying is straightforward. The control cost matrix R is symmetric positive definite (R > 0), and the state cost matrices Q1 , . . . , Qn are symmetric positive semidefinite (Qt ≥ 0). Each movement lasts n time steps; at t = n, the final cost is xnT Qn xn , and un is undefined. The independent random variables ξ t ∈ Rm , ω t ∈ Rk , εt ∈ Rc , and t ∈ Rd have multidimensional gaussian distributions with mean 0 and covariances
ξ ≥ 0,
ω > 0,
ε = I and
 = I respectively. Thus, the control-dependent and state-dependent noise terms   in equation 3.1 have covariances i Ci ut uTt CiT and i Di xt xTt DiT . When the

1090

E. Todorov

control-dependent noise is meant to be added to the control signal (which is usually the case), the matrices Ci should have the form B Fi where Fi are the actual noise scaling factors.  Then the control-dependent part of the plant dynamics becomes B(I + i εti Fi )ut . The problem of optimal control is to find the optimal control law, that is, the sequence of causal control functions ut (u1 , . . . , ut−1 , y1 , . . . , yt−1 ) that minimize the expected total cost over the movement. Note that computing the optimal sequence of functions u1 (·), . . . , un−1 (·) is a different, and in general much more difficult, problem than computing the optimal sequence of open-loop controls u1 , . . . , un−1 . When only additive noise is present (i.e., C1 , . . . , Cc = 0 and D1 , . . . , Dd = 0), this reduces to the classic LQG problem, which has the well-known optimal solution (Davis & Vinter, 1985) Linear-Quadratic Regulator ut = −L t
xt

Kalman Filter
xt + But + K t (yt − H
xt ) xt+1 = A


L t = (R + B T St+1 B)−1 B T St+1 A

K t = At H T (Ht H T +
ω )−1

St = Qt + AT St+1 (A − B L t )

t+1 =
ξ + (A − K t H) t AT

(3.2)

In that case, the optimal control law depends on the history of control and feedback signals only through the state estimate
xt , which is updated recursively by the Kalman filter. The matrices L that define the optimal control law do not depend on the noise covariances or filter coefficients, and the matrices K that define the optimal filter do not depend on the cost and control law. In the case of control-dependent and state-dependent noise, the above independence properties no longer hold. This complicates the problem substantially and forces us to adopt a more restricted formulation in the interest of analytical tractability. We assume that, as in equation 3.2, the entire history of control and feedback signals is summarized by a state estimate
xt , which is all the information available to the control system at time t. The feedback control law ut (·) is allowed to be an arbitrary function of
xt , but
xt can be updated only by a recursive linear filter of the form:
xt + But + K t (yt − H
xt ) + ηt . xt+1 = A
The internal noise η t ∈ Rm has mean 0 and covariance
η ≥ 0. The filter gains K 1 , . . . , K n−1 are nonadaptive; they are determined in advance and cannot change as a function of the specific controls and observations within a simulation run. Such a filter is always unbiased: for any K 1 , . . . , K n−1 , we have E [xt |
xt ] =
xt for all t. Note, however, that under the extended noise model, any nonadaptive linear filter is suboptimal: when
xt is computed as defined above, Cov [xt |
xt ] is generally larger than Cov [xt |u1 , . . . , ut−1 , y1 , . . . , yt−1 ]. The consequences of this will be explored numerically in section 8.

Methods for Optimal Sensorimotor Control

1091

4 Optimal Controller The optimal ut will be computed using the method of dynamic programming. We will show by induction that if the true state at time t is xt and the unbiased state estimate available to the control system is
xt , then the optimal cost-to-go function (i.e., the cost expected to accumulate under the optimal control law) has the quadratic form vt (xt ,
xt ) = xTt Stx xt + (xt −
xt )T Ste (xt −
xt ) + st = xTt Stx xt + eTt Ste et + st , xt is the estimation error. At the final time t = n, the optimal where et
xt −
cost-to-go is simply the final cost xnT Qn xn , and so vn is in the assumed form with Snx = Qn , Sne = 0, sn = 0. To carry out the induction proof, we have to show that if vt+1 is in the above form for some t < n, then vt is also in that form. Consider a time-varying control law that is optimal at times t + 1, . . . , n, and at time t is given by ut = π (
xt ). Let vtπ (xt ,
xt ) be the corresponding cost-to-go function. Since this control law is optimal after time t, we have π vt+1 = vt+1 . Then the cost-to-go function vtπ satisfies the Bellman equation: xt ) = xTt Qt xt + π(
xt )T Rπ (
xt ) + E [vt+1 (xt+1 ,
xt+1 )|xt ,
xt , π]. vtπ (xt ,
To compute the above expectation term, we need the update equations for the system variables. Using the definitions of the observation yt and the estimation error et , the stochastic dynamics of the variables of interest become xt+1 = Axt + Bπ (
xt ) + ξ t +



εti Ci π(
xt )

i

et+1 = (A − K t H)et + ξ t − K t ω t − η t +



εti Ci π (
xt ) −

i



ti K t Di xt .

i

(4.1) Then the conditional means and covariances of xt+1 and et+1 are xt , π ] = Axt + Bπ(
xt ) E [xt+1 |xt ,
E [et+1 |xt ,
xt , π ] = (A − K t H)et  Cov [xt+1 |xt ,
xt , π ] =
ξ + Ci π(
xt )π (
xt )T CiT i

Cov [et+1 |xt ,
xt , π ] =
+ ξ



Ci π(
xt )π (
xt )T CiT +
η

i ω

+ K t
K tT +

 i

K t Di xt xTt DiT K tT ,

1092

E. Todorov

and the conditional expectation in the Bellman equation can be computed. The cost-to-go becomes   x xt ) = xTt Qt + AT St+1 A + Dt xt vtπ (xt ,
e + eTt (A − K t H)T St+1 (A − K t H)et   T x + tr (Mt ) + π (
B + Ct π (
xt ) xt ) R + B T St+1 x Axt , + 2π(
xt )T B T St+1

where we defined the shortcuts   e  x Ci , CiT St+1 + St+1 Ct
i

Dt




e DiT K tT St+1 K t Di ,

and

i

 ξ  x e
ξ + St+1
+
η + K t
ω K tT . Mt
St+1 Note that the control law affects only the cost-go-to function through an expression that is quadratic in π(
xt ), which can be minimized analytically. But there is a problem: the minimum depends on xt , while π is only allowed to be a function of
xt . To obtain the optimal control law at time t, we have to take an expectation over xt conditional on
xt , and find the function π that minimizes the resulting expression. Note that the control-dependent expression is linear in xt , and so its expectation depends on the conditional mean of xt but not on any higher moments. Since E [xt |
xt ] =
xt , we have  π    x E vt (xt ,
xt )T R + B T St+1 xt ) xt )|
xt = const + π (
B + Ct π (
x A
xt , + 2π(
xt )T B T St+1

and thus the optimal control law at time t is  −1 T x x xt ; xt ) = −L t
L t
R + B T St+1 B + Ct B St+1 A. ut = π(
Note that the linear form of the optimal control law fell out of the optimization and was not assumed. Given our assumptions, the matrix being inverted is symmetric positive-definite. To complete the induction proof, we have to compute the optimal costto-go vt , which is equal to vtπ when π is set to the optimal control law −L t
xt . x x x Using the fact that L Tt (R + B T St+1 B + Ct )L t = L Tt B T St+1 A = AT St+1 B L t , and that
xT Z
x − 2
xT Zx = (x −
x )T Z(x −
x ) − xT Zx = eT Ze − xT Zx for a symx metric matrix Z (in our case equal to L Tt B T St+1 A), the result is   x xt ) = xTt Qt + AT St+1 (A − B L t ) + Dt xt + tr (Mt ) + st+1 vt (xt ,
  x e + eTt AT St+1 B L t + (A − K t H)T St+1 (A − K t H) et .

Methods for Optimal Sensorimotor Control

1093

We now see that the optimal cost-to-go function remains in the assumed quadratic form, which completes the induction proof. The optimal control law is computed recursively backward in time as Controller ut = −L t
xt −1     x e x x Ci L t = R + B T St+1 B+ CiT St+1 + St+1 B T St+1 A i  e x Stx = Qt + AT St+1 (A − B L t ) + DiT K tT St+1 K t Di ; Snx = Qn i

x e Ste = AT St+1 B L t + (A − K t H)T St+1 (A − K t H) ;  ξ   x e ξ η st = tr St+1
+ St+1
+
+ K t
ω K tT + st+1 ;

(4.2)

Sne = 0 sn = 0.

x1 + tr((S1x + S1e )1 ) + s1 . The total expected cost is
x T1 S1x
When the control-dependent and state-dependent noise terms are removed (i.e., C1 , . . . , Cc = 0, D1 , . . . , Dd = 0), the control laws given by equation 4.2 and 3.2 are identical. The internal noise term η, as well as the additive noise terms ξ and ω, do not directly affect the calculation of the feedback gain matrices L. However, all noise terms affect the calculation (see below) of the optimal filter gains K , which in turn affect L. One can attempt to transform equation 3.1 into a fully observable system by setting H = I ,
ω =
η = 0, D1 , . . . , Dd = 0, in which case K = A, and apply equation 4.2. Recall, however, our assumption that the control signal is generated before the current state is measured. Thus, even if we make the sensory measurement equal to the state, we would still be dealing with a partially observable system. To derive the optimal controller for the fully observable case, we have to assume that xt is known at the time when ut is generated. The above derivation is now much simplified: the optimal costto-go function vt is in the form xTt St xt + st , and the expectation term that needs to be minimized with regard to ut = π(xt ) becomes E [vt+1 ] = (Axt + But )T St+1 (Axt + But )

  T T Ci St+1 Ci ut + tr [St+1
ξ ] + st+1 , + ut i

and the optimal controller is computed in a backward pass through time as ut = −L t xt −1  T T R + B St+1 B + Ci St+1 Ci B T St+1 A

Fully observable controller

Lt =

i

St = Qt + A St+1 (A − B L t );

Sn = Qn

st = tr (St+1
ξ ) + st+1 ;

sn = 0.

T

(4.3)

1094

E. Todorov

5 Optimal Estimator So far, we have computed the optimal control law L for any fixed sequence of filter gains K . What should these gains be fixed to? Ideally they should correspond to a Kalman filter, which is the optimal linear estimator. However, in the presence of control-dependent and state-dependent noise, the Kalman filter gains become adaptive (i.e., K t depends on
xt and ut ), which would make our control law derivation invalid. Thus, if we want to preserve the optimality of the control law given by equation 4.2 and obtain an iterative algorithm with guaranteed convergence, we need to compute a fixed sequence of filter gains that are optimal for a given control law. Once the iterative algorithm has converged and the control law has been designed, we could use an adaptive filter in place of the fixed-gain filter in run time (see section 8). Thus, our objective here is the following: given a linear feedback control law L 1 , . . . , L n−1 (which is optimal for the previous filter K 1 , . . . , K n−1 ), compute a new filter that, in conjunction with the given control law, results in minimal expected cost. In other words, we will evaluate the filter not by the magnitude of its estimation errors, but by the effect that these estimation errors have on the performance of the composite estimation-control system. We will show that the new optimal filter can be designed in a forward pass through time. In particular, we will show that regardless of the new values of K 1 , . . . , K t−1 , the optimal K t can be found analytically as long as K t+1 , . . . , K n−1 still have the values for which L t+1 , . . . , L n−1 are optimal. Recall that the optimal L t+1 , . . . , L n−1 depend only on K t+1 , . . . , K n−1 , and so the parameters (as well as the form) of the optimal cost-to-go function vt+1 cannot be affected by changing K 1 , . . . , K t . Since K t affects only the computation of
xt+1 , and the effect of
xt+1 on the total expected cost is captured by the function vt+1 , we have to minimize vt+1 with respect to K t . But v is a function of x and
x, while K cannot be adapted to the specific values of x and
x within a simulation run (by assumption). Thus, the quantity we have to minimize is the unconditional expectation of vt+1 . In doing so, we will use that fact that E [vt+1 (xt+1 ,
xt+1 )] = Ext ,
xt [E [vt+1 (xt+1 ,
xt+1 )|xt ,
xt , L t ]]. The conditional expectation was already computed as an intermediate step in the previous section (not shown). The terms in E [vt+1 (xt+1 ,
xt+1 )|xt ,
xt , L t ] that depend on K t are



   T T e ω T T T e Di xt xt Di K t St+1 . et (A − K t H) St+1 (A − K t H)et + tr K t
+ i

Defining the (uncentered) unconditional covariances te
E [et eTt ] and tx
E [xt xTt ], the unconditional expectation of the K t -dependent expression

Methods for Optimal Sensorimotor Control

1095

above becomes  e   ; a (K t ) = tr (A − K t H)te (A − K t H)T + K t Pt K tT St+1  Pt

ω + Di tx DiT . i

The minimum of a (K t ) is found by setting its derivative with regard to K t to  0. Using the matrix identities ∂∂X tr (XU) = U T and ∂∂X tr XU XT V = V XU + e V T XU T , and the fact that the matrices St+1 ,
ω , te , tx are symmetric, we obtain     ∂a (K t ) e = 2St+1 K t Hte H T + Pt − Ate H T . ∂ Kt This expression is equal to 0 whenever K t = Ate H T (Hte H T + Pt )−1 , ree gardless of the value of St+1 . Given our assumptions, the matrix being inverted is symmetric positive-definite. Note that the optimal K t depends on K 1 , . . . , K t−1 (through te and tx ) but is independent of K t+1 , . . . , K n−1 e (since it is independent of St+1 ). This is the reason that the filter gains are reoptimized in a forward pass. To complete the derivation, we have to substitute the optimal filter gains and compute the unconditional covariances. Recall that the variables xt ,
xt , et are deterministically related by et = xt −
xt , so the covariance of any one of them can be computed given the covariances of the other two, and we have a choice of which pair of covariance matrices to compute. The resulting equations are most compact for the pair
xt , et . The stochastic dynamics of these variables are 
xt + K t Het + K t ω t + η t + ti K t Di (et +
xt ). xt+1 = (A − B L t )
i  xt et+1 = (A − K t H)et + ξ t − K t ω t − η t − εti Ci L t
(5.1) i  − ti K t Di (et +
xt ). i

Define the unconditional covariances,    T 
tx
E
xt
te
E et eTt ; xt ;

 T 
txe
E
x t et ,

x1 is a known constant, noting that 
tx is uncentered and te
x = (
txe )T . Since
the initialization at t = 1 is 1e = 1 , 
1x =
x1
xT1 , 
1xe = 0. With these defiT nitions, we have tx = E [(et +
xt )(et +
xt )T ] = te + 
tx + 
txe + 
txe . Using equation 5.1, the updates for the unconditional covariances are e = (A − K t H)te (A − K t H)T +
ξ +
η + K t Pt K tT t+1  + Ci L t 
tx L Tt CiT i

1096

E. Todorov

  x 
t+1 = (A − B L t )
tx (A − B L t )T +
η + K t Hte H T + Pt K tT + (A − B L t )
txe H T K tT + K t Hte
x (A − B L t )T xe 
t+1 = (A − B L t )
txe (A − K t H)T + K t Hte (A − K t H)T

−
η − K t Pt K tT . Substituting the optimal value of K t , which allows some simplifications to the above update equations, the optimal nonadaptive linear filter is computed in a forward pass through time as Estimator
xt+1 = (A − B L t )
xt + K t (yt − H
xt ) + η t     T −1

e T e T ω e e
x x xe K t = At H Ht H +
+ Di t + t + t + t Di i e t+1

ξ

η

=
+
+ (A −

K t H)te AT

+



Ci L t 
tx L Tt CiT ;

1e = 1

i x 
t+1 =
η + K t Hte AT + (A − B L t )
tx (A − B L t )T

+ (A − B L t )
txe H T K tT + K t Hte
x (A − B L t )T ; xe 
t+1 = (A − B L t ) 
txe (A − K t H)T −
η ;


1x =
x1
xT1 
1xe = 0.

(5.2) It is worth noting the effects of the internal noise η t . If that term did not exist (i.e.,
η = 0), the last update equation would yield 
txe = 0 for all t. Indeed, for an optimal filter, one would expect 
txe = 0 from the orthogonality principle: if the state estimate and estimation error were correlated, one could improve the filter by taking that correlation into account. However, the situation here is different because we have noise acting directly on the state estimate. When such noise pushes
xt in one direction, et is (by definition) pushed in the opposite direction, creating a negative correlation between
xt and et . This is the reason for the negative sign in front of the
η term in the last update equation. The complete algorithm is the following: Algorithm: Initialize K 1 , . . . , K n−1 , and iterate equation 4.2 and equation 5.2 until convergence. Convergence is guaranteed, because the expected cost is nonnegative by definition, and we are using a coordinate-descent algorithm, which decreases the expected cost in each step. The initial sequence K could be set to 0—in which case, the first pass of equation 4.2 will find the optimal open-loop controls, or initialized from equation 3.2—which is equivalent to assuming additive noise in the first pass. We can also derive the optimal adaptive linear filter, with gains K t that depend on the specific
xt and ut = −L t
xt within each simulation run. This is again accomplished by minimizing E [vt+1 ] with respect to K t , but the expectation is computed with
xt being a known constant rather than a random xTt and 
txe = 0, and so the last two update variable. We now have 
tx =
xt


Methods for Optimal Sensorimotor Control

1097

equations in equation 5.2 are no longer needed. The optimal adaptive linear filter is Adaptive estimator
xt+1 = (A − B L t )
xt ) + ηt xt + K t (yt − H
 −1    K t = At H T Ht H T +
ω + Di t +
xt
xTt DiT i  t+1 =
ξ +
η + (A − K t H) t AT + Ci L t
xt
xTt L Tt CiT ,

(5.3)

i

xt ] is the conditional estimation error covariance where t = Cov [xt |
(initialized from 1 , which is given). When the control-dependent, state-dependent, and internal noise terms are removed (C1 , . . . , Cc = 0, D1 , . . . , Dd = 0,
η = 0), equation 5.3 reduces to the Kalman filter in equation 3.2. Note that using equation 5.3 instead of equation 5.2 online reduces the total expected cost because equation 5.3 achieves lower estimation error than any other linear filter, and the expected cost depends on the conditional estimation error covariance. This can be seen from    E[vt (xt ,
xt )|
xt ] =
xTt Stx
xt ] xt + st + tr Stx + Ste Cov[xt |
6 Application to Reaching Movements We now illustrate how the methodology developed above can be used to construct models relevant to motor control. Since this is a methodological rather than a modeling article, a detailed evaluation of the resulting models in the context of the motor control literature will not be given here. The first model is a one-dimensional model of reaching, and includes control-dependent noise but no state-dependent or internal noise. The latter two forms of noise are illustrated in the second model, where we estimate the position of a stationary peripheral target without making a movement. 6.1 Models. We model a single-joint movement (such as flexing the elbow) that brings the hand to a specified target. For simplicity, the rotational motion is replaced with translational motion; the hand is modeled as a point mass (m = 1 kg) whose one-dimensional position at time t is p(t). The combined action of all muscles is represented with the force f (t) acting on the hand. The control signal u(t) is transformed into force f (t) by adding controldependent multiplicative noise and applying a second-order muscle-like ˙ + f (t) = low-pass filter (Winter, 1990) of the form τ1 τ2 f¨ (t) + (τ1 + τ2 ) f(t) u(t), with time constants τ1 = τ2 = 0.04 sec. Note that a second-order filter can be written as a pair of coupled first-order filters (with outputs g and f ) ˙ + g(t) = u(t), τ2 f˙ (t) + f (t) = g(t). as follows: τ1 g(t) The task is to move the hand from the starting position p(0) = 0 m to the target position p ∗ = 0.1 m and stop there at time tend , with minimal energy

1098

E. Todorov

consumption. Movement durations are in the interval tend ∈ [0.25 sec; 0.35 sec]. Time is discretized at  = 0.01 sec. The total cost is defined as ( p(tend ) − p ∗ )2 + (wv p˙ (tend ))2 + (w f f (tend ))2 +

n−1 r  u(k)2 . n − 1 k=1

The first term enforces positional accuracy; the second and third terms specify that the movement has to stop at time tend , that is, both the velocity and force have to vanish; and the last term penalizes energy consumption. It makes sense to set the scaling weights wv and w f so that wv p˙ (t) and w f f (t) averaged over the movement have magnitudes similar to the hand displacement p ∗ − p(0). For a 0.1 m reaching movement that lasts about 0.3 sec, these weights are wv = 0.2 and w f = 0.02. The weight of the energy term was set to r = 0.00001. The discrete-time system state is represented with the five-dimensional vector xt = [ p(t); p˙ (t); f (t); g(t); p ∗ ] initialized from a gaussian with mean
x1 = [0; 0; 0; 0; p ∗ ]. The auxiliary state variable g(t) is needed to implement a second-order filter. The target p ∗ is included in the state so that we can capture the above cost function using a quadratic with no linear terms: defining p = [1; 0; 0; 0; −1], we have pT xt = p(tend ) − p ∗ , and so xTt (ppT )xt = ( p(tend ) − p ∗ )2 . Note that the same could be accomplished by setting p = [1; 0; 0; 0; − p ∗ ] and xt = [ p(t); p˙ (t); f (t); g(t); 1]. The advantage of the formulation used here is that because the target is represented in the state, the same control law can be reused for other targets. The control law, of course, depends on the filter, which depends on the initial expected state, which depends on the target— and so a control law optimal for one target is not necessarily optimal for all other targets. Unpublished simulation results indicate good generalization, but a more detailed investigation of how the optimal control law depends on the target position is needed. The sensory feedback carries information about position, velocity, and force: yt = [ p(t); p˙ (t); f (t)] + ω t . The vector ω t of sensory noise terms has zero-mean gaussian distribution with diagonal covariance,
ω = (σs diag[0.02 m; 0.2 m/s; 1 N])2 , where the relative magnitudes are set using the same order-of-magnitude reasoning as before, and σs = 0.5. The multiplicative noise term added to the discrete-time control signal ut = u(t) is σc εt ut , where σc = 0.5. Note that

Methods for Optimal Sensorimotor Control

1099

σc is a unitless quantity that defines the noise magnitude relative to the control signal magnitude. The discrete-time dynamics of the above system are p(t + ) = p(t) + p˙ (t) p˙ (t + ) = p˙ (t) + f (t)/m f (t + ) = f (t) (1 − /τ2 ) + g(t)/τ2 g(t + ) = g(t) (1 − /τ1 ) + u(t) (1 + σc εt ) /τ1 , which is transformed into the form of equation 3.1 by the matrices 

1

0

0

0



  0 0  0 1 /m    A=   0 0 1 − /τ2 /τ2 0    0 1 − /τ1 0  0 0 0 0 0 0 1



0



   0     B=  0     /τ1  0



10000



  H = 0 1 0 0 0 00100 C1 = Bσc ; c = 1; d = 0 1 =
ξ =
η = 0.

The cost matrices are R = r , Q1,...,n−1 = 0, and Qn = ppT + vvT + ffT , where p = [1; 0; 0; 0; −1];

v = [0; wv ; 0; 0; 0];

f = [0; 0; w f ; 0; 0].

This completes the formulation of the first model. The above algorithm can now be applied to obtain the control law and filter, and the closed-loop system can be simulated. To replace the control-dependent noise with additive noise of similar magnitude (and compare the effects of the two forms of noise), we will set c = 0 and
ξ = (4.6 N)2 B B T . The value of 4.6 N is the average magnitude of the control-dependent noise over the range of movement durations (found through 10,000 simulation runs at each movement duration). We also model an estimation process under state-dependent and internal noise, in the absence of movement. In that case, the state is xt = p ∗ , where the stationary target p ∗ is sampled from a gaussian with mean
x1 ∈ {5 cm, 15 cm, 25 cm} and variance 1 = (5 cm)2 . Note that target eccentricity is represented as distance rather than visual angle. The state-dependent noise has scale D1 = 0.5, fixation is assumed to be at 0 cm, the time step is  = 10 msec, and we run the estimation process for n = 100 time steps. In one set of simulations, we use internal noise
η = (0.5 cm)2 without additive noise. In another set of simulations, we study additive noise with the same magnitude
ω = (0.5 cm)2 , without internal noise. There is no actuator to be controlled, so we have A = H = 1 and B = L = 0. Estimation is based on the adaptive filter from equation 5.3.

1100

E. Todorov

6.2 Results. Reaching movements are known to have stereotyped bellshaped speed profiles (Flash & Hogan, 1985). Models of this phenomenon have traditionally been formulated in terms of deterministic open-loop minimization of some cost function. Cost functions that penalize physically meaningful quantities (such as duration or energy consumption) did not agree with empirical data (Nelson, 1983); in order to obtain realistic speed profiles, it appeared necessary to minimize a smoothness-related cost that penalizes the derivative of acceleration (Flash & Hogan, 1985) or torque (Uno et al., 1989). Smoothness-related cost functions have also been used in the context of stochastic optimal feedback control (Hoff, 1992) to obtain bell-shaped speed profiles. It was recently shown, however, that smoothness does not have to be explicitly enforced by the cost function; open-loop minimization of end-point error was found sufficient to produce realistic trajectories, provided that the multiplicative nature of motor noise is taken into account (Harris & Wolpert, 1998). While this is an important step toward a more principled optimization model of trajectory smoothness, it still contains an ad hoc element: the optimization is performed in an open loop, which is suboptimal, especially for movements of longer duration. Our model differs from Harris and Wolpert (1998) in that not only the average sequence of control signals is optimal, but the feedback gains that determine the online sensory-guided adjustments are also optimal. Optimal feedback control of reaching has been studied by Meyer, Abrams, Kornblum, Wright, and Smith (1988) in an intermittent setting, and Hoff (1992) in a continuous setting. However, both of these models assume full state observation. Ours is the first optimal control model of reaching that incorporates sensory noise and combines state estimation and feedback control into an optimal sensorimotor loop. The predicted movement kinematics shown in Figure 1A closely resemble observed movement trajectories (Flash & Hogan, 1985). Another well-known property of reaching movements, first observed a century ago by Woodworth and later quantified as Fitts’ law, is the trade-off between speed and accuracy. The fact that faster movements are less accurate implies that the instantaneous noise in the motor system is controldependent, in agreement with direct measurements of isometric force fluctuations (Sutton and Sykes, 1967; Schmidt et al., 1979; Todorov, 2002) that show standard deviation increasing linearly with the mean. Naturally, this noise scaling has formed the basis of both closed-loop (Meyer et al., 1988; Hoff, 1992) and open-loop (Harris & Wolpert, 1998) optimization models of the speed-accuracy trade-off. Figure 1B illustrates the effect in our model: as the (specified) movement duration increases, the standard deviation of the end-point error achieved by the optimal controller decreases. To emphasize the need for incorporating control-dependent noise, we modified the model by making the noise in the plant dynamics additive, with fixed magnitude chosen to match the average multiplicative noise magnitude over the range of movement durations. With that change, the end-point error showed the opposite trend to the one observed experimentally (see Figure 1B).

Methods for Optimal Sensorimotor Control

1101

Figure 1: (A) Normalized position (Pos), velocity (Vel), and acceleration (Acc) of the average trajectory of the optimal controller. (B) A separate optimal controller was constructed for each instructed duration, the resulting closed-loop system was simulated for 10,000 trials, and the positional standard deviation at the end of the trial was plotted. This was done with either multiplicative (solid line) or additive (dashed line) noise in the plant dynamics. (C) The position of a stationary peripheral target was estimated over time, under internal estimation noise (solid line) or additive observation noise (dashed line). This was done in three sets of trials, with target positions sampled from gaussians with means 5 cm (bottom), 15 cm (middle), and 25 cm (top). Each curve is an average over 10,000 simulation runs.

It is interesting to compare the effects of the control penalty r and the multiplicative noise scaling σc . As equation 4.2 shows, both terms penalize large control signals—directly in the case of r and indirectly (via increased uncertainty) in the case of σc . Consequently, both terms lead to a negative bias in end-point position (not shown), but the effect is much more pronounced for r . Another consequence of the fact that larger controls are more costly arises in the control of redundant systems, where the optimal strategy is to follow a minimal intervention principle, that is, to leave task-irrelevant deviations from the average behavior uncorrected (Todorov & Jordan, 2002a, 2002b). Simulations have shown that this more complex effect is dependent on σc and actually decreases when r is increased while σc is kept constant (Todorov & Jordan, 2002b). Figure 1C shows simulation results from our second model, where the position of a stationary peripheral target is estimated by the optimal adaptive filter in equation 5.3, operating under internal estimation noise or additive observation noise of the same magnitude. In each case, we show results for three sets of targets with varying average eccentricity. The standard deviations of the estimation error always reach an asymptote (much faster in the case of internal noise). In the presence of internal noise, this asymptote depends on target eccentricity; for the chosen model parameters, the dependence is in quantitative agreement with our experimental results (Todorov, 1998). Under additive noise, the error always asymptotes to 0.

1102

E. Todorov

Figure 2: Relative change in expected cost as a function of iteration number, in (A) psychophysical models and (B) random models. (C) Relative variability (SD/mean) among expected costs obtained from 100 different runs of the algorithm on the same model (average over models in each class).

7 Convergence Properties We studied the convergence properties of the algorithm in 10 models of psychophysical experiments taken from Todorov and Jordan (2002b) and 200 randomly generated models. The psychophysical models had dynamics and cost functions similar to the above example. They included two models of planar reaching, three models of passing through sequences of targets, one model of isometric force production, three models of tracking and reaching with a mechanically redundant arm, and one model of throwing. The dimensionalities of the state, control, and feedback were between 5 and 20, and the horizon n was about 100. The psychophysical models included control-dependent dynamics noise and additive observation noise, but no internal or state-dependent noise. The details of all these models are interesting from a motor control point of view, but we omit them here since they did not affect the convergence of the algorithm in any systematic way. The random models were divided into two groups of 100 each: passively stable, with all eigenvalues of A being smaller than 1, and passively unstable, with the largest eigenvalue of A being between 1 and 2. The dynamics were restricted so that the last component of xt was 1—to make the random models more similar to the psychophysical models, which always incorporated a constant in the state description. The state, control, and measurement dimensionalities were sampled uniformly between 5 and 20. The random models included all forms of noise allowed by equation 3.1. For each model, we initialized K 1,...,n−1 from equation 3.2 and applied our iterative algorithm. In all cases convergence was very rapid (see Figures 2A and 2B), with the relative change in expected cost decreasing exponentially. The jitter observed at the end of the minimization (see Figure 2A) is due to numerical round-off errors (note the log scale) and continues indefinitely. The exponential convergence regime does not always start from the first iteration (see Figure 2A). Similar behavior was observed for the absolute

Methods for Optimal Sensorimotor Control

1103

change in expected cost (not shown). As one would expect, random models with unstable passive dynamics converged more slowly than passively stable models. Convergence was observed in all cases. To test for the existence of local minima, we focused on five psychophysical, five random stable, and five random unstable models. For each model, the algorithm was initialized 100 times with different randomly chosen sequences K 1,...,n−1 , and run for 100 iterations. For each model, we computed the standard deviation of the expected cost obtained at each iteration and divided by the mean expected cost at that iteration. The results, averaged within each model class, are plotted in Figure 2C. The negligibly small values after convergence indicate that the algorithm always finds the same solution. This was true for every model we studied, despite the fact that the random initialization sometimes produced very large initial costs. We also examined the K and L sequences found at the end of each run, and the differences seemed to be due to round-off errors. Thus, we conjecture that the algorithm always converges to the globally optimal solution. So far we have not been able to prove this analytically and cannot offer a satisfying intuitive explanation at this time. Note that the system can be unstable even for the optimal controller. Formally, that does not affect the derivation, because in a discrete-time finitehorizon system, all numbers remain finite. In practice, the components of xt can exceed the maximum floating-point number whenever the eigenvalues of (A − B L t ) are sufficiently large. In the applications we are interested in (Todorov, 1998; Todorov & Jordan, 2002b), such problems were never encountered. 8 Improving Performance via Adaptive Estimation Although the iterative algorithm given by equations 4.2 and 5.2 is guaranteed to converge, and empirically it appears to converge to the globally optimal solution, performance can still be suboptimal due to the imposed restriction to nonadaptive filters. Here we present simulations aimed at quantifying this suboptimality. Because the potential suboptimality arises from the restriction to nonadaptive filters, it is natural to ask what would happen if that restriction were removed in run time and the optimal adaptive linear filter from equation 5.3 were used instead. Recall that although the control law is optimized under the assumption of a nonadaptive filter, it yields better performance if a different filter, which somehow achieves lower estimation error, is used in run time. Thus, in our first test, we simply replace the nonadaptive filter with equation 5.3 in run time and compute the reduction in expected total cost. The above discussion suggests a possibility for further improvement. The control law is optimal with respect to some sequence of filter gains K 1,...,n−1 . But the adaptive filter applied in run time uses systematically different gains, because it achieves systematically lower estimation error. We can run

1104

E. Todorov

Table 2: Cost Reduction. Model Method

Psychophysical

Random Stable

Random Unstable

Adaptive Estimator Reoptimized Controller

1.9 % 1.7

0% 0

31.4 % 28.3

Notes: Numbers indicate percent reduction in expected total cost, relative to the cost of the solution found by our iterative algorithm. The two improvement methods are described in the text. Each method is applied to 10 models in each model class. For each model and method, expected total cost is computed from 10,000 simulation runs. A value of 0% indicates that with a sample size of 10 models, the improvement was not significantly different from 0% (t-test, p = 0.05 threshold).

our control law in conjunction with the adaptive filter and find the average 1,...,n−1 that are used online. Now, one would think that if we filter gains K 1,...,n−1 , which better reoptimized the control law for the nonadaptive filter K reflects the gains being used online by the adaptive filter, this will further improve performance. This is the second test we apply. As Table 2 shows, neither method improves performance substantially for psychophysical models or random stable models. However, both methods result in substantial improvement for random unstable models. This is not surprising. In the passively stable models, the differences between the expected and actual values of the states and controls are relatively small, and so the optimal nonadaptive filter is not that different from the optimal adaptive filter. The unstable models, on the other hand, are very sensitive to small perturbations and thus follow substantially different state-control trajectories in different simulation runs. So the advantage of adaptive filtering is much greater. Since musculoskeletal plants have stable passive dynamics, we conclude that our algorithm is well suited for approximating the optimal sensorimotor system. It is interesting that control law reoptimization in addition to adaptive filtering is actually worse than adaptive filtering alone—contrary to our intuition. This was the case for every model we studied. Although it is not clear where the problem with the reoptimization method lies, this somewhat unexpected result provides further justification for the restriction we introduced. In particular, it suggests that the control law that is optimal under the best nonadaptive filter may be close to optimal under the best adaptive filter. 9 Discussion We have presented an algorithm for stochastic optimal control and estimation of partially observable linear dynamical systems, subject to quadratic costs and noise processes characteristic of the sensorimotor system (see

Methods for Optimal Sensorimotor Control

1105

equation 3.1). We restricted our attention to controllers that use state estimates obtained by nonadaptive linear filters. The optimal control law for any such filter was shown to be linear, as given by equation 4.2. The optimal nonadaptive linear filter for any linear control law is given by equation 5.2. Iteration of equations 4.2 and 5.2 is guaranteed to converge to a filter and a control law optimal with respect to each other. We found numerically that convergence is exponential, local minima do not to exist, and the effects of assuming nonadaptive filtering are negligible for the control problems of interest. The application of the algorithm was illustrated in the context of reaching movements. The optimal adaptive filter, equation 5.3, as well as the optimal controller for the fully observable case, equation 4.3, were also derived. To facilitate the application of our algorithm in the field of motor control and elsewhere, we have made a Matlab implementation available at www.cogsci.ucsd.edu/∼todorov. While our work was motivated by models of biological movement, the results presented here could be of interest to a wider audience. Problems with multiplicative noise have been studied in the optimal control literature, but most of that work has focused on the fully observable case (Kleinman, 1969; McLane, 1971; Willems & Willems, 1976; Bensoussan, 1992; El Ghaoui, 1995; Beghi & D’Alessandro, 1998; Rami et al., 2001). Our equation 4.3 is consistent with these results. The partially observable case that we addressed (and that is most relevant to models of sensorimotor control) is much more complex, because the independence of estimation and control breaks down in the presence of signal-dependent noise. The work most similar to ours is Pakshin (1978) for discrete-time dynamics and Phillis (1985) for continuoustime dynamics. These authors addressed a closely related problem using a different methodology. Instead of analyzing the closed-loop system directly, the filter and control gains were treated as open-loop controls to a modified deterministic dynamical system, whose cost function matches the expected cost of the original system. With that transformation, it is possible to use Pontryagin’s maximum principle, which is applicable only to deterministic open-loop control, and obtain necessary conditions that the optimal filter and control gains must satisfy. Although our results were obtained independently, we have been able to verify that they are consistent with Pakshin (1978) by removing from our model the internal estimation noise (which to our knowledge has not been studied before); combining equations 4.2 and 5.2; and applying certain algebraic transformations. However, our approach has three important advantages. First, we managed to prove that the optimal control law is linear under a nonadaptive filter, while this linearity had to be assumed before. Second, using the optimal cost-to-go function to derive the optimal filter revealed that adaptive filtering improves performance, even though the control law is optimized for a nonadaptive filter. And most important, our approach yields a coordinatedescent algorithm with guaranteed convergence, as well as appealing numerical properties illustrated in sections 7 and 8. Each of the two steps of our

1106

E. Todorov

coordinate-descent algorithm is computed efficiently in a single pass through time. In contrast, application of Pontryagin’s maximum principle yields a system of coupled difference (Pakshin, 1978) or differential (Phillis, 1985) equations with boundary conditions at the initial and final time, but no algorithm for solving that system. In other words, earlier approaches obscure the key property we uncovered: that half of the problem can be solved efficiently given a solution to the other half. Finally, there may be an efficient way to obtain a control law that achieves better performance under adaptive filtering. Our attempt to do so through reoptimization (see section 8) failed, but another approach is possible. Using the optimal adaptive filter (see equation 5.3) would make E [vt+1 ] a complex function of
xt , ut , and the resulting vt would no longer be in the assumed parametric form (which is why we introduced the restriction to nonadaptive filters). But we could force that complex vt in the desired form by approximating it with a quadratic in
xt , ut . This yields additional terms in equation 4.2. We have pursued this idea in our earlier work (Todorov, 1998); an independent but related method has been developed by Moore, Zhou, and Lim (1999). The problem with such approximations is that convergence guarantees no longer seem possible. While Moore et al. did not illustrate their method with numerical examples, in our work we have found that the resulting algorithm is not always stable. These difficulties convinced us to abandon the earlier idea in favor of the methodology presented here. Nevertheless, approximations that take adaptive filtering into account may yield better control laws under certain conditions and deserve further investigation. Note, however, that the resulting control laws will have to be used in conjunction with an adaptive filter, which is much less efficient in terms of online computation. Acknowledgments Thanks to Weiwei Li for comments on the manuscript. This work was supported by NIH grant R01-NS045915. References Anderson, F., & Pandy, M. (2001). Dynamic optimization of human walking. J Biomech. Eng, 123(5), 381–390. Beghi, A., & D’Alessandro, D. (1998). Discrete-time optimal control with controldependent noise and generalized Riccati difference equations. Automatica, 34, 1031–1034. Bensoussan, A. (1992). Stochastic control of partially observable systems. Cambridge: Cambridge University Press. Bertsekas, D., & Tsitsiklis, J. (1997). Neuro-dynamic programming. Belmont, MA: Athena Scientific.

Methods for Optimal Sensorimotor Control

1107

Burbeck, C., & Yap, Y. (1990). Two mechanisms for localization? Evidence for separation-dependent and separation-independent processing of position information. Vision Research, 30(5), 739–750. Chow, C., & Jacobson, D. (1971). Studies of human locomotion via optimal programming. Math Biosciences, 10, 239–306. Davis, M., & Vinter, R. (1985). Stochastic modelling and control. London: Chapman and Hall. El Ghaoui, L. (1995). State-feedback control of systems of multiplicative noise via linear matrix inequalities. Systems and Control Letters, 24, 223–228. Flash, T., & Hogan, N. (1985). The coordination of arm movements: An experimentally confirmed mathematical model. Journal of Neuroscience, 5(7), 1688– 1703. Harris, C., & Wolpert, D. (1998). Signal-dependent noise determines motor planning. Nature, 394, 780–784. Hatze, H., & Buys, J. (1977). Energy-optimal controls in the mammalian neuromuscular system. Biol. Cybern., 27(1), 9–20. Hoff, B. (1992). A computational description of the organization of human reaching and prehension. Unpublished doctoral dissertation, University of Southern California. Jacobson, D., & Mayne, D. (1970). Differential dynamic programming. New York: Elsevier. Jones, K., Hamilton, A., & Wolpert, D. (2002). Sources of signal-dependent noise during isometric force production. Journal of Neurophysiology, 88, 1533–1544. Kleinman, D. (1969). Optimal stationary control of linear systems with controldependent noise. IEEE Transactions on Automatic Control, AC-14(6), 673–677. Kording, K., & Wolpert, D. (2004). The loss function of sensorimotor learning. Proceedings of the National Academy of Sciences, 101, 9839–9842. Kuo, A. (1995). An optimal control model for analyzing human postural balance. IEEE Transactions on Biomedical Engineering, 42, 87–101. Kushner, H., & Dupuis, P. (2001). Numerical methods for stochastic optimal control problems in continuous time (2nd ed.). New York: Springer. Li, W., & Todorov, E. (2004). Iterative linear-quadratic regulator design for nonlinear biological movement systems. In First International Conference on Informatics in Control, Automation and Robotics, vol. 1, 222–229. N.P.: INSTICC Press. Loeb, G., Levine, W., & He, J. (1990). Understanding sensorimotor feedback through optimal control. Cold Spring Harb. Symp. Quant. Biol., 55, 791–803. McLane, P. (1971). Optimal stochastic control of linear systems with state- and control-dependent disturbances. IEEE Transactions on Automatic Control, AC-16(6), 793–798. Meyer, D., Abrams, R., Kornblum, S., Wright, C., & Smith, J. (1988). Optimality in human motor performance: Ideal control of rapid aimed movements. Psychological Review, 95, 340–370. Moore, J., Zhou, X., & Lim, A. (1999). Discrete time LQG controls with control dependent noise. Systems and Control Letters, 36, 199–206. Nelson, W. (1983). Physical principles for economies of skilled movements. Biological Cybernetics, 46, 135–147. Pakshin, P. (1978). State estimation and control synthesis for discrete linear systems with additive and multiplicative noise. Avtomatika i Telemetrika, 4, 75–85.

1108

E. Todorov

Phillis, Y. (1985). Controller design of systems with multiplicative noise. IEEE Transactions on Automatic Control, AC-30(10), 1017–1019. Rami, M., Chen, X., & Moore, J. (2001). Solvability and asymptotic behavior of generalized Riccati equations arising in indefinite stochastic LQ problems. IEEE Transactions on Automatic Control, 46(3), 428–440. Schmidt, R., Zelaznik, H., Hawkins, B., Frank, J., & Quinn, J. (1979). Motor-output variability: A theory for the accuracy of rapid motor acts. Psychol Rev., 86(5), 415–451. Sutton, G., & Sykes, K. (1967). The variation of hand tremor with force in healthy subjects. Journal of Physiology, 191(3), 699–711. Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press. Todorov, E. (1998). Studies of goal-directed movements. Unpublished doctoral dissertation, Massachusetts Institute of Technology. Todorov, E. (2002). Cosine tuning minimizes motor errors. Neural Computation, 14(6), 1233–1260. Todorov, E. (2004). Optimality principles in sensorimotor control. Nature Neuroscience, 7(9), 907–915. Todorov, E., & Jordan, M. (2002a). A minimal intervention principle for coordinated movement. In S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in neural information processing systems, 15 (pp. 27–34). Cambridge, MA: MIT Press. Todorov, E., & Jordan, M. (2002b). Optimal feedback control as a theory of motor coordination. Nature Neuroscience, 5(11), 1226–1235. Todorov, E., & Li, W. (2004). A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. Manuscript submitted for publication. Uno, Y., Kawato, M., & Suzuki, R. (1989). Formation and control of optimal trajectory in human multijoint arm movement: Minimum torque-change model. Biological Cybernetics, 61, 89–101. Whitaker, D., & Latham, K. (1997). Disentangling the role of spatial scale, separation and eccentricity in Weber’s law for position. Vision Research, 37(5), 515–524. Whittle, P. (1990). Risk-sensitive optimal control. New York: Wiley. Willems, J. L., & Willems, J. C. (1976). Feedback stabilizability for stochastic systems with state and control dependent noise. Automatica, 1976, 277–283. Winter, D. (1990). Biomechanics and motor control of human movement. New York: Wiley.

Received June 21, 2002; accepted October 1, 2004.

LETTER

Communicated by Jochen J. Steil

Universal Approximation Capability of Cascade Correlation for Structures Barbara Hammer [email protected] Institute of Computer Science, Clausthal University of Technology, 38678 Clausthal-Zellerfeld, Germany

Alessio Micheli [email protected] Dipartimento di Informatica, Universit`a di Pisa, Pisa, Italy

Alessandro Sperduti [email protected] Dipartimento di Matematica Pura ed Applicata, Universit`a di Padova, Padova, Italy

Cascade correlation (CC) constitutes a training method for neural networks that determines the weights as well as the neural architecture during training. Various extensions of CC to structured data have been proposed: recurrent cascade correlation (RCC) for sequences, recursive cascade correlation (RecCC) for tree structures with limited fan-out, and contextual recursive cascade correlation (CRecCC) for rooted directed positional acyclic graphs (DPAGs) with limited fan-in and fan-out. We show that these models possess the universal approximation property in the following sense: given a probability measure P on the input set, every measurable function from sequences into a real vector space can be approximated by a sigmoidal RCC up to any desired degree of accuracy up to inputs of arbitrary small probability. Every measurable function from tree structures with limited fan-out into a real vector space can be approximated by a sigmoidal RecCC with multiplicative neurons up to any desired degree of accuracy up to inputs of arbitrary small probability. For sigmoidal CRecCC networks with multiplicative neurons, we show the universal approximation capability for functions on an important subset of all DPAGs with limited fan-in and fan-out for which a specific linear representation yields unique codes. We give one sufficient structural condition for the latter property, which can easily be tested: the enumeration of ingoing and outgoing edges should be compatible. This property can be fulfilled for every DPAG with fan-in and fan-out two via reenumeration of children and parents, and for larger fan-in and fan-out via an expansion of the fan-in and fan-out and reenumeration of children and parents. In addition, the result can be generalized to the case of input-output Neural Computation 17, 1109–1159 (2005)

© 2005 Massachusetts Institute of Technology

1110

B. Hammer, A. Micheli, and A. Sperduti

isomorphic transductions of structures. Thus, CRecCC networks constitute the first neural models for which the universal approximation capability of functions involving fairly general acyclic graph structures is proved.

1 Introduction Pattern recognition methods such as neural networks or support vector machines constitute powerful machine learning tools in a widespread area of applications. Usually they process data in the form of real vectors of fixed and finite dimensionality. Canonical extensions of these basic models to sequential data that occur in language processing, bioinformatics, or time-series prediction, for example, are provided by recurrent networks or statistical counterparts (Bengio & Frasconi, 1996; Kremer, 2001; Sun, 2001). Recently, an increasing interest in the possibility of dealing with more complex nonstandard data has been observed, and several models have been proposed within this topic: specifically designed kernels such as string and tree kernels or kernels derived from statistical models constitute a canonical interface for kernel methods for structures (Haussler, 1999; Jaakkola, Diekhans, & Haussler, 2000; Leslie, Eskin, & Noble, 2002; Lodhi, ShaweTaylor, Christianini, & Watkins, 2000; Watkins, 1999). Alternatively, recurrent neural models can be extended to complex dependencies that occur in tree structures, graphs, or spatial data (Baldi, Brunak, Frasconi, Pollastri, Soda, 1999; Frasconi, Gori, & Sperduti, 1998; Goller & Kuchler, ¨ 1996; Hammer, 2000; Micheli, Sona, & Sperduti, 2000; Pollastri, Baldi, Vullo, & Frasconi, 2002; Sperduti, Majidi, & Starita, 1996; Sperduti & Starita, 1997; Wakuya & Zurada, 2001). These structure processing models have the advantage that complex data such as structured data, tree structures, and graphs can be directly tackled, and thus a possibly time-consuming and usually not information-preserving encoding of the structures by a finite number of real-valued features can be avoided. Successful applications of these models have been reported in such areas as image and document processing, logic, natural language processing, chemistry, DNA processing, homology detection, and protein structure prediction (Baldi et al., 1999; Diligenti, Frasconi, & Gori, 2003; Goller, 1997; Jaakkola et al., 2000; Leslie et al., 2002; Lodhi et al., 2000; de Mauro, Diligenti, Gori, & Maggini, 2003; Pollastri et al., 2002; Sturt, Costa, Lombardo, & Frasconi, 2003; Vullo & Frasconi, 2003). Here, we are interested in a variant of so-called recursive networks, which constitute a straightforward generalization of well-known recurrent networks to tree structures and for which excellent results for large learning tasks have been reported (Frasconi, 2002). Training recurrent neural networks faces severe problems, such as the problem of long-term dependencies, and the design of efficient training algorithms is still a challenging problem of ongoing research (Bengio, Simard,

Universal Approximation Capability

1111

& Frasconi, 1994; Hammer & Steil, 2002). Since the number of potentially different structures increases exponentially with the size of the data, the space is often only sparsely covered by a given training set. In addition, the generalization ability might be fundamentally worse compared to simple pattern recognition tools because the capacity of the models (measured, for example, in terms of the VC dimension) depends on the structures (Hammer, 2001). For recursive models for structures, the long-term dependencies problem is often reduced because structural representations via trees or graphs yield more compact representation of data. However, the other problems remain. Thus, the design of efficient training algorithms that yield sparse models with good generalization ability is a key issue for recursive models for structures. Cascade correlation (CC) has been proposed by Fahlmann and Lebiere (1990) as a particularly efficient training algorithm for feedforward networks that simultaneously determines the architecture of the network and the parameters. Hidden neurons are created and trained consecutively, so as to maximize the correlation of the hidden neuron with the current residual error. Thus, hidden neurons can be used for error correction in consecutive steps, which often leads to very sparse solutions with good generalization ability. CC proved to be particularly appropriate for training problems where standard feedforward network training is difficult, such as the two spirals problem (Fahlmann & Lebiere, 1990). The main difference of CC networks with respect to feedforward networks lies in the particularly efficient constructive training method. Since any feedforward network structure can be embedded into the structure of a CC network and vice versa, the principled representational capabilities of CC networks and standard feedforward networks coincide (Hornik, Stinchcombe, & White, 1989). Like other standard feedforward networks, CC deals only with vectors of fixed dimensionality. Recurrent cascade correlation (RCC) has been proposed as a generalization of CC to sequences of unlimited length (Fahlmann, 1991). Recently, more powerful models for tree structures and acyclic directed graphs have been proposed and successfully applied as prediction models for chemical structures: recursive cascade correlation and contextual recursive cascade correlation, respectively (Bianucci, Micheli, Sperduti, & Starita, 2000; Micheli, 2003; Micheli, Sona, & Sperduti, 2002, in press; Micheli et al., 2000; Sperduti et al., 1996). The efficient training algorithm of CC can be transferred to these latter models, and very good results have been reported. Here, we are interested in the in-principle representational capabilities of these models for different types of structures. RCC extends simple CC by recurrent connections such that input sequences can be processed step by step. The iterative training scheme of hidden neurons is thereby preserved and training is very efficient. However, because of this iterative training scheme, hidden neurons have to be independent of all hidden neurons that are introduced later. This causes the fact that an RCC network, unlike RNNs, is not fully recurrent. Recurrent

1112

B. Hammer, A. Micheli, and A. Sperduti

connections of a hidden neuron are pointing toward the neuron itself and toward hidden neurons that are introduced later, but not toward previous hidden neurons. Because of this fact, RNNs in general cannot be embedded in RCC networks: the two models differ with respect to the network architectures. RNNs constitute universal approximators (Funahashi & Nakamura, 1993), and the question now occurs whether the restricted topology of RCC networks, which makes the particularly efficient iterative training scheme possible, preserves this property. This is not clear because local recurrence instead of full connectivity of neurons might severely limit the representational capabilities. Frasconi and Gori (1996) have shown that the number of functions that can be implemented by a specific locally recurrent network architecture with the Heaviside activation function can be limited regardless of the number of neurons, and thus restrictions apply. The work presented in Giles et al. (1995) explicitly investigates RCC architectures and shows that these networks equipped with the Heaviside function or a monotonically increasing activation function cannot implement all finite automata. Thus, in the long-term limit, RCC networks are strictly weaker than fully connected recurrent networks. However, the universal approximation capability of recurrent networks usually refers to the possibility of approximating every given function on a finite time horizon. This question, which has not yet been answered in the literature, is investigated in this article. We will show that RCC networks are universal approximators with respect to a finite time horizon; they can approximate every function up to inputs of arbitrary small probability arbitrarily well. Hence, RCC networks and recurrent networks do not differ with respect to this notion of approximation capability. Note that RCC networks, unlike RNN networks, are built and trained iteratively, adding one hidden neuron at a time. Thus, the architecture is determined automatically, and only small parts of the network are adapted during a training cycle. RCC networks have been generalized to recursive cascade correlation (RecCC) networks for tree-structured inputs (Bianucci et al., 2000; Sperduti et al., 1996; Micheli, 2003). Recursive networks constitute the analogous extension of recurrent networks to tree structures (Frasconi et al., 1998; Goller & Kuchler, ¨ 1996; Sperduti & Starita, 1997). For the latter, the universal approximation capability has been established in Hammer (2000). However, RecCC networks share the particularly efficient iterative training scheme of RCC and CC networks, and thus recurrence is restricted to recursive connections of hidden neurons to neurons introduced later, but not in the opposite direction also in RecCC networks. Thus, it is not surprising that the same restrictions as for RCC networks apply, and some functions, which can be represented by fully recursive neural networks, cannot be represented by RecCC networks in the long-term limit because of these restrictions (Sperduti, 1997). However, the practically relevant representational capabilities if restricted to a finite horizon, that is, input trees with restricted maximum depth, are not yet answered in the literature. We show

Universal Approximation Capability

1113

in this article that RecCC networks with multiplicative neurons possess the universal approximation property for finite time horizon and thus constitute valuable and fast alternatives to fully connected recursive networks for practical applications. Recurrent and recursive models put a causality assumption on data processing: sequences are processed from one end of the sequences to the other, and trees are processed from the leaves to the root. Thus, the state variables associated with a sequence entry depend on the left context, but not on the right one, and state variables associated with internal nodes in a tree depend on their children, but not the other way around. This causality assumption might not be justified by the data. For time series, the causality assumption implies that events in the future depend on events in the past, but not vice versa, which is reasonable. For spatial data such as sentences or DNA strings, the same causality assumption implies that the value at an interior position of the sequence depends on only the left part of the sequence but not on the right one. This is obviously not true in general. A similar argument can be stated for tree structures and more general graphs such as they occur in logic or chemistry, for example. Therefore, extensions of recursive models for sequences and tree structures have been proposed that also take additional contextual information into account. The model proposed in Baldi et al. (1999) trains two RNNs simultaneously to predict the secondary structure of proteins. The networks are thereby transforming the given sequence in reverse directions, such that the full information is available at each position of the sequence. A similar approach is proposed in Pollastri et al. (2002) for grids, whereby four recursive networks are trained simultaneously in order to integrate all available information for each vertex of a regular two-dimensional lattice. Similar problems for sequences have been tackled by a cascade correlation approach including context in Micheli et al. (2000). Another model with similar ideas has been proposed in Wakuya and Zurada (2001). Note that several of these models simply construct disjoint recursive networks according to different causality assumptions and afterward integrate the full information in a feedforward network. However, all reported experiments show that the accuracy of the models can be improved significantly by this integration of context if spatial data or graph structures are dealt with. The restricted recurrence of cascade correlation offers a particularly simple, elegant, and, as we will see, fundamentally different way to include contextual information: the hidden neurons are consecutively trained and, once they are trained, frozen. Thus, we can introduce recursive as well as contextual connections of these neurons to neurons introduced later without getting cyclic (i.e., ill-defined) dependencies; that is, given a vertex in a tree or graph, the hidden neuron activation for this vertex can have direct access to the state of its children and its parents for each previous hidden neuron. Thereby, the effective iterative training scheme of CC is preserved. Obviously, this possibility of context integration essentially depends on the

1114

B. Hammer, A. Micheli, and A. Sperduti

recurrences being restricted, and it cannot be transferred to fully recursive networks. This generalization of RecCC networks toward context integration has recently been proposed in Micheli et al. (2002, in press). These models are applied to prediction tasks for chemical structures, and they compare favorably to models without context integration. Plots of the principal components of the hidden neurons’ states of the contextual models indicate that the additional contextual information is effectively used by the network. Unlike the approaches noted, contextual information is here directly integrated into the recursive part of the network such that the information can be used in an earlier stage than in Baldi et al. (1999) and Pollastri et al. (2002). Thanks to these characteristics, in the cascade approach, the processing of contextual information is efficiently extended, with respect to the previous approaches, from sequence or grids to more complex structured data such as trees and acyclic graphs. The question now occurs whether the in principle extension to contextual processing and the improved accuracy in experiments can be accompanied by more general theoretical results on the universal approximation capability of the model. A first study of the enlarged capacity of the model can be found in Micheli et al. (in press) and Micheli (2003), where it is shown that in a directed acyclic graph, contextual RecCC (CRecCC) takes into account the context of each vertex, including both successors and predecessors of the vertex (as formally proven in Micheli, Sona, & Sperduti, 2003). Moreover, the integration of context extends the class of functions that can be approximated in the sense that more general structures can be distinguished: contextual models can differentiate between vertices with different parents but the same content and children. Finally, contextual RecCC can implement classes of contextual functions where a desired output for a given vertex may depend on the predecessors of the vertex, up to the whole structure (contextual transductions) (Micheli et al., in press; Micheli, 2003). Of course, contextual RecCC networks can still implement all the functions that are computable by RCC models (Micheli et al., in press; Micheli, 2003). We will show in this article that the proposed notion of context yields universal approximators for rooted directed acyclic graphs with appropriate positioning of children and parents. The argumentation can be transferred to the more general case of input-output (IO) isomorphic transduction of such structures. Hence, context integration enlarges the class of functions that can be approximated to acyclic graph structures. We first formally introduce the various network models and clarify the notation. We then consider the universal approximation capability of cascade correlation networks in detail. We start with the simplest case: the universal approximation property of standard RCC networks. Only two parts of the proof are specific for the case of sequences; the other parts can be transferred to more general cases. We then prove the universal approximation capability of RecCCs, substituting the specific parts of the previous proof. Finally we investigate CRecCCs, which are particularly interesting,

Universal Approximation Capability

1115

since CRecCCs do not constitute universal approximators for all structures, but only a subset. We provide one example for graph structures that cannot be differentiated by these models. We then define a linear encoding of acyclic graph structures and discuss one sufficient property that can easily be tested and ensures that this encoding is unique. This property requires that positioning of children and parents can be done in a compatible way. We show that this can always be achieved by reenumeration and expansion by empty vertices for directed positional acyclic graphs with limited fan-in and fan-out. Structures that fulfill this property are referred to as directed acyclic bipositional graphs (DBAGs). Afterward, we show the universal approximation for contextual RecCC networks on DBAGs. This result can immediately be extended to general graph structures for which the proposed encoding is unique. An informal discussion of this property is given. We conclude with a discussion. 2 Cascade Correlation Models for Structures Standard feedforward neural networks (FNN) consist of neurons connected in an acyclic directed graph. Neuron i computes the function
  xi = σi wi j x j + θi j→i

of the outputs x j of its predecessors, where wi j ∈ R are the weights, θi ∈ R is the bias, and σi : R → R constitutes the activation function of neuron i. The input neurons (i.e., neurons without predecessors) are directly set to external values, and the output of the network can be found at specified output neurons. All other neurons are called hidden neurons. Often neurons are arranged in layers with full connectivity of the neurons of one layer to the consecutive one. Activation functions σi that are of interest in this article include the logistic function, 1 , 1 + exp(−x) the Heaviside function, sgd(x) =

 H(x) =

1 if x ≥ 0 , 0 otherwise

and the identity id(x) = x. More generally, a squashing activation function f : R → R is a monotonic function such that values l f < h f exist with limx→∞ f (x) = h f , limx→−∞ f (x) = l f . A network that uses activation functions from a set F of functions is referred to as an F-FNN. It is well known that FNNs with only one hidden layer with a squashing activation function and linear outputs fulfill a universal approximation property (Hornik et al.,

1116

B. Hammer, A. Micheli, and A. Sperduti

1989). The universal approximation property holds for FNNs, if for a given probability measure P on Rn , any Borel-measurable f : Rn → Rm , and any , δ > 0 we can find an FNN that computes a function g such that P(x ∈ Rn | | f (x) − g(x)| > δ) <  . There exist alternative characterizations of the hidden neurons activation function for which the universal approximation property holds, for example, being analytic (Scarselli & Tsoi, 1998). Often, more powerful neurons computing products of inputs are considered (Alquezar & Sanfeliu, 1995; Forcada & Carrasco, 1995; Hornik et al., 1989). Let us denote the indices of predecessors of neuron i by i 1 , . . . , i j . A multiplicative neuron of degree d ≥ 1 computes a function of the form
xi = σi

d 



 wi j1 ... jk x j1 · . . . · x jk + θi ,

k=1 j1 ,..., jk ∈{i 1 ,...,i j }

where wi j1 ,..., jk ∈ R refers to the weights and θi is the bias as above. We will explicitly indicate if multiplicative neurons are used. Otherwise, the term neuron refers to standard neurons. Standard neural network training first chooses the network architecture and then fits the parameters on the given training set. In contrast, cascade correlation (CC) determines the architecture and the parameters simultaneously. CC starts with a minimum architecture without hidden neurons and iteratively adds hidden neurons unless the approximation accuracy is satisfiable. Each hidden neuron is connected to all inputs and all previous hidden neurons. The weights associated with these connections are trained in such a way as to maximize the correlation between their output and the residual error of the network constructed so far. Afterward, the weights entering the hidden neuron are frozen. All weights to the outputs are retrained according to the given task. If needed, new hidden neurons are iteratively added in the same way. One CC network with two hidden neurons is depicted in Figure 1. Since only one neuron is trained at each stage, this algorithm is very fast. Moreover, it usually provides excellent classification results with a small number of hidden neurons, which, because of the specific training method (Fahlmann & Lebiere, 1990), serve as error-correcting units. This training scheme is used for all further cascade correlation models introduced in this article. Since we are not interested here in the specific training method but in the in-principle representation and approximation capabilities of the architectures, we will not explain the training procedure in more detail. The only point of interest in this article is that the iterative training method assumes that hidden neurons are functionally dependent on previous hidden neurons, while they are functionally independent of hidden neurons introduced later, because of the iterative training scheme.

Universal Approximation Capability

1117

Figure 1: (Left) An example of a cascade correlation network with cascaded

hidden neurons xi added iteratively to the network. (Right) A recursive cascade correlation network where recursive connections, indicated by thick lines marked with r , are used to store contextual information from the first part of a processed sequence. The activation of the neuron of the previous time step is propagated through recurrent connections. Note that recurrent connections point from x1 to x2 but not vice versa.

2.1 Recurrent Models. Obviously, FNNs can deal only with input vectors of finite and fixed size. Recurrent networks (RNNs) constitute one alternative approach, which allows the processing of sequences of unlimited length. Denote by L a set where the labels are taken from, for example, L = Rn for continuous labels or L = {1, . . . , r } for discrete labels. Assume a sequence [l1 , . . . , l T ] of length T over L is given. Denote the set of finite sequences over L by L∗ . A recurrent network recursively processes a given sequence from left to right. Assume that the output of neuron i has to be computed for a given sequence entry lt at position t of the sequence. Assume N neurons are present. We assume for simplicity that the immediate predecessors of neuron i are the neurons from 1 to i − 1.1 The functional dependence of neuron i can then be written as xi (lt ) = τi (lt , x1 (lt ), . . . , xi−1 (lt ), x1 (lt−1 ), . . . , xN (lt−1 )),

1

(2.1)

It can always be achieved via reenumeration that the predecessors are a subset of this set of neurons. Thus, setting some weights to 0, we can represent every FNN in this form.

1118

B. Hammer, A. Micheli, and A. Sperduti

where τi is a function computed by a single neuron,2 and for the initial case t = 1, we set xi (l0 ), interpreted as the output for the empty sequence, to some specified value, for example 0. The computed values depend on the values of all neurons in the previous time step and thus, indirectly, on the left part of the sequence. The final output of the sequence is given by the values xi (l T ) obtained after processing the last entry. Recurrent cascade correlation (RCC) also refers to the activation of neurons in the previous time step. We assume that the neurons are enumerated in the order in which they are created during training—neuron i is added to the RCC network in the ith construction step. Because of the iterative training scheme, hidden neuron i is functionally dependent only on neurons introduced in previous steps. The functional dependence now reads as xi (lt ) = τi (lt , x1 (lt ), . . . , xi−1 (lt ), x1 (lt−1 ), . . . , xi (lt−1 )), where τi is the function computed by a single neuron. Again, xi (l0 ) is set to some specified initial value. Note that unlike RNNs, RCC networks are not fully connected. Recurrent connections that start at neurons j > i to neuron i are dropped, because all weights to neuron i are, once trained, frozen. One example network with two hidden neurons is depicted in Figure 1. It is well known that fully recurrent neural networks constitute universal approximators in the sense that, given appropriate activation functions of the neurons, any measurable function from sequences into a real vector space can be approximated arbitrarily well, that is, the distance of the desired output from the network output is at most δ for inputs of probability at least 1 −  (Hammer, 2000). We will show that an analogous result also holds for RCC networks with restricted recurrence. 2.2 Recursive Models. Both RNNs and RCC networks have been generalized to tree structures in the literature. A tree over L consists of either an empty structure ξt or a vertex v with label l(v) ∈ L and a finite number of subtrees, some of which may be empty. We refer to the ith child of v by chi (v). A tree has limited fan-out k if each vertex has at most k children or subtrees. In this case, we can always assume that exactly k children are present for any nonempty vertex, introducing empty vertices if necessary. The set of finite trees with fan-out k over L is denoted by L∗k . For simplicity, we deal only with fan-out 2 in the following. The generalization to larger 2 Here we assume only dependencies from the previous time step. Dependencies from time steps t − i, i > 1 could be considered and added to equation 2.1 since the definition would still be well defined. Because more information is then available at one time step, benefits for concrete learning problems could result. Obviously, the universal approximation capability of equation 2.1 immediately transfers to the more general definition. Thus, for simplicity, we restrict ourselves to equation 2.1 in the following, and we restrict in a similar way for the case of input trees or structures.

Universal Approximation Capability

1119

fan-out will be immediate. Fan-out 1 gives sequences. Since trees form a recursive structure, recursive processing of each vertex in the tree starting from the leaves up to the root is possible. This leads to the notion of recursive networks (RecNNs) as introduced, for example, in Frasconi et al. (1998), Goller and Kuchler ¨ (1996), and Sperduti and Starita (1997) and recursive cascade correlation (RecCC) as proposed in Bianucci et al. (2000) and Sperduti et al. (1996). As before, for RecNN, we assume direct dependence of neuron i from neurons