REVIEW
Communicated by John Lazzarro
Evolution of Time Coding Systems C. E. Carr University of Maryland, Department of Zoology, College Park MD 20742-4415, U.S.A.
M. A. Friedman Department of Neurobiology, Harvard Medical School, Boston, MA 02115, U.S.A.
The auditory and electrosensory systems contain circuits that are specialized for the encoding and processing of microsecond time differences. Analysis of these circuits in two specialists, weakly electric fish and barn owls, has uncovered common design principles and illuminated some aspects of their evolution.
1 Introduction Time coding systems in the auditory and electrosensory systems share similar physiological and morphological features (Carr, 1986). They also implement similar algorithms for the encoding of temporal information, despite their different neural substrates (Konishi, 1991; Amagai, Friedman, & Hopkins, 1998). It appears that the constraints of signal processing might have led to the parallel evolution of similar morphological and physiological features. For example, a comparison of nonhomologous (i.e., not derived from a similar structure in a common ancestor) neural circuits, which perform a similar behavior in the unrelated African and South American electric fish, has revealed that they employ identical computational algorithms through distinctly different neural implementations (Kawasaki, 1996, 1997). Thus, identical behavioral output may be mediated by different neural circuits. Comparisons of homologous circuits (derived from a common ancestor) in chickens and barn owls, however, provide a different view of the evolution of the nervous system. The chicken is a basal (primitive) land bird, and its circuit for the detection of interaural time differences can be transformed into the more specialized barn owl circuit through small modifications in the map of interaural time differences. These changes occur late in development, against a background of conserved features, and create a unique adult circuit that has increased greatly increased sensitivity to temporal cues. Neural Computation 11, 1–20 (1999)
c 1999 Massachusetts Institute of Technology °
2
C. E. Carr and M. A. Friedman
2 Time-Coding Systems Employ Similar Algorithms Both the electrosensory and auditory systems employ similar algorithms for the encoding of temporal information (Carr, 1986; Konishi, 1991; Kawasaki, 1996; Amagai et al., 1998). In both sensory systems, timing information is coded by phase-locked spikes and processed in a dedicated pathway in parallel with other stimulus variables. The elements of time-coding circuits have morphological and physiological features suited to their function. These specializations of the electrosensory and auditory systems are described below, accompanied by a discussion of their shared features and constraints (see Table 1). 2.1 Time Coding in the Electrosensory System. Weakly electric fish produce electric organ discharges that form an electric field around the fish. The electric organ discharge is detected by sensory cells termed electroreceptors. There are two broad classes of electroreceptors that are sensitive to phase or amplitude (Scheich, Bullock, & Hamstra, 1973; Szabo, 1965). Nerve fibers that innervate the phase-coding type of electroreceptors fire one spike on each cycle of the stimulus, phase locked with little jitter to the zero crossing of the stimulus. This knowledge of the timing of the electrical stimulus is essential for both electrolocation and communication. The unrelated African mormyriform and South American gymnotiform electric fish appear to have independently converged on identical algorithms for encoding the timing of the stimulus. In both groups, primary afferents convey phase-locked spikes to the lateral line lobe of the medulla, where phase and amplitude coding primary afferents terminate on different cell types (Bell, Zakon, & Finger, 1989; Maler, Sas, & Rogers, 1981). Thus, the segregation of phase and amplitude receptors in the skin is reinforced by the central connections formed in the medulla. The separation of phase and amplitude information into two parallel channels is a common feature of all time-coding systems. The two channels not only have separate connections but also distinct morphology. In gymnotiform fish, the phase-coding afferents terminate on spherical cells in the medulla, which in turn relay timing information directly to giant cells in the midbrain electrosensory torus. The accuracy of phase coding improves with the progression from receptors to primary afferents to spherical cells to giant cells in the midbrain torus (Carr, Heiligenberg, & Rose, 1986). The jitter of these spikes (the standard deviation of the response time to the stimulus) decreased threefold with the progression from medulla to midbrain. The basis for this improvement of accuracy may lie in the convergence from afferents to higher-level neurons. The accuracy of even the best single neurons in these first stations of the time-coding pathway is about 10 µsec and so does not match that of the submicrosecond behavioral sensitivity (Rose & Heiligenberg, 1985; Carr, Heiligenberg, & Rose, 1986). Since the ability to detect the small phase differences diminished when smaller
Barn owls
Chickens
Large (end-bulb) synapses, large cells thick axons Large (end-bulb) synapses, large cells, few dendrites, thick axons Large (end-bulb) synapses, large cells, few or no dendrites thick axons
Electronic synapses phase-locked spikes up to 10 kHz
No dendrites, thick axons, heavy myelination
Mammals
Phase-locked spikes, electrotonic synapses
Large cells, thick axons, heavy myelination
African wave-type electric fish (Gymnarchidae) African pulse-type electric fish (Mormyridae) Phase-locked spikes up to 2–3 kHz, short time constants Specialized glutamate receptors, phase-locked spikes up to 2–3 kHz, short time constants Specialized glutamate receptors, phase-locked spikes up to 8 kHz
Phase-locked spikes electrotonic synapses
Large cells, few or no dendrites, large synapses
South American wave-type electric fish (Gymnotidae)
Physiological Specializations for Phase Coding
Morphological Specializations for Phase Locking
Species
Dual delay line structure
Single delay line structure
Single delay line structure
Single delay line, anticoincidence detector (blanking)
Adaptive
Probably single delay line
Time Comparison Circuit Structure
Place code, multiple maps of ITD in each tonotopic band
Superimposed on somatotopic map, allows for each part to be compared with rest of body Inputs: cross-correlation structure; output organization unknown Inputs: cross-correlation structure; outputs: no apparent organization Place code, single map of ITD in each tonotopic band Place code, single map of ITD in each tonotopic band
Organization of Time Comparison Nucleus (Mapping)
Table 1: Similarities and Differences in Time Coding Between the Auditory and Electrosensory Senses.
Evolution of Time Coding Systems 3
4
C. E. Carr and M. A. Friedman
numbers of receptors were stimulated (Rose & Heiligenberg, 1985), this hyperacuity must result from the convergence within the central nervous system of parallel phase-coding channels from sufficiently large areas of the body surface. The two families of African mormyriform electric fish also segregate amplitude and phase information. The most striking parallels with the South American gymnotoids are in the mormyriform family Gymnarchidae. Gymnarchus niloticus is the only African electric fish that generates a continuous sine-wave-like electric discharge (Lissman, 1958). Gymnarchus has one type of electroreceptor that is specialized for encoding the phase of the electric field, while another type is specialized for encoding the amplitude (Bullock, Behrend, & Heiligenberg, 1975). The two types project to distinct areas within the hindbrain, where fine temporal and amplitude discriminations take place separately (Kawasaki & Guo, 1996). The temporal jitter of phasecoding afferents is roughly 5 µs, but some fish are behaviorally capable of discriminations as fine as 0.1 µs (Guo & Kawasaki, 1997). This temporal hyperacuity, as in the gymnotoid fish, presumably arises by comparing many receptors over the surface of the body. Interestingly, it is not impaired by adding 60 µs jitter to the signal, probably because all the receptors are modulated together (Guo & Kawasaki, 1997). Both the electric discharges and time-coding pathways of the other mormyriform family are unlike those of Gymnarchus. All the mormyridae produce pulse-type electric discharges of short duration, ranging from less than 100 microseconds up to over 20 milliseconds in different species, discharged at irregular intervals (Hopkins, 1986a). Mormyrid fish can discriminate electric organ discharge (EODs) using time-domain cues (Hopkins & Bass, 1981). The resolution of the system is at least as fine as the submillisecond range. Electric communication signals from other fish are detected by the class of electroreceptors called knollenorgans. The receptors respond to an outside positive voltage step with a single action potential that can phase-lock up to 10 kHz (Hopkins, 1986b). The temporal information analyzed by the knollenorgan system remains segregated from the rest of the electrosensory system, in their hindbrain targets in the nucleus of the electrosensory lateral line lobe (Bell & Grant, 1989) and in three specialized nuclei in the electrosensory torus (Enger, Libouban, & Szabo, 1976; Haugedé-Carré, 1979), where temporal information appears to be analyzed further (Amagai, 1998; Friedman & Hopkins, 1998). Interestingly, the differential phase-sensitive cells in Gymnarchus’s electrosensory lateral line lobe (ELL) respond adaptively to phase differences between different parts of the body (Kawasaki & Guo, 1996). That is, they respond best to small-phase fluctuations (e.g., 20 µsec) about a mean-phase difference. However, if the mean-phase difference changes (even by as much as 200 µsec), the cell may be briefly excited or depressed depending on the direction of change, after which it regains its sensitivity to the small-phase fluctuations. Although the exact mechanism for this remarkable response
Evolution of Time Coding Systems
5
is not yet known, it probably depends on feedback and complex synaptic properties, and not a simple delay-line/coincidence detector. 2.2 Time Coding in the Auditory System. Although the auditory system differs from the electrosensory systems described above in that it analyzes sound waves rather than electric fields, auditory stimuli are encoded in similar ways, and precise temporal information has direct behavioral relevance. Sound coming from one side of the body reaches one ear before the other, and the auditory system uses these time differences to localize the sound source. The auditory system actually encodes the phase of the auditory signal and then uses interaural phase differences to compute sound location (Heffner & Heffner, 1992; see Fay, 1988). The barn owl’s acute ability to detect small-phase or time differences enables it to catch mice in total darkness on the basis of auditory cues alone (Payne, 1971; Konishi, 1973). As in the electric sense, discrimination of small time differences requires accurate transduction and processing of the original stimulus. Time coding arises in the periphery, and it is preserved and improved in the central nervous system (CNS). Auditory nerve fibers phase-lock to the waveform of the acoustic stimulus (Kiang, Watanabe, Thomas, & Clark, 1965). Spikes do not necessarily occur in every tonal cycle (unlike electric fish), and the discharge pattern of a cochlear nerve fiber can encode the phase of a tone with a frequency above 1000 Hz with an average discharge rate of a few hundred spikes per second. Changes in sound level (loudness) are encoded by changes in spike rate. Thus, there is no predisposition toward coding for either loudness or phase in the periphery. Despite the lack of the electric fish’s specialized phase and amplitude receptors, the same parallel processing of phase and amplitude information that characterizes the electrosensory system is also found in the auditory system. Segregation into phase- and sound-level pathways begins with differences in auditory nerve terminals. In the bird, auditory nerve afferents divide into two. One branch ramifies in the dendritic field of the cochlear nucleus angularis, which codes for changes in sound level, and the other branch terminates in the cochlear nucleus magnocellularis and codes for phase (Takahashi, Moiseff, & Konishi, 1984). The synapse in the nucleus magnocellularis takes the form of a specialized end-bulb terminal (Brawer & Morest, 1974; Ryugo & Fekete, 1982). This synapse conveys the phase-locked discharge of the auditory nerve fibers to their postsynaptic targets in the nucleus magnocellularis. Thus, the synaptic specializations in the auditory nerve accomplish the same goal as the receptor specialization in electric fish. The end-bulb is a secure and effective connection; physiological measures show that phase locking is the same in the neurons of the nucleus magnocellularis as in the eighth nerve, while it is lost in the projection to the amplitude coding nucleus angularis (Sullivan & Konishi, 1984). The CNS uses phase-locked spikes to encode the timing of the stimulus.
6
C. E. Carr and M. A. Friedman
Phase information is preserved and improved, and interaural time differences are detected, in a circuit composed of the auditory nerve, the cochlear nucleus magnocellularis, and the nucleus laminaris (see Figure 1). Many of the features of this circuit may represent specializations for the encoding of timing information. Both avian and mammalian time coding cells are well suited to preserve the temporal firing pattern of auditory nerve inputs; they fire only one or two spikes in response to electrical stimulation of the auditory nerve, and have nonlinear current-voltage relationships around the resting potential (see Oertel, 1997 for review). The effects of excitation are brief and do not summate in time (Wu & Oertel, 1984; Oertel, 1985). Similar physiological responses characterize phase-coding neurons in guinea pig ventral cochlear nucleus (Manis & Marx, 1991) and magnocellular neurons in chickens (Reyes, Rubel, & Spain, 1994; Zhang & Trussell, 1994). Rapidly activating and slowly inactivating potassium current(s) appears to underlie the rapid repolarization and ability of time-coding neurons to transmit well-timed events (Reyes et al., 1994; Brew & Forsythe, 1995). 2.3 Improving Temporal Precision. Behavioral evidence makes it clear that phase-coding systems can extract precise temporal information despite variability (jitter) in the phase locking of neural spikes to the electrical or auditory stimulus. Extensive convergence is often invoked as a mechanism for reducing temporal jitter (Carr et al., 1986; Kawasaki, Rose, & Heiligenberg, 1988; Heiligenberg, 1989). In the simplest model, the averaging of n√ presynaptic units should reduce the jitter in the postsynaptic neuron by 1/ n. As has been pointed out, however, an important requirement for this convergence to be effective is that the synapse and the spike-generating mechanism in the postsynaptic neuron not themselves contribute significant additional jitter (Carr & Amagai, 1996). This requirement of temporal accuracy at the synapse, although frequently assumed, evidently has played an important role in the evolution of pathways that process precise temporal information. Improvement in intrinsic accuracy of neurons can be achieved through anatomical and physiological specializations of both presynaptic and postsynaptic structures that maximize the signal while minimizing the noise. One general strategy is to make everything large. Larger somata and axons are less vulnerable to noise caused by stray currents since their low input resistance and large current-generating ability would keep the influence of voltage fluctuations to a minimum. Many of the known time-coding pathways include large cells: the spherical cells of the gymnotoid ELL (Maler et al., 1981), the giant cells in Gymnarchus (Kawasaki & Guo, 1996), the cells of the mormyrid nucleus of the ELL (Bell & Russell, 1978), nucleus magnocellularis in birds (Jhaveri & Morest, 1982), or the bushy cells of the anteroventral cochlear nucleus in mammals (Rhode, Oertel, & Smith, 1983; Wu & Oertel, 1984). Enlarged size has to be accompanied by concomitant increase in the synaptic current. Further, the currents also should have a fast rise time
Evolution of Time Coding Systems
7
to minimize the influence of ambient voltage fluctuations on the timing of spikes. One solution is to have large terminals that partially engulf the postsynaptic cell, presumably translating into massive release of neurotransmitter without depletion or large injections of current through gap junctions (Zhang & Trussell, 1994; Trussell, 1997). These occur as the end-bulbs of Held in birds (Brawer & Morest, 1974) and club endings in electric fish at numerous points in the time-coding pathway: Gymnotoid electrosensory lateral line lobes (Maler et al., 1981), Gymnotoid torus semicircularis (Carr, Maler, & Taylor, 1986), Mormyrid nucleus of the electrosensory lateral line lobe (Bell & Russell, 1978; Szabo, Ravaille, Libouban, & Enger, 1983), and Mormyrid nucleus exterolateralis anterior (Mugnaini & Maler, 1987). Fast rise times can be enhanced by reducing the electrotonic distance between the synapse and the site of integration, minimizing the attenuation of synaptic current. This occurs in time-coding electric fish neurons and in the cells of the nucleus magnocellularis and the nucleus laminaris in birds (Kawasaki & Guo, 1996; Amagai et al., 1998; Bell & Szabo, 1986; Jhaveri & Morest, 1982; Smith & Rubel, 1979; Carr & Boudreau, 1993). Glutamate receptor splice variants are another major adapatation for large, brief synaptic currents (Trussell, 1997). The chicken nucleus magnocellularis neurons contain glutamate receptors with unusually fast kinetics and large conductances, characteristics that are well suited for use in time-coding pathways (Raman & Trussell, 1992; Zhang & Trussell, 1994; Levin, Schneider, Kubke, Wenthold, & Carr, 1997; Ravindranathan, Parks, & Rao, 1996). A less-wellunderstood physiological specialization common to neural pathways that process temporal information is the presence of high concentrations of the calcium-binding proteins calretinin and calbindin. These two proteins appear mainly in cell types that show high degrees of phase locking, in gymnotoids (Maler, Jande, & Lawson, 1984), mormyrids (Friedman & Kawasaki, 1997), and the auditory brain stem (Takahashi, Carr, Brecha, & Konishi, 1987; Parks et al., 1997). A clue to the role(s) of calretinin and calbindin may lie in the high Ca2+ permeability of the fast glutamate receptors (Otis, Raman, & Trussell, 1995). 3 Similar Algorithms But Different Neural Circuits: The Evolution of Temporal Coding in Weakly Electric Fish Behavioral experiments have shown that electric fish are capable of great accuracy in detecting phase differences between different parts of the body surface. Since African and South American electric fish have independently evolved electrosensory and electromotor systems, comparison of how each fish detects phase differences has revealed which components of the neural circuit are important for encoding and detecting phase differences. Pulse- and wave-type electric organ discharges have evolved independently within both African and South American groups. Close examination of phase comparison circuits and behavior in both the African wave-type
8
C. E. Carr and M. A. Friedman
electric fish, Gymnarchus, and South American wave-type fish, Eigenmannia, has shown that these fish use identical sets of complex computational rules for detecting and evaluating phase difference information (Kawasaki, 1993; Kawasaki & Guo, 1996). Reflecting their independent evolution, however, neuronal implementation of the computational steps appears to take different forms. One of the essential computational steps, phase comparison, is performed in the hindbrain in Gymnarchus and in the midbrain in Eigenmannia (Carr, 1986; Kawasaki & Guo, 1996; Kawasaki, 1996, 1997). Interestingly enough, the pulse-type mormyrid fish have a similar neuronal organization to the unrelated Eigenmannia, in that their phase comparison circuit is in the midbrain. Thus, it appears that phase comparison circuits have evolved at least three times in electric fish. When Eigenmannia is presented with sinusoidally varying electrical fields on two parts of its body, it can distinguish phase differences between these signals smaller than 1 µs. The circuit for detection of these phase differences is constructed of phase-coding afferents from the medulla that project to two cell types of the midbrain torus, synapsing on giant cell bodies and on the dendrites of the small cells (see Figure 2). These afferents form local
Evolution of Time Coding Systems
9
connections that encode the phase of the electric organ discharge from one part of body surface. The giant cell’s axons form horizontal connections that distribute this local phase information to small cells throughout the lamina, so that timing information from one part of the body surface may be compared with any other part. Small cells compare information from one patch of the body surface, through afferent input onto their dendrites, with phase information from any other part of the body surface, through the giant cell input to their cell bodies (Carr, Maler, & Taylor, 1986). Smallcell responses encode either phase advance or phase delay (Heiligenberg & Rose, 1985). The small cell circuit allows the fish to perform all possible comparisons between different parts of the body surface, as required for correct performance of the jamming avoidance response (JAR).
Figure 1: Facing page. Jeffress model and schematic of brain stem auditory circuits for detection of interaural time differences in the barn owl and the chicken. (A) In the barn owl, axons from the ipsilateral cochlear nucleus magnocellularis (IPSI NM) divide and enter the nucleus laminaris at several points along the dorsal surface. These axons act as delay lines within laminaris, interdigitating with inputs from the contralateral cochlear nucleus magnocellularis (CONTRA NM). In the center, the owl circuit has been modified to show the principles of the Jeffress model. Binaural coincidence detectors A–E fire maximally when inputs from the two sides arrive simultaneously. This can occur only when the interaural phase differences are compensated for by an equal and opposite delay. For example, neuron A fires maximally when sound reaches the contralateral ear first and is delayed by the long path from the contralateral ear so as to arrive simultaneously with the input from the ispislateral ear. Thus, this array forms a map of interaural time difference in the dorsoventral dimension of the nucleus. In the owl, sound from the front is mapped toward the ventral surface of the nucleus, and each nucleus contains place maps of the contralateral and part of the ipsilateral hemifield. (B) In the chicken, laminaris cells receive input from the IPSI NM onto their dorsal dendrites and input from the CONTRA NM onto their ventral dendrites. The ipsilateral inputs arrive simultaneously along the mediolateral extent of the nucleus laminaris, while the contralateral axons form delay lines, giving off collateral branches along the ventral surface of the nucleus. These delays are hypothesized to form a place map along the mediolateral axis of the nucleus. Delays from the ipsilateral and contralateral sides are approximately equal at the medial portion of the nucleus and could map sound from the front of the bird. Since contralateral delays are longer than ispilateral delays in the lateral portion of the nucleus, this region could map sounds that reached the contralateral ear first. (Modified from Carr, 1993.)
10
C. E. Carr and M. A. Friedman
Figure 2: Schematic circuit in the gymnotid electric fish midbrain for computation of phase differences between signals on any two parts of the body surface. Phase-coding electroreceptors converge on spherical cells in the medulla, which in turn converge in a topographic projection onto giant cell bodies and small cell dendrites in the torus. Giant cells relay the phase-locked signal all over the torus, with their terminals synapsing on the cell bodies of small cells. Small cells are therefore able to compare phase information from different parts of the body surface. (Modified from Carr, 1993)
The neural circuits and computational steps used in the Gymnarchus JAR are almost identical to that of Eigenmannia except that the phase computations take place at the level of the medulla in the electrosensory lateral line lobe (see Figure 3). Just as in the midbrain of Eigenmannia, the afferents carrying phase information terminate on giant cells and smaller differentialphase-sensitive cells of ELL, though the latter connection has not yet been shown to be direct (Kawasaki & Guo, 1996). Like Eigenmannia, the giant cells project bilaterally and branch extensively in the same regions as the differential-phase-sensitive cells. The giant cells in Gymnarchus ELL therefore serve the same function as the giant cell of the torus of Eigenmannia, distributing local phase information to all the phase-computation neurons. The temporal analysis circuit in pulse mormyriformes exhibits the major features of the other time-coding systems. Organizationally, it shares many of the features exhibited by Eigenmannia. The knollenorgan affer-
Evolution of Time Coding Systems
11
Figure 3: Schematic circuit for computation of phase differences between signals on any two parts of the body surface from the wave-type mormyriform electric fish, Gymnarchus niloticus. Phase-coding electroreceptors project to both giant cells and the inner cell layer (ICL) of the electrosensory lateral line lobe. The giant cells also project bilaterally to the ICL. Although the details of the synaptic interactions in the ICL are unknown, the ICL cells are phase sensitive. Their phase sensitivity presumably results from some form of phase comparison. (From Kawasaki & Guo, 1996)
ents terminate on the large cells of the nucleus of ELL (NELL), which project bilaterally to nucleus exterolateralis anterior (ELa) in the torus, where they synapse immediately on large GABAergic cells and then, after a long, winding axonal delay, throughout the ELa on small output cells (see Figure 4) (Mugnaini & Maler, 1987; Friedman & Hopkins, 1998). The large cells, by contrast, project relatively directly to the small cells, so a time delay arises between the arrival of the indirect, inhibitory input and the delayed, excitatory input. Because the large cells are probably inhibitory, the computational mechanism is unlikely to be simple coincidence detection, but rather a kind of blanking detection, where pulses longer than a given duration are detected. The lengths of the delay lines from the NELL appear to be somewhat variable, such that different small cells may be sensitive to different durations (Friedman & Hopkins, 1998). Indeed, cells in the nucleus postsynaptic to the ELa respond to pulses longer than some threshold duration, and different cells have different thresholds (Amagai, 1998).
12
C. E. Carr and M. A. Friedman
Figure 4: Schematic circuit in the pulse mormyriform electric fish midbrain for analysis of EOD waveforms. Phase-coding Knollenorgan inputs project to the cells of the nucleus of the electrosensory lateral line lobe (NELL), which in turn project bilaterally to the nucleus extrolateralis anterior (ELa) of the torus and terminate on small-output cells and large GABAergic cells. The NELL axon winds extensively through the ELa after terminating on the large cells and before terminating on the small cells, and is modeled here forming the delay lines. The small cells do not appear to be classical coincidence detectors, because the large cell input is inhibitory. However they probably make the initial fine temporal discrimination, which is continued in the adjacent nucleus exterolateralis posterior (Elp). (From Amagai et al., 1998; Friedman & Hopkins, 1998.)
4 Transformation of Plesiomorphic to Derived Circuits: Detection of Interaural Time Differences in Chickens and Barn Owls The barn owl is capable of great accuracy in detecting time differences, and its auditory system is hypertrophied in comparison to birds like the chicken whose auditory systems are less specialized. The development of the owl and chicken auditory systems has been compared in order to determine what ontogenetic (developmental) changes underlie brain hypertrophy. Like other animals, birds use interaural time differences to determine the azimuthal location of a sound. The circuits in the auditory brain stem that detect interaural phase differences conform to the requirements of the Jeffress model (1948). The circuits are composed of two elements, delay lines and coincidence detectors (see Figure 1). The delay lines are created by
Evolution of Time Coding Systems
13
cochlear nucleus axons of varying lengths, and the coincidence detectors are neurons in the nucleus laminaris (birds) or the medial superior olive (mammals) that respond maximally when they receive simultaneous inputs, that is, when the interaural time difference is exactly compensated for by the axonal delay introduced by the inputs. The model not only explains how time differences may be measured but also how they may be encoded. The circuit contains an array of neurons, and because of its position in the array, each neuron responds best to sound coming from a particular direction. Thus, the anatomical place of the neuron encodes the location of that sound (see Figure 1). These neurons compute time difference and a new variable, and transform the time code (phase-locked spikes) into a place (rate) code. The selectivity of all higher-order auditory neurons to time difference derives from the “labeled-line” output of the place map (Konishi, 1986). Place-mapped delay-line circuits have been described or inferred for dogs, chickens, barn owls, and cats (Goldberg & Brown, 1969; Rubel & Parks, 1975; Young & Rubel, 1983; Sullivan & Konishi, 1986; Carr & Konishi, 1990; Yin & Chan, 1990; Smith, Joris, & Yin, 1993; Overholt, Rubel, & Hyson, 1992; Joseph & Hyson, 1993). The details of delay-line circuit organization vary among species. In the plesiomorphic pattern in the chicken, the nucleus laminaris is composed of a monolayer of bipolar neurons, which receive input from ipsi- and contralateral cochlear nucleus onto their dorsal and ventral dendrites, respectively (Rubel & Parks, 1975; Young & Rubel, 1983; Agmon-Snir, Carr, & Rinzel, 1998). These dendrites increase in length with decreasing best frequency. Only the projection from the contralateral cochlear nucleus acts as a delay line, while inputs from the ipsilateral cochlear nucleus arrive simultaneously at all neurons (Overholt et al., 1992). This pattern of inputs creates a single map of interaural time difference (ITD) in any tonotopic band in the mediolateral dimension of the nucleus laminaris (Young & Rubel, 1983). In the barn owl, magnocellular axons from both cochlear nuclei act as delay lines (Carr & Konishi, 1988, 1990). They convey the phase of the auditory stimulus to the nucleus laminaris such that axons from the ipsilateral nucleus magnocellularis enter the nucleus laminaris from the dorsal side, while axons from the contralateral nucleus magnocellularis enter from the ventral side. Recordings from these interdigitating ipsilateral and contralateral axons show regular changes in delay with depth in the nucleus laminaris (Carr & Konishi, 1990). Thus, these afferents interdigitate to innervate dorsoventral arrays of neurons in nucleus laminaris in a sequential fashion and produce multiple representations of ITD within the nucleus (see Figure 1B). The barn owl nucleus laminaris is not only transformed from the plesiomorphic monolayer structure, but is also hypertrophied. The adult chicken laminaris is a single lamina made up of about 1000 bipolar neurons, while in the owl it is a 1 mm thick neuropil with about 10,000 neurons (Massoglia, 1997). Note that this hypertrophy does not stem from the owl’s larger size; the crow has about 2540 laminaris neurons (Winter & Schwartzkopff,
14
C. E. Carr and M. A. Friedman
1961). The hypertrophy of the owl’s nucleus laminaris appears to be due to increased birth of neurons during development rather than to prevention of cell death (Massoglia, 1997). Similar findings characterize another example of brain hypertrophy, the development of the mammalian neocortex (Finlay & Darlington, 1995). Comparisons of homologous circuits that detect interaural time differences in chickens and barn owls have shown how a plesiomorphic circuit in a basal land bird (the chicken) may be transformed into a derived circuit in an advanced land bird (barn owl). A few small changes appear against a backdrop of conserved features to create a different phenotype and a large change in behavioral acuity. The barn owl has a wider frequency range than the chicken (0.1–10 kHz versus 0.1–5 kHz), and part of the increase in the size of the nucleus laminaris is taken up by mapping the wider range. Nevertheless, the owl has many more cells per octave than the chicken, and the increase in auditory brain stem cell numbers may in part account for the owl’s improved acuity. 5 Comparison of Auditory and Electrosensory Systems The orderly projection from the nucleus magnocellularis to the nucleus laminaris in the avian auditory hindbrain contrasts starkly with the widespread, almost random axonal arborizations in the areas that perform temporal analysis in the electric fish. These differences in anatomy appear to reflect differences in the computational algorithm (see Table 1). In the avian hindbrain, comparisons in timing are made within isofrequency bands between the left and right ears, in what amounts to a cross-correlation between the two ears (Carr & Konishi, 1990; Keller & Takahashi, 1996). The relative simplicity of this calculation lends itself well to the generation of an orderly computational map (Knudsen, du Lac, & Esterly, 1987). In electric fish, by contrast, under the computational models for the JAR and pulse duration discrimination, the phase at each receptor must be compared against many other receptors in disparate regions of the body. Thus, the giant cells in Gymnarchus (Kawasaki & Guo, 1996), spherical cells in Eigenmannia (Carr, Maler, & Taylor, 1986), and NELL cells in the pulse mormyrids (Friedman & Hopkins, 1998) distribute temporal information widely through their respective targets. For the fish, this would be equivalent to making a cross-correlation between all receptors on the body surface, which is too complicated to represent in an orderly map. Analysis of the encoding and processing of temporal information in electric fish and barn owls has uncovered some common organizational principles (see Table 1). These time-coding systems implement similar algorithms for the encoding and processing of temporal information (Konishi, 1991). Despite different neural substrates, CNS time channels share numerous morphological and physiological adaptations to improve the time coding of the signal. A dichotomy exists between coding for the timing or
Evolution of Time Coding Systems
15
phase of the signal and coding for its intensity. This appears to be a common solution to the constraints of signal analysis. In electric fish, the dichotomy between phase and amplitude coding begins at the receptor level, while in the auditory system, the separation into phase and amplitude coding is derived within the CNS. In the unrelated African and South American electric fish, the comparison of nonhomologous neural circuits that perform a similar behavior has revealed that they employ essentially similar computational algorithms, implemented through distinctly different neural circuits (Carr, Maler, & Taylor, 1986; Kawasaki, 1996; Amagai, 1998; Friedman & Hopkins, 1998). The similar design features and disparate evolutionary origins of the time-coding systems of the African and South American electric fish not only provide another example of convergent evolution, but also identify general features of temporal coding systems. Specialists such as the barn owl and the electric fish display brain development correlated with their behavioral abilities. The barn owl is capable of great accuracy in detecting time differences, and its auditory system is greatly hypertrophied in comparison to other less specialized birds such as the chicken. The development of the owl and chicken auditory systems has been compared in order to determine what ontogenetic changes underlie brain hypertrophy. Comparisons of homologous circuits that detect interaural time differences in chickens and barn owls have shown how a plesiomorphic circuit in a basal land bird (the chicken) may be transformed into a derived circuit in an advanced land bird (barn owl). These comparisons may permit identification of the regulatory changes that occur with the modification of existing neural structures.
Acknowledgments We gratefully acknowledge helpful discussions with Satoshi Amagai on the subject of jitter and with Masashi Kawasaki on temporal coding in Gymnarchus. Carl Hopkins supported M.A.F.’s work on the mormyrid time-coding pathway, and provided thoughtful insights. This research was supported by the National Institutes of Health (DCD 00436) and by the Sloan Foundation.
References Agmon-Snir, H., Carr, C. E. and Rinzel, J. (1998). A case study for dendritic function: Improving the performance of auditory coincidence detectors. Nature, 393, 268–272. Amagai, S. (1998). Time-coding in the midbrain of mormyrid electric fish II: Stimulus selectivity in the nucleus exterolateralis pars posterior. J. Comp. Physiol. A, 182, 131–143.
16
C. E. Carr and M. A. Friedman
Amagai, S., Friedman M. A., & Hopkins, C. D. (1998). Time-coding in the midbrain of mormyrid electric fish I: Physiology and anatomy of cells in the nucleus exterolateralis pars anterior. J. Comp. Physiol. A, 182,115–130. Bell, C. C., & Grant, K. (1989). Corollary discharge inhibition and preservation of temporal information in a sensory nucleus of mormyrid electric fish. J. Neurosci., 9, 1029–1044. Bell, C. C., & Russell, C. J. (1978). Termination of electroreceptor and mechanical lateral line afferents in the mormyrid acousticolateral area. J. Comp. Neurol., 182, 367–382. Bell, C. C, & Szabo, T. (1986). Electroreception in mormyrid fish: Central anatomy. In T. H. Bullock & W. Heiligenberg (Eds.) Electroreception (pp. 375– 421). New York: Wiley. Bell, C. C., Zakon, H., & Finger, T. E. (1989). Mormyromast electroreceptor organs and their afferent fibers in mormyrid fish: I. Morphology. J. Comp. Neurol., 286, 391–409. Brawer, J. R., & Morest, D. K. (1974). Relations between auditory nerve endings and cell types in the cat’s anteroventral cochlear nucleus seen with Golgi method and Nomarski optics. J. Comp. Neurol., 160, 491–506. Brew, H. M., & Forsythe, I. D. (1995). Two voltage-dependent K+ conductances with complementary functions in postsynaptic integration at a central auditory synapse. J. Neurosci., 15(12), 8011–8022. Bullock, T. H., Behrend, K., & Heiligenberg, W. (1975). Comparison of the jamming avoidance responses in gymnotoid and gymnarchid electric fish: A case of convergent evolution of behavior and its sensory basis. J. Comp. Physiol., 103, 97–121. Carr, C. E. (1986). Time coding in electric fish and barn owls. Brain Behav. Evol., 28, 122–133. Carr C. E., & Amagai, S. (1996). Processing of temporal information in the brain. In M. A. Pastor & J. Artieda (Eds.), Time, internal clocks and movement (pp. 27– 52). New York: Elsevier. Carr, C. E., & Boudreau, R. E. (1993). Organization of the nucleus magnocellularis and the nucleus laminaris in the barn owl: encoding and measuring interaural time differences. J. Comp. Neurol., 16, 223–243. Carr, C. E., Heiligenberg, W., & Rose, G. (1986). A time-comparison circuit in the electric fish midbrain. I. Behavior and physiology. J. Neurosci., 6, 107–119. Carr, C. E., & Konishi, M. (1988). Axonal delay lines for time measurement in the owl’s brainstem. Proc. Natl. Acad. Sci., 85, 8311–8315. Carr, C. E. (1993). Timing mechanisms in the CNS. Annu. Rev. Neurosci., 16, 223–243. Carr, C. E., & Konishi, M. (1990). A circuit for detection of interaural time differences in the brainstem of the barn owl. J. Neurosci., 10, 3227–3246. Carr, C. E., Maler, L., & Taylor, B. (1986). A time comparison circuit in the electric fish midbrain. II. Functional morphology. J. Neurosci., 6, 1372–1383. Enger, P. S., Libouban, S., & Szabo, T. (1976). Rhombencephalic connections in the fast conducting electrosensory system of the mormyrid fish, Gnathonemus petersii. An HRP study. Neurosci. Letter, 3, 239–243. Fay, R. R. (1988). Hearing in vertebrates: A psychophysics databook. Winnetka, IL: Hill-Fay Associates.
Evolution of Time Coding Systems
17
Finlay, B., & Darlington, R. (1995). Linked regularities in the development and evolution of mammalian brains. Science, 268, 1578–1584. Friedman, M. A., & Hopkins, C. D. (1998). Neural substrates for species recognition in the time-coding electrosensory pathway of mormyrid electric fish. J. Neurosci., 18, 1171–1185. Friedman, M. A., & Kawasaki, M. (1997). Calretinin-like immunoreactivity in mormyrid and gymnarchid electrosensory and electromotor systems. J. Comp. Neurol., 387, 341–357. Goldberg, J. M., & Brown, P. B. (1969). Response of binaural neurons of dog superior olivary complex to dichotic tonal stimuli: Some physiological mechanisms of sound localization. J. Neurophysiol., 32, 613–636. Guo, Y-X., & Kawasaki, M. (1997). Representation of accurate temporal information in the electrosensory system of the African electric fish, Gymnarchus niloticus. J. Neurosci., 17, 1761–1768. Haugedé-Carré, F. (1979). The mesencephalic exterolateral posterior nucleus of the mormyrid fish Brienomyrus niger: Efferent connections studied by the HRP method. Brain Res., 178, 179–184. Heffner, R. S., & Heffner, H. E. (1992). Evolution of sound localization in mammals. In D. B. Webster, R. R. Fay, & A. N. Popper (Eds.), The evolutionary biology of hearing. (pp. 691–716). New York: Springer-Verlag. Heiligenberg, W. (1989). Coding and processing of electrosensory information in gymnotiform fish. J. Exp. Biol., 146, 255–275. Hopkins, C. D. (1986a). Behavior of mormyridae. In T. H. Bullock & W. Heiligenberg (Eds.), Electroreception. (pp. 527–576). New York: Wiley. Hopkins, C. D. (1986b). Temporal structure of non-propagated electric communication signals. Brain Behav. Evol., 28, 43–59. Heiligenberg, W., & Rose, G. (1985). Phase and amplitude computations in the midbrain of an electric fish: Intracellular studies of neurons participating in the jamming avoidance response of Eigenmannia. J. Neurosci., 5, 515–531. Hopkins, C. D., & Bass, A. H. (1981). Temporal coding of species recognition signals in an electric fish. Science, 212, 85–87. Jeffress, L. A. (1948). A place theory of sound localization. J. Comp. Physiol. Psych., 41, 35–39. Jhaveri, S., & Morest, K. (1982). Neuronal architecture in nucleus magnocellularis of the chicken auditory system with observations on nucleus laminaris: A light and electron microscope study. Neurosci., 7, 809–836. Joseph, A. W., & Hyson, R. L. (1993). Coincidence detection by binaural neurons in the chick brain stem. J. Neurophysiol., 69, 1197–1211. Kawasaki, M. (1993). Independently evolved jamming avoidance responses employ identical computational algorithms: A behavioral study of the African electric fish, Gymnarchus niloticus. J. Comp. Physiol., 173, 9–22. Kawasaki, M. (1996). Comparative analysis of the jamming avoidance response in African and South American wave-type electric fishes. Bio. Bull., 191, 103– 108. Kawasaki, M. (1997). Sensory hyperacuity in the jamming avoidance response of weakly electric fish. Curr. Opin. Neurobiol., 7, 473–479. Kawasaki, M., & Guo, Y-X. (1996). Neuronal circuitry for comparison of timing
18
C. E. Carr and M. A. Friedman
in the electrosensory lateral line lobe of an African wave-type electric fish, Gymnarchus niloticus. J. Neurosci., 16, 380–391. Kawasaki, M., Rose, G., & Heiligenberg, W. (1988). Temporal hyperacuity in single neurons of electric fish. Nature, 336, 173–176. Keller, C. H., & Takahashi, T. T. (1996). Binaural cross-correlation predicts the responses of neurons in the owl’s auditory space map under conditions simulating summing localization. J. Neurosci., 16, 4300–4309. Kiang, N. Y. S., Watanabe, T., Thomas, E. C., & Clark, E. F. (1965). Discharge patterns of single fibers in the cat’s auditory nerve. Cambridge, MA: MIT Press. Knudsen, E. I., du Lac, S., & Esterly, S. D. (1987). Computational maps in the brain. Ann. Rev. Neurosci., 10, 41–65. Konishi, M. (1973). How the owl tracks its prey. Am. Sci. 61, 414–424. Konishi, M. (1986). Centrally synthesized maps of sensory space. Trends Neurosci., 9, 163–168. Konishi, M. (1991). Deciphering the brain’s codes. Neural Computation, 3, 1–18. Levin, M. D., Schneider, M., Kubke, M., Wenthold, R., & Carr, C. E. (1997). Localization of glutamate receptors in the auditory brainstem of the barn owl. J. Comp. Neurol., 378, 239–253. Lissman, H. (1958). On the function and evolution of electric organs in fish. J. Exp. Biol., 35, 156–191. Maler, L., Jande, S., & Lawson, E. M. (1984). Localization of vitamin D–dependent calcium binding protein in the electrosensory and electromotor system of high frequency gymnotid fish. Brain Res., 301, 166–170. Maler, L., Sas, E., & Rogers, J. (1981). The cytology of the posterior lateral line lobe of high frequency weakly electric fish (Gymnotoidei): Dendritic differentiation and synaptic specificity in a simple cortex. J. Comp. Neurol., 195, 87–140. Manis, P. B., & Marx, S. O. (1991). Outward currents in isolated ventral cochlear nucleus neurons. J. Neurosci., 11, 2865–2880. Massoglia, D. P. (1997). Embryonic development of the time coding nuclei in the brainstem of the barn owl. Unpublished master’s thesis, University of Maryland. Mugnaini, E., & Maler, L. (1987). Cytology and immunocytochemistry of the nucleus extrolateralis anterior of the mormyrid brain: Possible role of GABAergic synapses in temporal analysis. Anat. Embryol., 176, 313–336. Oertel, D. (1985). Use of brain slices in the study of the auditory system: Spatial and temporal summation of synaptic inputs in cells in the anteroventral cochlear nucleus of the mouse. J. Acoust. Soc. Am., 78, 328–333. Oertel, D., (1997). Encoding of timing in the brain stem auditory nuclei of vertebrates. Neuron, 19, 959–962. Otis, T. S., Raman, I. M., & Trussell, L. O. (1995). AMPA receptors with high Ca2+ permeability mediate synaptic transmission in the avian auditory pathway. J. Physiol. (Lond.), 482, 309–315. Overholt, E. M., Rubel, E. W., & Hyson, R. L. (1992). A circuit for coding interaural time differences in the chick brain stem. J. Neurosci., 12, 1698–1708. Parks, T. N., Code, R. A., Taylor, D. A., Solum, D. A., Strauss, K. I., Jacobowitz, D. M., & Winsky, L. (1997). Calretinin expression in the chick brainstem auditory nuclei develops and is maintained independently of cochlear nerve input. J. Comp. Neurol., 383, 112–121.
Evolution of Time Coding Systems
19
Payne, R. S. (1971). Acoustic localization of prey by barn owls (Tyto alba). J. Exp. Biol., 54, 535–573. Raman, I. M., & Trussell, L. O. (1992). The kinetics of the response to glutamate and kainate in neurons of the avian cochlear nucleus. Neuron, 9, 173–186. Ravindranathan, A., Parks, T. N., & Rao, M. S. (1996). Flip and flop isoforms of chick brain AMPA receptor subunits: Cloning and analysis of expression patterns. Neuroreport, 7, 2707–2711. Reyes, A. D., Rubel, E. W., & Spain, W. J. (1994). Membrane properties underlying the firing of neurons in the avian cochlear nucleus. J. Neurosci., 14, 5352–5364. Rhode, W. S., Oertel, D., & Smith, P. H. (1983). Physiological response properties of cells labeled intracellularly with horseradish peroxidase in cat ventral cochlear nucleus. J. Comp. Neurol., 213, 448–463. Rose, G., & Heiligenberg, W. (1985). Temporal hyperacuity in the electric sense of fish. Nature, 318, 178–180. Rubel, E. W., & Parks, T. N. (1975). Organization and development of brainstem auditory nuclei of the chicken: Tonotopic organization of N. magnocellularis and N. laminaris. J. Comp. Neurol., 164, 411–434. Ryugo, D. K., & Fekete, D. M. (1982). Morphology of primary axosomatic endings in the anteroventral cochlear nucleus of the cat: A study of the endbulbs of Held. J. Comp. Neurol., 210, 239–257. Scheich, H., Bullock, T. H., & Hamstra, R. H. (1973). Coding properties of two classes of afferent nerve fibers: High frequency electroreceptors in the electric fish Eigenmannia. J. Neurophysiol., 36, 39–60. Smith, P. H., Joris, P. X., & Yin, T. C. T. (1993). Projections of physiologically characterized spherical bushy cell axons from the cochlear nucleus of the cat: Evidence for delay lines to the medial superior olive. J. Comp. Neurol., 331, 245–260. Smith, Z. D. J., & Rubel, E. W. (1979). Organization and development of brainstem auditory nuclei of the chicken: Dendritic gradients in nucleus laminaris. J. Comp. Neurol., 186, 213–239. Sullivan, W. E., & Konishi, M. (1984). Segregation of stimulus phase and intensity coding in the cochlear nucleus of the barn owl. J. Neurosci., 4, 1787–1799. Sullivan, W. E., & Konishi, M. (1986). Neural map of interaural phase difference in the owl’s brainstem. Proc. Natl. Acad. Sci. 83, 8400–8404. Szabo, T. (1965). Sense organs of the lateral line system in some electric fish of the Gymnotidae, Mormyridae and Gynmarchidae. J. Morphol., 117, 229–250. Szabo, T., Ravaille, M., Libouban, S., & Enger, P. S. (1983). The mormyrid rhombencephalon. I. Light and EM investigations on the structure and connections of the lateral line lobe nucleus with HRP labeling. Brain Res., 26, 1–19. Takahashi, T. T., Carr, C. E., Brecha, N., & Konishi, M. (1987). Calcium binding protein-like immunoreactivity labels the terminal field of nucleus laminaris of the barn owl. J. Neurosci., 7, 1843–1856. Takahashi, T., Moiseff, A., & Konishi, M. (1984). Time and intensity cues are processed independently in the auditory system of the owl. J. Neurosci., 4, 1781–1786.
20
C. E. Carr and M. A. Friedman
Trussell, L. O. (1997). Cellular mechanisms for preservation of timing in central auditory pathways. Curr. Opin. Neurobiol., 7, 487–492. Winter, P., & Schwartzkopff, J. (1961). Form und Zellzahl der akustischen Nervenzentren in der Medulla oblongata von Eulen (Striges). Experentia, 16, 515– 517. Wu, S. H., & Oertel, D. (1984). Intracellular injection with horseradish peroxidase of physiologically characterized stellate and bushy cells in slices of mouse anteroventral cochlear nucleus. J. Neurosci., 4, 1577–1588. Yin, T. C. T., & Chan, J. C. K. (1990). Interaural time sensitivity in medial superior olive of cat. J. Neurophysiol., 64, 465–488. Young, S. R., & Rubel, E. W. (1983). Frequency-specific projections of individual neurons in chick brainstem auditory nuclei. J. Neurosci., 7, 1373–1378. Zhang, S., & Trussell, L. O. (1994). Voltage clamp analysis of excitatory synaptic transmission in the avian nucleus magnocellularis. J. Physiol. (Lond.), 480, 123–136. Received February 19, 1998; accepted June 2, 1998.
ARTICLE
Communicated by Gunther Palm
Computing with Self-Excitatory Cliques: A Model and an Application to Hyperacuity-Scale Computation in Visual Cortex Douglas A. Miller∗ Steven W. Zucker Center for Computational Vision and Control, Departments of Computer Science and Electrical Engineering, Yale University, New Haven, CT 06520, U.S.A.
We present a model of visual computation based on tightly inter-connected cliques of pyramidal cells. It leads to a formal theory of cell assemblies, a specific relationship between correlated firing patterns and abstract functionality, and a direct calculation relating estimates of cortical cell counts to orientation hyperacuity. Our network architecture is unique in that (1) it supports a mode of computation that is both reliable and efficent; (2) the current-spike relations are modeled as an analog dynamical system in which the requisite computations can take place on the time scale required for an early stage of visual processing; and (3) the dynamics are triggered by the spatiotemporal response of cortical cells. This final point could explain why moving stimuli improve vernier sensitivity. 1 Introduction Consider a region of our visual field innervating a patch of primary visual cortex a few millimeters on a side, containing, say, 106 mostly mutually excitatory neurons. Does it make sense to talk about this small patch of cortex as a basic unit of our visual system, and, if so, how crude is it? For example, how many distinct lines could just this one patch of cortex reliably represent? We suggest that knowing whether the biggest possible number is 10, 104 , or 107 is critical to the understanding of our visual system, because to determine this number we must address how this unit of cortex is functioning. Thus our main goal in this article is to develop a model of computation sufficiently biological to apply to this patch of cortex, but that can also be viewed as an abstract model of computational vision that can ∗
Douglas A. Miller, formerly of the Center for Intelligent Machines, McGill University, Montreal, Canada, passed away in 1994. This article is derived from “A Model of Hyperacuity-scale Computation in Visual Cortex by Self-excitatory Cliques of Pyramidal Cells,” TR-CIM-93-12, August 1993. Portions were also presented at the Workshop on Computational Neuroscience, Marine Biological Laboratories, Woods Hole, MA, in August, 1993. Neural Computation 11, 21–66 (1999)
c 1999 Massachusetts Institute of Technology °
22
Douglas A. Miller and Steven W. Zucker
represent lines. To reduce this goal to manageable proportions, we focus on the representation of orientation for short line segments. The choice of studying the representation of short line segments was made for several reasons. To begin, we are developing a theory of visual information processing based on tangents to image curves (Zucker, Dobbins, & Iverson, 1989; Dobbins, Zucker, & Cynader, 1987). The concrete realization of these tangents is short, oriented segments, the natural limit to which is given by orientation hyperacuity (Westheimer, 1990). However, while most neurons in visual cortex are orientation selective, their tuning is broad (±7.5 degrees in monkeys before significant falloff). Even the smallest receptive fields are ≥ 12 arc min2 on average in monkeys and humans (Hubel & Wiesel 1968; Wilson, Levi, Maffei, Rovamo, & DeValois, 1990). These facts stand in crude comparison with the precision of hyperacuity involving contours with widths of about 1 arc min. Thus, our second motivation is to understand how this (and only this) precise level of functional performance can be achieved with coarse units. Such questions raise the more general one: Are individual neurons the natural basis for expressing visual computations, or are visual computations most naturally expressed at the circuit level? We derive the surprising result that, according to our model, a circuit structure can take processing elements (neurons) that by themselves are both extremely crude and highly unreliable, and create aggregate units that are both extremely precise and highly reliable. This result specifies a type of cell assembly very much in the spirit of Hebb’s (1949) original proposal and those of Braitenberg (1974, 1978, 1985; Braitenberg & Schuez, 1991) and Palm (1980, 1982, 1993; Palm & Aertsen, 1986; Wennekers, Sommer, & Palm, 1995). Nevertheless, there are important technical differences from these earlier works. We consider analog neural models, while Braitenberg, Palm, Aertsen, and the others consider spiking ones. (The relevant issues of time constants are discussed later.) Our model postulates a level of organization among neurons more precise than single-cell receptive field properties, and it provides a formal bridge between spike-train distributions and abstract function (cf. Softky & Koch, 1992; Tsodyks, Mitkov, & Sompolinsky, 1993). Most important, our model differs in fundamental ways from previous attempts to construct reliable systems from unreliable components. In particular, Moore and Shannon (1956) developed the foundations of highly redundant recursive systems, but these seem unsuitable for biology because the redundant circuitry is largely a sink for energy without providing any interim benefit. Our third motivation, then, is to seek architectures that achieve reliability without wasted redundancy. In particular, we will show how, in our parallel analog model of computation, the extra number of processors needed to achieve reliability at the same time increases the information content of the system (cf. Winograd & Cowan, 1963; Barlow, 1983). The key ideas behind our model and the corresponding mode of com-
Computing with Self-Excitatory Cliques
23
putation, are, first, the idea of groups of tightly interconnected excitatory neurons capable of bringing themselves to saturation feedback response following a modest initial afferent bias current, rather like a match igniting a conflagration, to borrow a simile from Douglas and Martin (1992). We call these groups cliques.1 Second, these cliques of neurons, although there may be an extremely large number of them, are themselves relatively small, consisting of perhaps a few dozen excitatory pyramidal cells with certain characteristics. On the other hand, an individual such cell may simultaneously belong to hundreds of different cliques, and therein lies the system’s ability to store and retrieve large quantities of information reliably with highly unreliable processing elements. We schematically illustrate the basic retrieval computation in Figure 1. The assumption that excitatory groups of neurons exist is based on the following facts (see also Douglas, Koch, Mahowald, & Martin, 1995): 1. Most connections are from spiny excitatory cortical cells to other spiny excitatory cortical cells within a few millimeters horizontal distance (Douglas & Martin, 1990). 2. Most, if not all, direct afferents from the lateral geniculate nucleus are excitatory (Bishop, Coombs, & Henry, 1973; Ferster & Lindstrom, ¨ 1983; Douglas & Martin, 1992). 3. Inhibition is much less specific than excitation, both temporally and spatially, and is additive, not multiplicative (Bishop et al., 1973; Douglas, Martin, & Whitteridge, 1988). 4. Most excitatory connections from layer 2/3 pyramidal cells are to other layer 2/3 or possibly layer 5 pyramidal cells with about the same orientation specificity (Rockland & Lund, 1982, 1983; Rockland, Lund, & Humphrey, 1982; Gilbert & Wiesel, 1983, 1989; Callaway & Katz, 1990). 5. Most, if not all, connections in visual cortex undergoing long-term potentiation and depression (Hebbian learning) are between spiny excitatory cells (Artola, Brocher, ¨ & Singer, 1990; Douglas & Martin, 1990). (This point is also important for learning cliques, which we shall not discuss in this article.) In addition to these structural features, two aspects of neuronal dynamics are central to our model of computing with cliques. First, most excita1 Clique in directed graph theory refers to a subset of nodes (neurons, in this case) that is completely interconnected, that is, having an arc (synapse) from each node to each other node. Such a perfect arrangement is unlikely in biology, and indeed subsets of neurons that are merely interconnected to a sufficiently high degree are sufficient for our analysis (cf. section 3.1). Thus, we will use clique to refer to either kind of set and assume that the exact meaning will be made clear from the context. Braitenberg (1985) stressed the role of synchronous firing in establishing such connections.
24
Douglas A. Miller and Steven W. Zucker
tory cortical cells exhibit a typical “regular spiking” behavior (McCormick, Connors, Lighthall, & Prince, 1985; Douglas & Martin, 1990), which adapts within a 25 msec period from an initial high spiking rate (e.g., 300 Hz) to a much lower sustained rate (see Figure 2). We shall take this initial higher rate of activity as fundamental and use it to carry the relevant information. In particular, the conflagration referred to earlier will be carried by short bursts of rapid activity for all neurons within a clique. This also implies that the time scale for a computation should be around 25 msec, a time scale that is relevant for cortical computations in V1 given that total processing can occur within 200 msec. Diffuse inhibition then returns the system to base state. Heller, Hertz, Kjaer, and Richmond (1995), in an attempt to characterize the information transmitted about visual patterns as a function of time,
Computing with Self-Excitatory Cliques
25
found that 25 ms emerged as a critical interval in visual cortex. This empirical finding agrees with the above observations. The second dynamical aspect of our model derives from the observation that whereas orientation sensitivity is crude, spatiotemporal precision in response to moving edges is high for a large class of cortical cells (Bishop, Coombs, & Henry, 1971; Bishop et al., 1973; Bishop, Dreher, & Henry, 1972; Schiller, Finlay, & Volman, 1976a,b,c; Henry, 1977). Henry (1977) called these S cells, and we shall take an abstraction of them as the basic neural unit for our clique-based computation. It is this specificity that will provide the necessary timing to ignite the “conflagration.” The S cell category is derived from the Hubel and Wiesel (1962, 1968) simple cell, but is different in that its response is based primarily on moving edges. In particular, the S cell, while often (but not always) having distinct stationary ON/OFF response zones, does not respond to moving contour stimuli with additive summation in the direction perpendicular to orientation, nor does it tend to exhibit antagonism between the ON/OFF regions for such stimuli. Furthermore, its response zones to moving contours cannot in general be predicted more than very approximately from the location of any classic ON/OFF zones it may have (Bishop et al., 1972; see the discussion in Bishop et al., 1973, p. 58). In addition the direction selectivity
Figure 1: Facing page. A schematic illustration of the two phases of the computation for activating a cortical clique. The model is sketched in terms of layer 2/3 pyramidal S cells, for reasons developed in the text. (Top) Prior to phase I, the layer 2/3 pyramidal cells in a patch of cortex are initially largely quiescent, corresponding to an initial state for the computation. Among these hundreds of thousands of cells are several times that number of cortical cliques, each containing about 33 highly interconnected cells. A “computation” amounts to activating the cells in a single clique, but no others, to saturation feedback response levels. (Middle) In the beginning of phase I, afferent stimulation, either directly or indirectly (e.g., from the LGN) produces a single spike in a majority of the cells (filled) in one clique (whose cells can be distributed among many different isoorientation areas, including the three illustrated), as well as a certain number of other cells (also filled) outside the clique (noise). (Bottom) Because the clique has a sufficient level of excitatory interconnections, all of its cells (filled) drive themselves, as part of an analog dynamical system (cf. Figure 5), to saturation response levels of about 5 spikes in 25 msec (end of phase I), whereas the initially activated cells outside the clique (now open) ultimately return to their resting membrane potentials and do not spike further (end of phase II). Thus, the clique has been “retrieved” through a parallel analog computation, and during this 25 msec period is available as input for further processing in the cortex or elsewhere. Adaptive spiking properties of pyramidal cells (cf. Figures 2 and 5, top), possibly combined with general inhibitory mechanisms, terminate the computation, and return the system to the initial quiescent state.
26
Douglas A. Miller and Steven W. Zucker
Figure 2: Spike train of a regular spiking cortical neuron under constant 0.6 nA current stimulation. The cell was recorded from layer 5 of rat visual cortex, in vitro. Note the characteristic rapid transient and subsequent decrease in the firing rate to mean level (cf. Figure 5, top). (Data kindly supplied by R. Douglas)
of response can differ from what would be predicted from the Hubel and Wiesel criteria for simple cells (Hubel & Wiesel, 1962). Rather, when spontaneous activity is artificially enhanced, or exists naturally at an exceptionally high level, well-defined response peaks tend to appear in both directions of motion for both light and dark edges (Bishop et al., 1973, Fig. 1; Schiller et al., 1976a, Fig. 14A).2 Such an S cell response histogram is given in Figure 3. Furthermore within broad limits, the spatial location of S cell edge responses appears to be independent of contour velocity (Bishop et al., 1971, Figures 5, and 6). In the absence of natural or artificial spontaneous firing, an S cell typically responds to only certain edges and directions of motion. 2 A light edge is one whose light portion is increasing, and a dark edge is one whose dark portion is increasing. Thus, the same instantaneous visual configuration can be a light or a dark edge depending on its direction of motion.
Computing with Self-Excitatory Cliques
27
The spatiotemporal precision of S cells differs from the average firing rate class of codes normally considered. This raises the question of whether the retina or the lateral geniculate nucleus (LGN) are capable of providing a cohort of spikes sufficently close in time to start the process we are studying. Although the question is still open experimentally, recent retinal analyses suggest that such information may well be available in synchronous spike distributions (Meister, 1996). Indirect evidence for this has been in the literature for some time (Victor, 1987). Research in the LGN also has uncovered quite a bit of information in spike timing (McClurkin, Gawna, Optical, & Richmond, 1991). Finally, we note that another coding scheme (different from ours) has recently been proposed that is also triggered by rapid spikes (doublets) (see Traub, Whittington, Stanford, & Jefferys, 1996). The clique, then, will consist of a collection of S cells scattered slightly in position and orientation (see Figure 4). This mode of computation amounts (cf. Georgopoulos, Lurito, Petrides, Schwartz, & Massey, 1989; Gilbert & Wiesel, 1990; Lehky & Sejnowski, 1990) to a distributed code for hyperacuity, but several critical features distinguish it from other distributed code models. In particular there is no population averaging. The fact that a highly specific visual event has occurred at a specific time is announced to the rest of the brain by the (roughly) simultaneous saturationlevel activation of a few dozen normally quiescent cells in primary visual cortex. In particular, one does not need to measure the time intervals of the spike trains that are produced or find a higher-level cell to average the response histograms of the cells that are spiking. Rather, it is the fact that a particular clique of cells are firing at saturation levels that is the crucial piece of information being conveyed. While others have stressed the possibilities of coincident firing (e.g., Abeles, 1991), a major aspect of our work lies in demonstrating, with a combination of existing empirical data and mathematical and probabilistic analysis, that a primate’s primary visual cortex is capable of storing a sufficient number of self-excitatory cliques (and hence this information) to constitute a hyperacuity-scale representation of the external visual world. Thus, although our model is in the spirit of those that attempt to solve binding problems by a form of synchrony (for a review, see Konig ¨ & Engel, 1995), the proposal is substantially more concrete. Two limits come into play. One is the sheer number of cells, particularly the adaptive or regular spiking (McCormick et al., 1985) orientation selective pyramidal S and SH cells (Henry, 1977),3 which are the predominant type in layers 2/3 in cats (Henry, Harvey, & Lund, 1979; Martin, 1984) and a major if not predominant type in layers 2/3 in monkeys (Schiller et al., 1976a; see the discussion in section 2.2). The other limit is the temporalspatial resolution of these cells (Bishop et al., 1971, 1972, 1973; Schiller et al.,
3 The S cell is an end-stopped S cell. Hereafter, we shall generally include S cells in H H the S category.
28
Douglas A. Miller and Steven W. Zucker
1976a,b,c; cf. Figure 2). We shall show that by increasing the numbers of such cells, one can make optimal use of this resolution, resulting in the potential for hyperacuity-scale performance. Thus, we can view hyperacuity as the end result of what is available from optics, the physiology of neurons, the
Computing with Self-Excitatory Cliques
29
amount of brain tissue we can support, and the need for high-resolution vision. The remainder of this article is organized in three sections. In section 2 we develop our model informally, concentrating on its physiological interpretation and justification. Section 3 is mathematical. There we state our model precisely and prove several of its key properties. In particular Proposition 1 describes the dynamic behavior properties of a system of excitatory neurons modeled as a dynamical system of analog voltage amplifiers (cf. Hopfield, 1984, and Figure 5). In particular, for the latter system, an arbitrarily small output from a sufficiently large proportion of cells will drive the entire group of cells to saturation response. In a clique of real neurons, we argue that this corresponds to the case of a significantly large proportion of a group of highly interconnected cells receiving single-spike lower-level afferent stimulation, thus driving the entire group to saturation firing.
Figure 3: Facing page. Neuronal response to a narrow slit of light is a composition of separate finely tuned responses to each of its two edges. (Top) A response histogram for an S cell from the parafoveal primary visual cortex of a macaque monkey, for 20 passes of a 0.27 degree slit of light from right to left visual field at a velocity of 2 deg/sec. Histogram is from left to right with time, and for each pass, each bin counts spikes over a period of 23.4 msec and corresponds to a stimulus movement of .046 deg. (Middle) Same as above, but with slit width of 0.63 degree. Two separate peak responses appear, consistent with the hypothesis that the cell is responding separately to the leading (light) and trailing (dark) edge of the slit. Separation of the peaks is about 0.21 degree less than the separation of the slit, which Schiller et al. (1976a) interpret as meaning that the light and dark edge response regions are separated by about that much, with the light edge response region to the left in the visual field of that for the dark edge. This cell had an exceptionally high spontaneous activity level, and therefore one can observe sideband inhibition caused by the stimuli in both top and middle (zero-level portion at extreme right of both graphs is not part of data). Two vertical arrows in the bottom graph indicate the beginning of the kind of sudden response onset predicted by our theory. The 25 msec spike train described in the text could be expected to fall within two bins. The two adjacent bins beginning with the first arrow represent an average response of 4.3 spikes, and the two adjacent bins beginning with the second arrow represent an average response of 7.2 spikes. (Bottom) Spatiotemporal receptive field positions of the edges signaled by the beginnings of the spike trains described above. Light edge location is indicated by 4.3 spikes on average, dark edge by 7.2 spikes. (Histograms redrawn from Schiller et al. 1976a)
30
Douglas A. Miller and Steven W. Zucker
Figure 4: Receptive fields for representing a thin line contour. A white slit contour stimulus about 1.5 arc min width (heavy black outline) excites a highly interconnected clique of cortical S cells, whose receptive fields are represented here by (12 arc min)2 rectangles, to maximal saturated feedback response by crossing, in the appropriate direction and within a narrow time interval, the edge response region (cf. Figure 3) of a sufficiently large proportion of the clique’s cells. Three such receptive fields, forming part of a larger clique, are illustrated here. According to the theory proposed in the text, a small change in orientation of the contour will cause a modest decrease in the probability of an initial spike from the cells in the clique that would be sufficient to prevent a saturated feedback response from the clique as a whole (cf. Figure 6).
Proposition 2 is a technical result used in our calculation (done in section 3.2) of the number of storable cliques of cells and the amount of tolerable input noise (cf. Figure 6 and section 3.2). In addition, we rigorously define in this section the key notions of self-excitatory set, minimal self-excitatory set,
Computing with Self-Excitatory Cliques
31
and clique. We then provide several applications of the model. An important result of the next section is the calculation of a cortical packing density for layer 2/3 S-type pyramidal cells, which would be necessary and sufficient for observed orientation hyperacuity data in humans. The cell density calculated turns out to be very close to that observed for macaque monkeys. Following the discussion we also include an appendix containing a glossary of symbols. 2 A Clique-Based Model of Cortical Computation Let us describe in more detail our analog network model for intrinsic horizontal connections in a small patch of primary visual cortex (an exact mathematical description is given in section 3). Note, in particular, how it differs from the input-output view of controlled amplification proposed in Douglas et al. (1995). Our cortical computational model supposes that we start from a stable reference state in which all or most neurons are quiescent and then activate a relatively small number of them. Thus, our model stores states as highly interconnected self-excitatory cliques of neurons (represented by voltage amplifiers), where activation of some proportion greater than half of the clique members beyond a certain moderate level fully activates the entire clique. There are two phases to this procedure, corresponding to two computations by the network. In the first phase, all input amplifiers are activated at some moderate level by a fixed set of input biases. Those amplifiers undergoing self-excitatory feedback then drive to their saturation values. In the second phase, the input biases are removed, and all amplifier responses not
Figure 5: Facing page. Comparison of cortical pyramidal cell input-output with that of model neurons. (Top) First, second, and fifth interspike intervals in spikes per second as a function of input current for a rat layer 5 pyramidal cell (redrawn from Douglas & Martin, 1990). (Middle) Expected total spike output during 3.5 msec of a hypothetical clique of 33 neurons of the kind described in the top part, interpreting the spike rate of each of these neurons as the arrival rate of an independent Poisson process. Each member of the clique experiences the clique output as input, so that we may model the clique response as a whole with an equivalent collection of purely analog amplifiers, as in the bottom. (Bottom) Since a cortical somatic time constant is typically much larger than 3.5 msec (e.g., 10–20 msec [Douglas & Martin, 1991]), we may assume an approximately linear current-voltage relation, and thus model a clique of these cells with a clique of voltage amplifiers with the same shape input-output function. Examples of possible input-output function for piecewise-linear voltage amplifiers are used in model neurons. The simplest is given by a solid line. A more complicated function, based on that empirically derived in Figure 8, middle, is given by the dashed line.
32
Douglas A. Miller and Steven W. Zucker
Computing with Self-Excitatory Cliques
33
Figure 6: Number of cliques and noisy inputs as a function of network size. (Top) Computed lower bounds on the maximum numbers of randomly chosen self-excitatory cliques that can be formed in a network of analog model neurons as described in the text, as a function of number of neurons N, for clique sizes 15, 25, 33, and 60. In all cases except size 15, for sufficiently large networks, the number of cliques becomes limited by the number of connections available, so that the graphs become linear in N, taking the value Nf/M2 , where f is the dendritic fan-in and M is the clique size. (Bottom) Maximum number of random inputs (noise) in addition to coherent input for the same clique sizes as above, and assuming the number of cliques stored is at the maximum as given above. (See the text for the calculation of the top and bottom graphs.)
34
Douglas A. Miller and Steven W. Zucker
Figure 7: Probability of an entire clique firing as a function of the probability of its individual neurons firing. In this idealized case, all neurons have the same independent probability p (abscissa) of an initial spike within a given time interval. Assumed clique size is 33, and at least 17 neurons firing are assumed necessary for activation of the clique. The steep sigmoid curve implies that the reliability of the clique is virtually perfect even if individual neurons fire with 0.3 probability when they should not, and only 0.75 probability when they should (The computation is based on the 17-out-of-33 system discussed in section 5.)
undergoing feedback excitation decay out. We illustrate this process schematically in Figure 1. This two-phase computational model becomes relevant to the visual cortex by considering a 25 msec computational time frame and the phenomenon of regular adaptive spiking to constant input current (McCormick et al., 1985; Douglas & Martin, 1990), which can be understood in terms of cell membrane properties (McCormick, 1990). Thus, a moderately stimulated cell would be likely to produce no more than a single spike in a 25 msec interval (see Figure 5, top, responses for 0.3—0.6 nA), but one that is fully excited (1.5 nA) could produce about five (cf. also McCormick et al., 1985, Fig. 1). Since a cortical somatic time constant is typically much larger than the intervals we shall be concentrating on (as described below, 3.5 msec as compared with the 10–20 msec time constant; Douglas & Martin, 1991), we may assume an approximately linear current-voltage relation, and thus model a clique of these cells with a clique of voltage amplifiers with the same-shape input-output function.
Computing with Self-Excitatory Cliques
35
We presume that under either direct or indirect afferent stimulation, there is sufficient input current arriving at a given target cell to produce a single spike (see Figure 1, middle). For members of a clique, this will produce a large number of additional excitatory postsynaptic potentials within a few msec (see Figure 1, bottom), thus producing more spikes in these same cells, and any others in the clique, and bringing current input to these cells to saturation levels, as in the higher abscissa values in Figure 5 (middle). This chain of events will happen if and only if the initial proportion of activated clique members is sufficiently large. We model this dynamic ensemble feedback process among excitatory neurons with piecewise-linear voltage amplifiers (see Figure 5, bottom). During this 25 msec computation, the mild additive feedback inhibition which appears to be an intrinsic part of the visual cortex (Douglas et al., 1988; Douglas, Martin, & Whitteridge, 1989), could suppress any tendency of a nonclique cell receiving little or no excitatory feedback to respond more than once in the interval, but would be ineffective against clique cells undergoing short-term feedback excitation. In effect, the inhibition would sharpen the distinction between the two classes of cells and reduce noise. The first class of cells could be regarded as undergoing an imprecise decaying input current, the second a highly precise, mutually activating response (see Figure 1, middle and bottom). In addition (or alternately), inhibition could serve to deactivate the system following a given computation, thus setting the stage for the next one, or allowing the same cells to represent other (e.g., downflowing) information in a separate time regime. As we show elsewhere (Miller & Zucker, in press), the end results of both phases can be computed in a number of computational steps, which is polynomial in the number of bits needed to specify the problem (as defined in section 3). This can be done by a type of algorithm that we have described previously (Miller & Zucker, 1991, 1992) and is closely related to the wellknown simplex method for linear programming. What this analysis shows is that the problem of retrieving information within our cortical model is not in the class of computationally hard problems such as the traveling salesman (Hopfield & Tank, 1985) for which no such algorithm is likely to exist (Garey & Johnson, 1979). Knowing that a fundamental computation is performable in polynomial time would appear to be a distinct advantage in modeling systems on the scale of the visual cortex. Two points should be stressed. First, the relative smallness (sparsity) of these cliques and the relatively small number of afferents that may be active at any given time is critical, in order to obtain sufficiently many cliques to represent line contours on hyperacuity levels (one contour per clique) and in order for the computation not to end up activating the wrong cliques or, at the extreme, producing a catastrophic epileptic response among the entire cell population. Second, the specificity of response of an individual S cell to a movingedge contour need only be probabilistic. That is, given our clique model, for
36
Douglas A. Miller and Steven W. Zucker
the entire clique to fire reliably when the contour is in a specific location and not otherwise, it is necessary only that the individual neurons composing the clique have a modest increase or decrease, respectively, in their individual probabilities of firing. Sufficient probabilities for the model we shall derive are given in Figure 7. We now describe some of the principal neurophysiological assumptions of our cortical model and describe how these assumptions are justified by empirical evidence. The first assumption is to limit a priori the possible number of input units that can be activated. This is consistent with our knowledge of cells in the visual cortex, where investigators often find it difficult to find any stimulus that will make a given neuron respond, and where there is in general an extremely low level of intrinsic activity (cf. Bishop et al., 1971, 1973; Schiller et al., 1976a,b,c). Thus, while it may seem strange at first to suppose a model of cortical computation, as ours is, based primarily on excitation, the reason a model of excitatory computation is possible in the cortex is that there has already been a tremendous amount of filtering in the visual system prior to reaching this stage. Perhaps LGN feedback is involved in this filtering as well. We may observe the vast contrast between this assumption and that of the Hopfield (1982) discrete model, in which a randomly chosen storage state is as likely to have as many on as off states, and the Hopfield (1984) analog model where all the neurons are potentially firing. This assumption is untenable in the visual cortex, not only because cortical neurons are typically quiescent, but because, even if they all could be firing, their average spike rates of 100 Hz and less could not specify the interior state or trajectory of an analog dynamical system with anything approaching the precision that Hopfield’s system seems to demand. However, versions of analog neural network models that are both biologically and computationally plausible are possible, and we have described broad classes of such models in previous work (Miller & Zucker, 1991, 1992). This article may be viewed as an application of this work to the visual cortex. Along these lines, we should note that the information-theoretic efficiency of sparse input discrete associative and autoassociative networks with large numbers of computing elements has been known for some time (Willshaw, Buneman, & Longuet-Higgins, 1969; Willshaw & LonguetHiggens, 1970; Palm, 1981; Amit, Gutfreund, & Sompolinsky, 1987; Amari, 1989; Amit, 1989). Roughly speaking we are extending these discrete sparseinput models to the analog case. We thereby derive a greater biological realism with regard to variable spiking rates and capacitive time constants, and also with regard to the distinction between average and saturation firing rates (cf. Figures 2 and 5). This latter point arises only when considering visual representations through tightly interconnected, self-excitatory cliques of neurons. In addition, such representations permit a number of additional random noisy inputs many times larger than the number of
Computing with Self-Excitatory Cliques
37
cells that would ultimately be responding at saturation (cf. Figure 6, bottom). Our second assumption (also a basis for the sparse-discrete models cited above) is that all recurrent connections (synapses) are capable of being modified only within the same sign—excitatory (see also Abeles, 1991). This assumption is based on the prevalence of glutamergic excitatory connections in the neocortex (Douglas & Martin, 1990), as well as the evidence for NMDA receptor-based long-term potentiation and depression in the primary visual cortex (Artola et al., 1990; Singer, 1990), particularly during the critical period. These receptors are found primarily on dentritic spines, which are the major sites of excitatory input to excitatory cells in the neocortex (Douglas & Martin, 1990). The most dramatic evidence, however, is morphological, since one can literally view the pruning and growth of excitatory pyramidal-pyramidal connections in cat area 17 in the first six postnatal weeks (Callaway & Katz, 1990). Our third assumption, which amounts to saying that a patch of cells in layers 2/3 can be reasonably modeled as though it were a continuous or analog dynamical system (cf. Hopfield, 1984; Cohen & Grossberg, 1983; Sejnowski, 1981), is treated in the following section (see also Gopalsamy & He, 1994; Marcus and Westervelt, 1989; Marcus, Waugh, & Westervelt, 1990; Waugh, Marcus, & Westervelt, 1991). (For different approaches to such modeling, see Abbott & van Vreeswijk, 1993; Harth, Csermely, Beek, & Lindsay, 1970; Nelken, 1988; and Treves, 1993.) Let us consider here, however, whether the inputs to this system can be viewed purely as a bias applied to a quiescent or off state of the system. This assumption is suggested by both the limited analog information storage capacity of cortical neurons (e.g., in the form of average membrane potential) and the growing body of empirical evidence that specific cortical populations respond to specific visual imputs with self-imposed correlations due to mutual feedback excitation combined with mild local inhibitory feedback within periods of 25 msec and less. (Cf. “cortical oscillations” for cats [Eckhorn et al., 1988; Gray & Singer, 1989; Gray, Konig, ¨ Engel, & Singer, 1989], “spatiotemporal distributions” for monkeys [Kruger ¨ & Becker, 1991], and cortical feedback circuits [Douglas et al., 1988, 1989; Douglas & Martin 1990, 1992; see especially Douglas et al., 1995].) Thus, a fundamental lack of short-term precision, and a lack of time for any kind of smoothing integration, argue against a dynamic model in which a well-defined trajectory takes us from a specific interior state to a final computed state, as one finds, for example, in Hopfield (1984), Hopfield and Tank (1985), and Hummel and Zucker (1983). Finally, there is the question of the excitatory cliques themselves, which we may regard as our fourth assumption. The existence of a large number of small subnetworks of cells that are tightly interconnected is consistent with our prior assumptions if we are interested in large-scale efficient storage of information. But what independent evidence is there for such kinds of con-
38
Douglas A. Miller and Steven W. Zucker
nectivity? In fact, it is here that we have the most striking empirical evidence of all, which becomes apparent once we consider the precise “information” we want to store. It is well known that at a coarse resolution, there is a (piecewise) continuous topographical map of visual inputs in the cortex for orientation, ocular dominance, and location of receptive fields. However, if we look more closely, there is within this globally orderly situation a tremendous amount of local randomness. From Hubel and Wiesel (1962, 1968, 1977) and Albus (1975, Part I), we see that (for monkeys and cats) there is a variance (“radial scatter”) in the receptive field location of a cell at a given cortical point. Furthermore, as observed in cats (Albus, 1975, Part II), while the mean orientation may progress continuously across iso-orientation bands, there is at any given point a considerable random variation imposed on the mean selectivity of a given cell. Thus, a given oriented contour in the visual field (see Figure 4) will in all likelihood simultaneously innervate multiple neurons in several locations with about the same orientation preference. This observation has been supported and extended by later ones of Rockland and Lund (1982) and Rockland et al. (1982) in tree shrews, Rockland and Lund (1983) in monkeys, and Gilbert and Wiesel (1983, 1989) (also in cats) that layer 2/3 pyramidal cells in neighboring iso-orientation columns connect primarily to each other (see Figure 1). The improved sensitivity of a group of cells was also noted by Lehky and Sejnowski (1990) for stereo. Finally, we observe that “computational analysis” in the sense popularly used in the neurobiological community has largely come to mean computer simulation of the behaving organism of one kind or another, on either a compartmental scale of individual neurons (e.g., Segev, Fleshman, & Burke, 1989; Douglas & Martin, 1992), a compartmental scale of relatively small networks of neurons (Worg ¨ otter ¨ & Koch 1991), or large-scale simulations that do not attempt to model individual neurons (e.g., Miller, Keller, & Stryker, 1989). This article is significantly different from these in that we use a purely mathematical analysis of systems of individual cells in the visual cortex, rather than computer simulation, to derive results on the behavior of numbers of cells on the order of 106 , and synapses on the order of 1010 , which are typical numbers for a small patch of cortex. While not wishing to downplay the importance of simulation, we believe that at this stage in the development of our knowledge of the cortical visual system, a global analytic theory is badly needed, and it is this large gap that we are attempting at least partially to fill. In the context of existing neural models, ours may be viewed as combining an analog dynamical system model of the type popularized by Hopfield (1984) and the canonical cortical circuit of Douglas et al. (1989) (cf. also Douglas & Martin, 1990, 1992). Roughly speaking, our model makes the former much more biological and the latter much more specific, at least with regard to layer 2/3 of primary visual cortex. Finally, it suggests a dynamics very different from the (constrained)
Computing with Self-Excitatory Cliques
39
gradient descent normally employed in neural networks (cf. Hummel & Zucker, 1983; Hopfield, 1984). 3 A Mathematical Description of the Model 3.1 Self-Excitatory Sets. We have previously described (Miller & Zucker, 1992) a version of the well-known recurrent analog network discussed in Hopfield (1984) with continuous piecewise-linear amplifiers rather than smooth sigmoids. We showed that the relative simplicity of this version can be useful in computational analysis, while not reducing the plausibility of the model with respect to real neurons. In fact, as we have already implied in Figure 5 (bottom), piecewise-linear amplifiers are in general much more realistic than smooth sigmoids for empirically based input-output relations at the neuronal level (cf. also McCormick et al., 1985; Calvin, 1978). This is hardly surprising since for all practical purposes they are a much more general class of functions (see Miller & Zucker, 1992, Figure 6). We shall continue using this kind of analog network here, and familiarity with our earlier article is helpful. Let ui and Vi be interpreted as the input and output voltages of the (instantaneous) amplifier i described by a monotonically increasing piecewise-linear function gi (ui ) : [αi , βi ] → R given by
gi (ui ) =
0 γ i,1 ui + δi,1 u + δi,ω(i) γ i,ω(i) i 1
.. .
ui < αi,1 αi,1 ≤ ui ≤ βi,1 (3.1) αi,ω(i) ≤ ui ≤ βi,ω(i) ui > βi,ω(i)
where αi < αi,1 < βi,1 = αi,2 < · · · < αi,ω(i) < βi,ω(i) < βi and γi,k = [gi (βi,k ) − gi (αi,k )]/[βi,k − αi,k ] δi,k = gi (αi,k ) − γi,k αi,k for all integers k, 1 ≤ k ≤ ω(i). (ω is an index for the piecewise-linear slope component of amplifier i. The notation is based on Miller & Zucker, 1992.) In general we will choose the bounds αi and βi small and large enough so that they will not enter into the evolution of the system, so that we get the same form Hopfield (1984) gave for his equations: ci
X dui = Tij Vj − ui /Ri + Ii dt j6=i
Vi = g(ui ),
(3.2)
40
Douglas A. Miller and Steven W. Zucker
P where 1/Ri = j6=i Tij + 1/ρi . |Tij | is the conductance between the output of amplifier j and the input of amplifier i.4 As with our previous article, we do not assume the connections Tij to be symmetric. However, consistent with the assumption stated in the introduction, we shall assume all Tij ≥ 0. For convenience but without loss of generality we will also assume αi,1 > 0 for all i. This implies that the zero state, in which for all i, ui = 0, is asymptotically stable (cf. Hirsch & Smale, 1974) whenever the bias terms Ii are zero. Let S be a nonempty set of amplifiers. Suppose there is an asymptotically stable unbiased equilibrium of equation 3.2 in which the amplifiers S have output 1, and all other amplifiers have output 0. We shall call such an S a selfexcitatory set. Assuming nonsingularity of the system and nondegeneracy of solutions, we can make the following statement about S: Proposition 1. Let S be a self-excitatory set for the system (see equation 3.2) and suppose that for all amplifiers i, and ui such that αi,1 ≤ ui ≤ βi,ω(i) gi (ui ) ≥ γi ui + δi ,
(3.3)
where γi = 1/(βi,ω(i) − αi,1 ) and δi = −αi,1 /(βi,ω(i) − αi,1 ). Suppose a bias Ii > αi,1 /Ri is applied to each amplifier i ∈ S, and a zero bias is applied to all other amplifiers. Then the system will evolve from the zero state to an asymptotically stable state such that Vi = 1 for all i ∈ S, and Vi = 0 for all i 6∈ S. Basically the proof shows how the system, viewed as a system of differential equations, can bring each neuron in the clique up to maximum potential and that this state is stable. Equation 3.3 is a single-slope version of the multislope equation 3.1 used to simplify the proof; the general case follows directly. Proof.
Assume first that
gi (ui ) = γi ui + δi
(3.4)
for all i (cf. the solid line in Figure 5, bottom). The assumption that all Tij are nonnegative implies that if the system is in a state at time t0 such that dui /dt ≥ 0 for all i, then for all future times t > t0 , we have dui /dt ≥ 0, or in other words, all ui (t) are monotonically increasing. Furthermore, since all the amplifiers not in S start with zero output, and hence receive input only from members of S, they will, by the definition of S, continue to have zero 4 In general, we shall assume the scale of input to output is on the order of mvolts to volts (cf. Figure 5, bottom), so that we can assume Ri ≈ ρi , and hence the time constant of each cell is independent of the connection strengths.
Computing with Self-Excitatory Cliques
41
output. After some finite time t, for each i ∈ S, the capacitor ci will have charged, and we will have αi,1 < ui ≤ βi,ω(i) . Because of the monotonicity of the ui (t), each ui that reaches βi,ω(i) will stay there. Therefore, at time t, letting S˜ be the subset of S that have not already reached maximum output, equation 3.2 will be equivalent to a system of linear ordinary differential ˜ equations such that for all i ∈ S: X X X dui = Tij γj uj − ui /Ri + Ii + Tij δj + Tij . (3.5) ci dt ˜ ˜ ˜ j∈S−i
j∈S−i
j∈S\S
(The product with voltage = 1 term in the last sum is not shown explicitly.) With regard to the system in equation 3.5, there are two possibilities: either ˜ or it does not. it evolves to a point where ui ≥ βi,ω(i) , for some i ∈ S \ S, In the latter case, by monotonicity of the Q ui (t), it must reach an asymptotically stable equilibrium uˆ within the set i∈S˜ [αi,1 , βi,ω(i) ). Let Tii = −1/Ri and assume the matrix [Tij ]i,j∈S˜ defining equation 3.5 is nonsingular. Then the coordinatewise solutions ui (t) of equation 3.5 are, up to translation by the constant uˆ i , linear combinations of functions of the form tk eta cos(bt) and tl eta sin(bt) where l and k are nonnegative integers and a+bi is an eigenvalue of [Tij ]i,j∈S˜ . Assuming the nondegenerate case in which all eigenvalues affect each solution, we must have a < 0, or in other words, uˆ is the unique asymptotically stable equilibrium of equation 3.5. Now consider equation 3.5 in ˜ By the definition of S we know the state u where ui = βi,ω(i) , for all i ∈ S. equation 3.5 can evolve only in a nondecreasing fashion from u, and since uˆ is the unique asymptotically stable equilibrium for equation 3.5, ui (t) must actually increase with t for some i, and then decrease again as it moves toward ui . This is impossible since ui (t) is monotonic. We therefore have that ˜ ui (t) reaches βi,ω(i) . Iterating the proof, we see that eventually for some i ∈ S, S˜ will be empty and the proposition is proved for gi (ui ) = γi ui + δi . For the more general form (see equation 3.1) of gi (cf. the dashed line in Fig. 5, bottom) we simply observe that each Vi (t) is bounded from below by the value of gi (ui ) given by equation 3.4 so that if, in the latter case, at some time Vi = 1, this will be true for the value of gi (ui ) given by equation 3.3. On the other hand, if for the case of equation 3.4, Vi (t) = 0, for some i, this will continue to be the case for equation 3.3, since in both cases in the equilibrium state ui < αi,1 , and hence gi (ui ) = Vi = 0. For each amplifier i (not necessarily in S) let S(i) be the set of all j ∈ S such that Tij > 0. Also let Tmax (Tmin ) be the maximum (minimum) over all Tij . It follows that for each i ∈ S, ci
X dui = Tij Vj − βi,ω(i) /Ri ≥ 0; dt j∈S(i)
(3.6)
42
Douglas A. Miller and Steven W. Zucker
hence |S(i)| Tmax ≥ βi,ω(i) /Ri ; hence ¨ § |S(i)| ≥ βi,ω(i) /Tmax Ri ≡ ξil .
(3.7)
Conversely, if ¨ § |S(i)| ≥ βi,ω(i) /Tmin Ri ≡ ξiu , then we must have i ∈ S. Similarly, for each i 6∈ S, we must have ci
X dui = Tij Vj − αi,1 /Ri ≤ 0; dt j∈S(i)
hence, ¦ ¥ |S(i)| ≤ αi,1 /Tmin Ri ≡ ηiu ; and conversely if ¦ ¥ |S(i)| ≤ αi,1 /Tmax Ri ≡ ηil ,
(3.8)
then we must have i 6∈ S. We state these results as a proposition: Proposition 2. Let S be a self-excitatory set. Then i ∈ S implies |S(i)| ≥ ξil , and |S(i)| ≥ ξiu implies i ∈ S. Also i 6∈ S implies |S(i)| ≤ ηiu , and |S(i)| ≤ ηil implies i 6∈ S. A self-excitatory set will be called minimal if no proper subset is also a self-excitatory set. For a simple example of a nonminimal self-excitatory set, note that the union of two distinct self-excitatory sets is necessarily contained in such a set. On the other hand, Proposition 2 immediately gives us an important class of minimal self-excitatory sets. This corresponds to the case where, in terms of its nonzero connectivity, S is a complete graph or clique on at least ξ u + 1 amplifiers, such that for all i 6∈ S, |S(i)| < ηl , where ξ u is the maximum over all ξiu , and ηl is the minimum over all ηil . More T generally we can let S = ki=1 Si , where each set of amplifiers Si is completely connected, and for certain of the pairs (Si , S j ) such that i, j ∈ {1, . . . , k}, T j i S is completely connected as well. The latter type of self-excitatory set S could be distributed over a larger cortical surface than perfect cliques since
Computing with Self-Excitatory Cliques
43
it would not be necessary to connect the most distant cells corresponding to the most distant pairs (Si , S j ). However, we shall base our analysis for the remainder of this article on pure cliques. With respect to a clique of actual neurons, it of course seems unlikely that a perfectly symmetric arrangement could arise in cortical development, and in fact the connectivity does not have to be perfectly symmetric for our analysis to hold. For example, with the clique sizes discussed in the next section, 10 percent of connections could be randomly omitted, with the primary negative effect on performance being to raise the self-sustaining activation level ξiu by about that percentage. 3.2 Storage and Input Bounds for Random Data. Suppose we wish to store some collection C of randomly chosen self-excitatory cliques of amplifiers, each of size M ≥ ξ u + 1. The question we wish to address is how large we can reliably allow |C| to be and how many extraneous amplifiers we can allow to be biased in the process of retrieving a member of C. This amounts to evaluating the probabilities of two types of error events, following an attempt to activate a given clique Cp ∈ C by biasing a set of amplifiers E sufficient to saturate Cp . E will not be independent of Cp , but we shall assume that E \ Cp is independent of the other Cq ∈ C − Cp . Also we shall assume that formation of a clique that connects amplifiers j to i does not change the value of an existing connection Tij > 0. Error type 1 occurs when there exists an amplifier i 6∈ Cp with an input bias from Cp that causes it to have a significant nonzero output. A necessary condition is thus X Tij Vj > αi,1 /Ri . j∈Cp
However, observe that for a response curve such as the dashed line in Figure 5 (bottom) with a large initial slope, requiring X Tij Vj ≤ αi,1 /Ri j∈Cp
is unnecessarily stringent, since the response has the same nonsaturated level for a bias considerably larger than αp,1 /Rp . In terms of the top and middle graphs of Figure 5, we get just a single spike in a 25 msec time interval for any bias between .3 and .6 nA. Thus, the level of bias that we shall view as a type 1 error can be more reasonably taken as X Tij Vj ≥ βi,ω(i) /2Ri . (3.9) j∈Cp
Similarly to equation 3.8, we define ¨ § ηi ≡ βi,ω(i) /2Tmax Ri ,
(3.10)
44
Douglas A. Miller and Steven W. Zucker
which is the least number of nonzero connections that could cause amplifier i to have an input ≥ βi,ω(i) /2, and let η be the minimum over all ηi . Error type 2S occurs when at least one of the cliques Cq ∈ C − Cp can be saturated by E Cp . Observe that errors type 1 and 2 produce unwanted effects that persist after the bias E that saturates Cp is removed. It is only these persistent effects that we shall be interested in, since the state of the system during the biasing by E is not otherwise relevant to the final computation. Of course, the connections that form the cliques in C can inadvertently form new self-excitatory sets, by extending members of C or creating entirely new self-excitatory sets that contain no member of C as a subset. However, we shall arrange for the expected number of type 1 errors to be sufficiently small that the probable number of amplifiers involved in either of these scenarios will be small, and thus may be neglected. Thus, we shall assume that with Cp saturated, the only remaining self-excitatory sets that E \ Cp could activate are those in C. To estimate the probability of a type 1 error, note that, with respect to equation 3.9, this event can occur only if i and at least ηl such j are in the same clique. That i and any other random amplifier j are in the same clique has probability ¶ µ M (M − 1) |C|−1 . ρ =1− 1− N (N − 1)
(3.11)
If M is small relative to |C|, the probability of obtaining at least ηˆ such matches for j ∈ Cp will be given approximately by the binomial distribution, that is, M X
C(M, k)ρ k (1 − ρ)M−k .
(3.12)
k=η
Let Xi be the indicator (0 or 1) random variable for amplifier i 6∈ Cp receiving the input (see equation 3.9) from Cp , and let E() denote expectation. Then since expectations add the total expected number of such amplifiers is ! Ã X X X Xi = E(Xi ) = Prob{Xi = 1}. (3.13) E i6∈Cp
i6∈Cp
i6∈Cp
Thus, since Prob{Xi = 1} is approximately equation 3.12, requiring (3.13) to be ≤ ² is approximately equivalent to (N − M)
M X k=η
C(M, k)ρ k (1 − ρ)M−k ≤ ²,
(3.14)
Computing with Self-Excitatory Cliques
45
which, solving as an equality for ρ, gives us an upper bound ρ² ≤ ρ. Alternatively we can use the Poisson approximation à (N − M) 1 − e
−ρM
η−1 X (ρM)k k=0
!
k!
≤ ².
From equation 3.11, we observe |C| ≤
ln(1 − ρ² ) ³ ´ + 1, ln 1 − M(M−1) N(N−1)
where ρ² is the solution to equation 3.14. If we choose ² sufficiently small, say ≤ 1 (cf. Willshaw et al., 1969, p. 961), the binomial distribution with a large number of trials and small success probability per trial becomes a reasonable approximation, and we may in turn approximate the latter with the Poisson distribution, so that the probability of k type 1 events is exp(−²)² k /k! and in particular the probability of no type 1 events is exp(−²). Now let us consider the probability of a type 2 error, assuming ² ≤ 1 as above. Since the probable number of type 1 errors is being kept small, we shall neglect them and simply assume, using as an upper bound the binomial distribution in equation 3.12, that each of the N − M amplifiers outside Cp receives a bias from Cp and the other biased amplifiers E, which is ˆ ≤ Tmax (ρ² (M + |E|/2)) ≡ I. Adding the bias Iˆ to the left-hand side of equation 3.6, we derive a new version of equation 3.7, namely »µ |S(i)| ≥
¶ ¼ ¼ » βi,ω(i) βi,ω(i) ˆ − I /Tmax = − ρ² (M + |E|/2) ≡ ξi , Ri Tmax Ri
or in other words, neglecting ceilings, ξi = ξil − ρ² (M + |E|/2). Let ξ be the minimum over all ξi . Then the probability of a type 2 error for arbitrary Cq ∈ C − Cp , using the hypergeometric distribution, is ≤
M X C(|E| + M, k)C(N − |E| − M, M − k) k=ξ
C(N, M)
.
46
Douglas A. Miller and Steven W. Zucker
Thus, if ²ˆ is a desired bound on the probability of a type 2 error, we can set |C|−1 M X C(|E| + M, k)C(N − |E| − M, M − k) = ²ˆ , (3.15) 1 − 1 − C(N, M) k=ξ and solve numerically for |E|. As opposed to ², an expected value that we could plausibly leave at about 1, ²ˆ is a probability that should be small, since setting off the wrong clique ruins the computation. Exactly how small ²ˆ should be would depend on how sensitive higher-level mechanisms would be to an error. Biologically, one would expect some tolerance, so we will take ²ˆ as 10−4 . In fact the specific choice of ²ˆ is not very critical, since the probability of a type 2 error goes effectively from zero to one within a relatively narrow range of |E|. In order to give some computed examples, we need to specify or provide ranges for the additional parameters. We shall assume in all our examples that the Tij are independently distributed around a mean Tµ such that (M/2)Tµ = βi,ω(i) /Ri for all i, with standard deviation σ = Tµ /2. Thus, biasing above threshold M/2 members of a clique, all of whose members have connections Tµ , will be just sufficient to activate the clique. Since the average √ output of n random amplifiers will then be Tµ with standard deviation σ/ n, we can, in order to get a reasonably tight √ lower bound on the maximum values of |C| and |E|, take Tmax as Tµ + σ/ n. Then substituting in equation 3.7 gives us · µ q ¶¸ βi,ω(i) 1 βi,ω(i) 2 + / ξil ) = ξil , βi,ω(i) / Ri Ri M Ri M q which solving as a quadratic (in ξil ) gives us q
ξil
=
−1 +
√ M 1 + 8M ⇒ ξil ≈ . 4 2
Similarly from equation 3.10, we have √ −1 + 1 + 4M M √ ⇒ ηi ≈ . ηi = 4 4 With regard to the size of N, we provide a range of 10,000 to 400,000. Note that each clique takes M2 connections, and with our choice of ², each connection will generally belong to just one clique, so we must have M2 |C| ≤ Nf , where f is the average dendritic fan-in, and hence |C| ≤ Nf/M2 .
(3.16)
Taking the dendritic fan-in for a pyramidal cortical cell of 6000 (Douglas & Martin, 1990) as a value for f , we obtain the graphs for |C| and |E| in Figure 6, top and bottom, respectively.
Computing with Self-Excitatory Cliques
47
4 Application: Hyperacuity-Scale Representations of Lines in the Visual Cortex We now proceed to show, using the cortical model described in the previous sections, that groups of pyramidal cells in human and monkey primary visual cortex are capable of representing short line contours on hyperacuity scales. By“hyperacuity” we mean specifically the ability, under optimal circumstances, to discriminate straight line contour orientations an order of magnitude better than our ability to resolve two stars (or other points of light), when we are comparing the displacement of the end points of the contour with the separation of the stars (see Figure 8).5 For the sake of this analysis, we assume that pyramidal cell counts in monkey and human visual cortex are comparable. To carry out this analysis we view each line contour as an item of information in a database consisting of subsets of cortical neurons. Hyperacuity can then be interpreted as the ability to store a certain amount of information in this database corresponding to an extremely large number of such subsets. The question then becomes whether the brain is equipped to do this. Our answer is in the affirmative, under the assumption that excitatory cliques of S cells exist, and leads to the surprising result that cliques are smaller than one might expect on intuitive grounds. In the end, we argue that cliques should consist of about 33 cells. 4.1 On the Existence of Excitatory Cliques of S Cells. We begin by expanding on points raised in the introduction. The connectivity described by Rockland and Lund (1982) and Rockland et al. (1982) in tree shrews, Rockland and Lund (1983) in monkeys, and Gilbert and Wiesel (1983, 1989) and Callaway and Katz (1990) in cats seems ideal for connecting groups of cells that respond to approximately the same contour orientation. The result is a visual field significantly greater than the individual receptive fields. We illustrate this in Figure 4, which shows a thin moving slit of light and three rectangular receptive fields (cf. Figure 3, bottom) whose corresponding cells each simultaneously responds to one edge of the slit (cf. Figure 3, middle, and Figure 1, bottom). Thus, the connectivity requirement would seem to be met. Within this connected clique, how do we know that such a group of S cells can respond, as a group, to a specific contour? In particular, how do we know, first, that it is S cells (Henry, 1977; Bishop et al., 1971, 1973, 1972; Hubel & Wiesel, 1962) that are being interconnected, and second, how do 5 A prior conjecture has been put forth by Barlow (1979) that the basis for hyperacuity lies in sufficiently large populations of granule cells in primary visual cortex. However, he did not propose a network model, and based on the network model described here, granule cells seem much less likely candidates than pyramidal cells, due to their shorter range and lower degree connectivity.
48
Douglas A. Miller and Steven W. Zucker
Figure 8: Comparison of normal acuity and hyperacuity. (Upper left) Two point light sources such as stars separated by 1 min of arc correspond at the human fovea (lower left) to a separation of the two peak image intensities of two cone spacings. Thus, both theoretically and empirically 1 min of arc is the smallest resolution of two points of light. However, for a thin line contour 0.5 degree in length (right), we can detect a change in orientation of as little as 1/6 degree, representing a relative displacement in the end points of only 6 sec of arc, or 1/5 of an average cone separation (cone represented as rectangle) (lower left) and thus far exceeding normal resolution limits. This is an example of hyperacuity. (Adapted from Westheimer, 1990)
we know that these cells are responding in a precise spatiotemporal fashion to moving stimulus contours? With regard to the first question, we must consider both substantive and (for historical reasons) nomenclature issues. Both Henry et al. (1979, Figure 2A) and Martin (1984, Figure 13) have found that the majority of identifiable superficial layer cells in cats are either type S, or type S end stopped (SH ). Again, while it is tempting to identify S cells (Henry, 1977) with simple cells (Hubel & Wiesel, 1962), one must be careful (cf. Gilbert & Wiesel, 1979; Martin, 1984); we stress that our focus on S cell types, rather than on the simple cell type, is the spatiotemporal aspect of the definition.
Computing with Self-Excitatory Cliques
49
Schiller et al. (1976a) have found a substantial proportion of the identifiable cells in superficial macaque layers to be of this type (which they also call “S”), as well as of another larger type they identify as “CX.” However they observe that these superficial layer CX cells have the receptive field size, orientation tuning, end stopping, binocularity, and low spontaneous spiking more typical of their S cells. A major reason for not putting these cells in the S category is the inability to distinguish a relatively large spatial separation between the cell’s edge-response regions. From our point of view, such a criterion is significant, and if this were the only difference, we would call these S cells. We note Schiller et al. (1976a, Fig. 13) as an example of this type of cell, which, if one were using the separation criteria described in Figure 3 instead of gaussian fitting, could have been classified as an S cell. Thus, S cells appear to exist in layer 2/3; many of differences in the literature may be largely nomenclature. As for the second question, we know that hyperacuity phenomena occur in the presence of significant motion of 3 degrees per second or more, and indeed some degree of motion can actually improve hyperacuity (Westheimer & McKee, 1975). Given this tolerance to motion, the precise spatial and temporal response acuity cited above to moving-edge contours would appear to be a basis for a highly specific group response to these stimuli, provided one could count on a certain continual contour motion. This is certainly the situation, at least for humans, whose eyes experience ongoing motions due to tremor, drifts, and microsaccades (Alpern, 1962). Given that a highly interconnected group or clique of S-type pyramidal cells can exist and can respond as a group to a highly specific stimulus contour, what is the dynamic behavior of this group that makes its presence known to the rest of the brain and distinguishes it from mere noise? First, we recall that such cells typically exhibit the adaptive spike train behavior in Figure 2 and further note that the early interspike intervals tend to vary systematically and dramatically with the input current (see Figure 5, top). From the point of view of each member of a clique, that cell’s own behavior is determined not by its own sparse intermittent spiking, but by the output of the rest of the clique. To enable calculation, and for reasons that will become clear shortly, suppose there are 33 cells per clique. If we assume simultaneous first spikes followed by Poisson interarrivals for the second spike, the clique would produce the incoming spike rate for each cell in the first 3.5 msec as given in Figure 5 (middle). This kind of input-output relation, with its relatively high and randomized group spike rate and short time interval in relation to each neuron’s time constant of 10 to 20 msec (Douglas & Martin, 1990), permits an analog approximation to the input current and the assumption of an approximately linear currentvoltage relationship for each cell. We therefore can model each cell with a purely analog, piecewise-linear input-output voltage amplifier as in Figure 5 (bottom).
50
Douglas A. Miller and Steven W. Zucker
We have discussed such piecewise-linear amplifiers in previous work (Miller & Zucker, 1992), where we showed that they have certain computationally attractive properties for modeling nonsymmetric analog recurrent networks. We show in separate work (Miller & Zucker, in press) that these results imply that the cliques we are interested in can be modeled and computed in a manner that scales up efficiently with the size of the network. The main point here, however, is that excitatory cliques of such amplifiers tend to exhibit a dynamic behavior that drives each member of the group to saturation response. Given that cells in primary visual cortex are normally quiescent, we can assume that a substantial number of cells in superficial layers responding within a single 25 msec interval at saturation levels would be a sufficiently unlikely event to identify this group of cells reliably as an excitatory clique responding to a specific class of visual input. It is important in this model of self-excitatory neuronal cliques that each clique member, under a typical lower-level afferent stimulation, be capable of responding with at most one or perhaps two spikes within a temporal precision of a few msec. The occurrence of such spikes in a large enough percentage of a given clique’s neurons constitutes in our theory the match that ignites the entire clique to saturation firing. However, the brain must be able to distinguish the match from the conflagration. Thus, it is important that lower-level afferent stimulation produce a typical spike train that is not characteristic of saturation firing. Notice that in Figure 5 (top), we can describe the single spike response (within a given 25 msec interval) as corresponding approximately to an input current of ≤ 0.7 nA, whereas saturation response is reached at about twice that input level. Thus, this kind of cell appears to have a dynamic range of response perfectly suitable for distinguishing between single spike and saturation response. Furthermore, we note that a typical one- or two-spike response of layer 3 pyramidal cells to LGN afferents has been observed both empirically and with compartmental models (Douglas & Martin, 1992). (See Konig ¨ & Engel, 1995, for a more general review of relevant data.) 4.2 An Analog Random Storage Model: Storage Capacity and Resistance to Noise. We now consider the question of whether enough selfexcitatory cliques in the primary visual cortex can be stored to represent lines on hyperacuity scales. Additionally, we can determine how resistant the resulting system would be to noise. We begin by considering an idealized model in which cliques of a specified size M are chosen at random from the total population of neurons and then completely interconnected. The interconnection (i.e., synaptic) strengths are themselves independently distributed around a mean Tµ such that M/2 members of this clique at full output would be just sufficient to maintain another member of the clique at full output, given that the connection strengths to this other member were all Tµ . The standard deviation of the connection strength is chosen to be half the mean value.
Computing with Self-Excitatory Cliques
51
Figure 6 (top) gives lower bounds, calculated in section 3.2, on the number of storable randomly chosen cliques for four clique sizes M, as a function of the neuron population N. These bounds are designed to ensure that the expected number of model neurons inadvertently saturated by an activated clique is ≤ 1. Observe that eventually curves for all values of M become the linear function Nf/M2 , as all f available synapses per cell are used up (we assume f = 6000 for pyramidal cells [Douglas & Martin, 1990]). Thus, the curve for M = 60 reaches its linear bound at a much lower value (N ≈ 100,000) than the curve for M = 25 (N ≈ 350,000). On the other hand, the curve for M = 15 does not even come near its linear bound on this graph, owing to the necessarily high level of excitability of a system of very small cliques that can maintain themselves at saturation response. Observe also that even for M = 25, about the maximum in terms of ratio of cliques to neuronal population over the range considered (N ≤ 400k), the linear bound is reached at about N = 350,000, which corresponds to an area of primary visual cortex of about 3 mm2 in monkeys, and about twice that in cats and rats (Peters, 1987). At this point this ratio is about 9.6 for M = 25. Similarly for M = 33 the maximum ratio is reached at about N = 290,000, and has a value of about 5.5. In Figure 6 (bottom) we consider, for the same model and clique sizes given in Figure 6 (top) lower bounds on the number of randomly chosen additional model neurons that could be given an impulse current sufficient to produce a single spike (corresponding to lower-level afferent input), without inadvertently setting off a clique in addition to the one targeted. More specifically, we wish to keep the probability very low for such an event. Details of the calculations were given in section 3.2. We can view these additional randomly activated model neurons as noise, and therefore the number of such model neurons that are allowable as the system’s tolerance to noise. Observe that, for example, with M = 33, the noise tolerance is in general at least 20 times greater than the clique size. 4.3 Mapping from a Region of Visual Field to a Region of Cortex. We suppose now that the brain is trying to store in a (6 mm)2 region Ac of primary visual cortex local information about line contours within a visual field Avf . Portions of these regions are illustrated in Figures 3 and 4, respectively. Assuming a human foveal magnification of 15 mm/deg, there will be a mean shift in receptive field location of the cells of Ac by about 24 arc minutes, traversing its width. The receptive fields for cells in Ac will, however, have a mean width of about 12 arc minutes (Wilson et al., 1990, Fig. 9) so that we may take Avf to be approximately (36 arc minutes)2 . In fact Avf could be taken as somewhat larger due to a radial scatter of about half the receptive field size in either direction (cf. Albus, 1975; Hubel & Wiesel, 1977), but this would result in a rapid drop-off in density of receptive field coverage of Avf toward the borders. Note that Ac will include about the widths of four human orientation
52
Douglas A. Miller and Steven W. Zucker
columns, assuming human iso-orientation patterns are similar to the macaque (e.g., Ts’o, Frostig, Lieke, & Grinvald, 1990) and using the scaling factors for human ocular dominance columns (Horton, Dagi, McCrane, & de Monasterio, 1990). Thus we may assume that the receptive fields of the S-type cells in Ac have uniformly distributed orientations in Avf . Note, however, that the receptive field centers, although radially symmetric, will be somewhat more concentrated toward the center due to the radial scatter. 4.4 Calculation of Cortical Packing Density from Hyperacuity Data. We shall now calculate both the clique size M and the cortical packing density of S-type pyramidal cells necessary for a sufficient number of cliques to perform hyperacuity-scale discriminations of straight line orientations. We shall assume that all available synapses are being efficiently used in the formation of cliques, which would correspond in the above random storage model to the assumption that the appropriate curve in Figure 6 (top) has reached its linear bound of Nf/M2 . Otherwise all parameter assumptions are based on empirical observation. We assume the network of S-type cells is storing the cliques efficiently, that is, near its capacity of |C| cliques, each of size M, where C is the total set of cliques. Then, each S cell must, on average, belong to |C|M/N cliques. (To consider a totally inefficient arrangement, imagine each cell belonging to just one clique, so that the number of cliques is N/M ¿ |C|.) Now let us consider an S cell that belongs primarily to straight line cliques of the kind indicated in Figure 4. We assume (Bishop et al., 1971, 1972, 1973; Schiller et al., 1976a,b,c) that an S cell responds to a slit of light or a bar by responding to its edges. In general, this response is direction dependent and may strongly depend on variable and relatively nonspecific inhibition (Bishop et al., 1973). For this analysis, we shall make the simplifying assumption that each cell responds to both light and dark edges, but only in one direction.6 Thus, the distinctly shaded regions in each receptive field in Figure 4 are not to be taken as separate ON/OFF regions, but rather their borders are to be taken as the points on the axis perpendicular to orientation of maximum probability of initial spike response to a particular edge, in a particular direction of motion. For the network to be efficient, there must be |C|M/N distinct contours passing through its receptive field, skirting either of its side regions. From geometry, it is clear that the only way this can happen is for the contours to have varying orientations. In fact, for each orientation, there are 6 This was the most common class of S cells found by both Bishop et al. (1971) and Schiller et al. (1976a).
Computing with Self-Excitatory Cliques
53
just two contour positions that would put the cell at the appropriate place on its response tuning curve, one corresponding to each inhibitory flank. We therefore shall assume that each S cell belongs to exactly two cliques for each discriminable orientation of a contour within its tuning range, which in monkeys will be about 15 degrees before a steep falloff of over 10 percent in average response occurs (Hubel & Wiesel, 1968; De Valois, Yund, & Hepler, 1982, Figure 1C; Schiller et al., 1976b, Fig. 2). If h is the number of cliques (hence lines) per degree, this means |C|M/N = 2 × 15 deg × h.
(4.1)
We assume that storage is at its limiting value of |C| = Nf/M2 .
(4.2)
Martin (1984) has found that S cells in cats may be both layer 2/3 pyramidal and spiny stellate, so it is not clear a priori whether to take these cells’ respective fan-in values f of 2000 or 6000 (Douglas & Martin, 1990). However, since it is between layer 2/3 pyramidal cells that the intrinsic horizontal inter-iso-orientation area connections have been observed by Gilbert and Wiesel (1983, 1989) and Callaway and Katz (1990), it is the pyramidal fan-in figure of f = 6000 that appears most justified. Furthermore, the data from orientation hyperacuity (Westheimer, 1990) imply a value of h in equation 4.1 of 6 lines/deg. Therefore equations 4.1 and 4.2 imply h=
(Nf/M2 )(M/N) = 6 lines/deg, 2 × 15 deg
which implies M ≈ 33. Observe that this clique size is about optimal for the random model if we give equal weight to both clique capacity and noise tolerance (see Figure 6, top and bottom). This also supports the assumption made in section 4.1 about clique size. What about the size of N? Let us look at the S cell population from the point of view of a single S cell whose receptive field is centered in Avf and oriented vertically (see Figure 9). There are 15 × 6 = 90 contours passing through a given side of its receptive field. Although there may be some overlap between the corresponding orientation cliques of any two of these, N must be chosen so as to keep such an overlap to less than half. Thus, if we consider the 1.58 arc minute window, 6 arc minutes above the center line, through which all these contours have to pass, and which is 36/33 arc minutes deep (see Figure 10), there must be, on average, at least 90
54
Douglas A. Miller and Steven W. Zucker
Figure 9: Orientation hyperacuity and cortical cell density. A model S cell with an orientation tuning width of 15 degrees, whose receptive field is at the center of the visual field represented here, belongs for each edge-response location (vertical lines within rectangular receptive field) to 90 cliques of cells representing 90 line orientations, based on an orientation hyperacuity of 1/6 degree (see text for calculation). Each line clique consists of M receptive fields (shown here for one orientation). The angle has been exaggerated for display purposes.
distinct and appropriately oriented S cell receptive fields to accommodate them. Here we assume that 6 arc minutes above center represents a mean horizontal receptive field density and that M = 33 as derived above. This gives us a receptive field density of 90/(1.58 arc min × 36/33 arc min) = 52.2/(arc min)2 . Allowing for the 12 other orientation ranges to accommodate 180 degrees, we get 626.4 receptive fields per (arc min)2 or 812,000 for all of Avf . Translating this back to Ac gives us a minimum sufficient density of 22.6k S cells per mm2 of foveal cortex. Allowing for enough cells to accommodate the opposite direction of motion as well, we would need to double the cell count to 45.2k cells per mm2 . In comparison we note that the number of layer 2/3 pyramidal cells that have been estimated for macaque monkey primary visual cortex (Peters &
Computing with Self-Excitatory Cliques
55
Sethares 1991, Fig. 19, and Table 1, and assuming 10 percent GABAergic cells), is (35 cells/cluster) × (1271 clusters/mm2 ) ≈ 44,500 cells/mm2 . 5 Discussion We have presented a model of cortical computation that is sophisticated enough to model a range of biological events in the visual cortex, but simple enough to be subject to mathematical analysis. In any such undertaking, there will invariably be modeling simplifications that have important consequences and parameters whose specification cannot be determined more than approximately by existing objective measurements. Several of these exist here. In particular, because of the focus on orientation hyperacuity at a point, we did not consider the question of nearby cliques in the direction orthogonal to the bar orientation. The analysis concentrated on the orientation variability and the effects of length; it remains to be seen whether this additional analysis will keep the cell counts consistent. Regarding parameters, the chosen area Avf , the 6 arc minutes vertical displacement chosen in Figures 9 and 10, the assumption that each S cell responds to exactly two edges, and the length of the contour to which the cliques are responding could, of course, have been chosen somewhat differently. Somewhat different results would then have ensued. However, we have attempted to choose parameters that seem representative, given the existing empirical data and species differences. What is remarkable about the outcome of this procedure is that a prediction of monkey cortical cell packing densities could be derived for any reasonable choice of parameters that agreed to within an order of magnitude or two. Our result of 45,200 predicted versus 44,500 observed should clearly be taken as a first example calculation; no doubt more complete models and more careful estimates will be possible in the future. Nevertheless, we believe it provides an argument in support of the class of clique-based computations proposed. Our model addresses the question of how information can be represented in the brain in a way that is consistent with known anatomical, physiological, and biophysical facts. It provides a clear description of how the brain could go about representing great masses of information—in this case, pertaining to the orientation of lines and visual hyperacuity—in a way that at the same time implies a high degree of reliability from inherently unreliable units, that is, neurons. Additionally our model says how this can be done in a time scale, 25 msec, that we know is necessary in order to make lower-level processing available sufficiently quickly to the rest of the visual system. None of these features is addressed by a theory based on a vector averaging of long-term cortical cell responses. Furthermore, our model makes the testable prediction that there exist cliques of layer 2/3 pyramidal cells in human, monkey, tree shrew, and
56
Douglas A. Miller and Steven W. Zucker
cat primary visual cortex that are close to being completely interconnected synaptically and can drive themselves to saturation feedback responses within periods of approximately 25 msec in response to small lines or edges moving in a specific direction perpendicular to orientation. Furthermore, the model predicts that in monkeys and humans, the number of such cliques is several times larger than the actual population of cells from which they are formed and that the average size of these cliques is about 33. Let us consider the reliability issue further. There are intimate connections between this work and traditional reliability theory (cf. Winograd & Cowan, 1963). Roughly speaking, we are suggesting that line detection on hyperacuity levels consists of activating a K-out-of-M or quorum component system (Moore & Shannon, 1956). In this context, we can view each neuron as a component and each clique as a system. As the appropriate contour (or a corrupted version) passes through the visual field, a neuron has a certain probability p > .5 + ² of spiking at least once, whereas in the absence of such stimulation, the neuron’s probability of spiking is p < .5−². In Figure 6 we give the probability for a 17-out-of-33 clique to respond as a function of its component neurons’ probabilities of firing. In effect, the clique transforms a group of highly unreliable response units (small ²) into a single reliable system. The 17-out-of-33 system is just one possible example of a K-out-of-M clique, although we described in the section 4 our reasons for choosing M = 33. Our theory does not imply a specific value for K, and indeed, in deriving the lower bounds on number of cliques and tolerance to noise in Figure 7, we have assumed that K is at least M/2, although smaller values are possible. Obviously, however, K must be substantially less than M if the system is to improve the reliability of individual cortical neurons. In any event, the model implies that the number of excitatory postsynaptic potentials from superficial-layer pyramidal cells needed to produce an action potential in another such cell is less than about 33, and probably closer to 17. These numbers are low compared to what is necessary, for example, to elicit action potentials in spiny cells in the hippocampus (McNaughton, Douglas, & Goodard, 1978), but consistent with estimates for activation of layer 4 S cells being directly innervated by the LGN (Bullier, Mustari, & Henry, 1982), given that each LGN afferent in general forms only one synapse for each target cell (Freund, Martin, Somogyi, & Whitteridge, 1985). The degree of lateral precision implied by orientation hyperacuity (cf. Figures 9 and 10) is sufficient to account for both standard vernier acuity, and the two-dot variety demonstrated by Westheimer and McKee (1977). Interestingly, the two-dot phenomenon, as Westheimer and McKee remark, could also be explained by an orientation model capable of detecting incomplete or implicit lines, which our model is capable of doing. In terms of orientation discrimination, the optimal separations for outsides of the two dots were about 6 arc min, and for the end points of the complete lines were about 6 arc min or higher (Westheimer, 1982, Fig. 1). It is therefore
Computing with Self-Excitatory Cliques
57
Figure 10: Completion of the density calculation begun in Figure 9. In order for the line cliques to have enough cells to fill them without excessive overlap between any two cliques, there must be about 90 receptive fields whose centers are within the region 36/M arc min high (shaded) by 1.58 arc min wide (between labeled arrows), and which is located about 6 arc min above the center (dashed line). From these numbers, we can calculate an overall cortical density for this kind of cell (see the text).
interesting to note that our model, together with the receptive field layouts illustrated in Figures 9 and 10, implies that as the length of the contour is reduced and its end points brought closer together, the maximum number of distinct cliques whose receptive fields would be approximately centered at these end points is also proportionately reduced. This in turn implies that the maximum line orientation discrimination ability (in degrees of rotation) is proportionately reduced (since angles are small), given that the lateral discrimination of the end points remains constant. Our model predicts that these relations should hold down to a point determined by the edge-response characteristics of the receptive field, at which performance deteriorates. This in fact seems consistent with what has been observed (Westheimer, 1982, Fig. 1, bottom; Westheimer, 1990). We do not expect the visual cortex to be able to form perfectly interconnected cliques, nor is it clear that this would even be desirable from an
58
Douglas A. Miller and Steven W. Zucker
efficiency viewpoint. Many kinds of incompletely connected self-excitatory groups of neurons could exist that would span considerably larger areas of cortex than a totally connected clique could, and we have described a general class of these. Furthermore a significant percentage of connections from any clique could be randomly removed with the only possibly negative effect being to increase the number of spikes needed for activation of the clique. The implications for learning (e.g., Lowel ¨ & Singer, 1992) and for fast synaptic facilitation deriving from this model should also be mentioned (Varela et al., 1997; Fischer, Zucker, & Carew, 1997). There are several related issues that we have not addressed in this article, perhaps the most important of which involve the relationship between spatial vision and motion. All of the constructs were based on slowly moving visual stimuli of the type that are natural for S cells. One way to take this model into the motion domain is to postulate a sequence of cliques at nearby positions that fire in temporal sequence. This seems natural for slowly moving configurations, in which the additive inhibition or each cell’s biophysics would bring each clique down to quiescent, and an adjacent clique would ignite. To place some rough numbers on this, we might calculate that (according to the data in Figure 3 and the model in Figure 5) the stimulus moves about 3 minutes in the time it takes for the clique to saturate. From the Westheimer (1990) constraint of 6 lines per degree, we calculate that there is about a factor of 4 before the next clique must saturate, which should be sufficent. However, what this implies for apparent motion must be checked, and whether cells with different velocity labels exist must be considered. Moreover, this model at high stimulus velocities is questionable, and perhaps other velocity-encoding techniques should be considered. In its most general terms, our theory suggests the necessity for a precise distinction between the individual response of a cortical neuron to lowerlevel afferent stimuli and that neuron’s response as part of a cortical group. To observe cortical cliques as functioning entities would require disentangling the two types of cell firings. Important steps in this direction have been taken by a number of the works, which we have cited, and we believe this trend should be accelerated. The rewards could be a considerably increased understanding of the visual cortex, and perhaps other parts of the brain as well. Appendix: A Glossary of Symbols Avf An area of the visual field corresponding to an area Ac of primary visual cortex. Ac An area of primary visual cortex corresponding to an area Avf of visual field. αi Lower bound on the domain of a piecewise-linear amplifier function gi (ui ).
Computing with Self-Excitatory Cliques
59
αi,j Beginning of jth piecewise linear segment of amplifier function gi (ui ). βi Upper bound on the domain of a piecewise-linear amplifier function gi (ui ). αi,j+1 = βi,j beginning of j + 1th piecewise linear segment of amplifier function gi (ui ). ci Input capacitance of amplifier i. C Total set of model neuronal cliques. |C| Number of cliques in the set C. Cp An individual clique of model neurons belonging to the total set of cliques C. C − Cp The set of cliques C minus the clique Cp . C(i, j) Number of ways in which j items can be chosen from a set of i items: “i choose j.” C(i, j) = i!/(j!(i − j)!). δi,j Constant used in the definition of the jth piecewise-linear segment of amplifier i. E Set of model neurons that is initially activated with a bias current. This corresponds to a set of cortical neurons that spike initially in response to lower-level afferent stimulation. E(X) Expected value of the random variable X. ηil Upper bound (sufficient) condition (based on Tmax ) on |S(i)| for a model neuron i not to belong to the self-excitatory set S. ηiu Upper bound necessary condition on |S(i)| (based on Tmin ) for a model neuron i not to belong to the self-excitatory set S. ηl Minimum over all ηil . f Dendritic fan-in of a model neuron. gi ith piecewise-linear amplifier. Corresponds to a single model neuron. γi,j Coefficient used in the definition of the jth piecewise-linear segment of amplifier i. h Number of lines per degree of orientation that can be discriminated by hyperacuity. For example, if one can discriminate two line orientations 1/6 degrees apart, then h = 6. M Number of model neurons in a clique.
60
Douglas A. Miller and Steven W. Zucker
N Number of model neurons corresponding to the cortical area Ac . Ri Inverse of model neuron i’s membrane conductance plus synaptic conductance. ρ i Model neuron i’s membrane resistance. S A self-excitatory set of model neurons, that is, a set of neurons sufficiently interconnected to drive themselves to saturated feedback levels. Si Completely interconnected subset of a self-excitatory set S. S(i) Set of amplifiers j (model neurons) in a self-excitatory set S that have nonzero connectivity Tij to an amplifier i. |S(i)| Size of the set S(i). Tij Conductance from amplifier j to amplifier i. Tmax Maximum over all Tij . Tmin Minimum over all Tij . Tµ Expected value of Tij . ui Input voltage to amplifier i. Vi Output voltage of amplifier i. ξil Lower bound on |S(i)| for a model neuron i to belong to the self-excitatory set S based on the max conductance (see section 3.1). ξiu Lower bound condition on |S(i)| for a model neuron i to belong to the self-excitatory set S based on the min conductance. ξ u Maximum over all ξiu . Acknowledgments We thank Gerald Westheimer, David Jones, and Allan Dobbins for valuable discussion, David Tank for (repeatedly) stressing dynamics, and Rodney Douglas for kindly supplying the data used in Figure 2. The reviewers were extremely helpful. Research was supported by grants from AFOSR, NSERC, NSF, and Yale University. References Abbott, L., & van Vreeswijk, C. (1993). Asynchronous states in networks of pulse-coupled oscillators. Phys. Rev. E, 48, 1483–1490.
Computing with Self-Excitatory Cliques
61
Abeles, M. (1991). Corticonics. New York: Cambridge University Press. Albus, K. (1975). A quantitative study of the projection area of the central and the paracentral visual field in area 17 of the cat. Exp. Brain Res., 24, 159–179 (Part I), 181–202 (Part II). Alpern, M. (1962). Types of movement. In H. Davson (Ed.), The eye (Vol. 3, pp. 63– 151). New York: Academic Press. Amari, S. (1989). Characteristics of sparsely encoded associative memory. Neural Networks, 2, 445–457. Amit, D. J. (1989). Modeling brain function: The world of attractor neural networks. Cambridge: Cambridge University Press. Amit, D. J., Gutfreund, H., & Sompolinsky, H. (1987). Information storage in neural networks with low levels of activity. Phys. Rev. A, 35, 2293–2303. Artola, A., Brocher, ¨ S., & Singer, W. (1990). Different voltage-dependent thresholds for inducing long-term depression and long-term potentiation in slices of rat visual cortex. Nature, 347, 69–72. Barlow, H. B. (1979). Reconstructing the visual image in space and time. Nature, 279, 189–190. Barlow, H. B. (1983). Understanding natural vision. In O. J. Braddick & A. C. Sleigh (Eds.), Physical and biological processing of images (pp. 2–14). New York: Springer-Verlag. Bishop, P. O., Coombs, J. S., & Henry, H. (1971). Responses to visual contours: Spatio-temporal aspects of excitation in the receptive fields of simple striate neurones. J. Physiol., 219, 625–657. Bishop, P. O., Coombs, J. S., & Henry, H. (1973). Receptive fields of simple cells in the cat striate cortex. J. Physiol., 231, 31–60. Bishop, P. O., Dreher, B., & Henry, H. (1972). Simple striate cells: Comparison of responses to stationary and moving stimuli. J. Physiol., 227, 15–17P. Braitenberg, V. (1974). Thoughts on the cerebral cortex. J. Theor. Biol., 46, 421–447. Braitenberg, V. (1978). Cell assemblies in the cerebral cortex. In R. Heim & G. Palm (Eds.), Theoretical approaches to complex systems (pp. 171–188). New York: Springer-Verlag. Braitenberg, V. (1985). Charting the visual cortex. In A. Peters and E. Jones (Eds.), Cerebral cortex, Vol. 3: Visual cortex (pp. 379–414). New York: Plenum. Braitenberg, V., & Schuez, A. (1991). Anatomy of the cortex. Berlin: SpringerVerlag. Bullier, J., Mustari, M. J., & Henry, G. H. (1982). Receptive-field transformations between LGN neurons and S-cells of cat striate cortex. J. Neurophysiology, 47, 417–438. Callaway, E. M., & Katz, L. C. (1990). Emergence and refinement of clustered horizontal connections in cat striate cortex. J. Neurosci., 10, 1134-1153. Calvin, W. H. (1978). Setting the pace and pattern of discharge: Do CNS neurons vary their sensitivity to external inputs via their repetitive firing processes? Federation Proceedings, 37, 2165–2170. Cohen, M. A., & Grossberg, S. (1983). Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Trans. Sys. Man Cyber., 13, 815–826. De Valois, R. L., Yund, E. W., & Hepler, N. (1982). The orientation and direction
62
Douglas A. Miller and Steven W. Zucker
selectivity of cells in macaque visual cortex. Vision Res., 22, 531–544. Dobbins, A., Zucker, S. W., & Cynader, M. S. (1987). Endstopping in the visual cortex as a substrate for calculating curvature. Nature, 329, 438–441. Douglas, R. J., Koch, C., Mahowald, M., Martin, K. A. C., & Suarez, H. (1995). Recurrent excitation in neocortical circuits. Science, 269, 981–985. Douglas, R. J., & Martin, K. A. C. (1990). Neocortex. In G. M. Shepherd (Ed.), The synaptic organization of the brain (3rd ed.) (pp. 389–438). New York: Oxford University Press. Douglas, R. J., & Martin, K. A. C. (1991). Opening the grey box. Trends in Neuroscience, 14, 286–293. Douglas, R. J., & Martin, K. A. C. (1992). Exploring cortical microcircuits: A combined physiological and computational approach. In T. McKenna, J. Davis, & S. F. Zornetzer (Eds.), Single neuron computation, neural nets: Foundations to applications (Ser.) (pp. 381–412). New York: Academic Press. Douglas, R. J., Martin, K. A. C., & Whitteridge, D. (1988). Selective responses of visual cortical cells do not depend on shunting inhibition. Nature (London), 332, 642–644. Douglas, R. J., Martin, K. A. C., & Whitteridge, D. (1989). A canonical microcircuit for neocortex. Neural Computation, 1, 480–488. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., & Reitboeck, H. J. (1988). Coherent oscillations: A mechanism of feature linking in the visual cortex? Biol. Cybern., 60, 121–130. Ferster, D., & Lindstrom, ¨ S. (1983). An intracellular analysis of geniculo-cortical connectivity in area 17 of the cat. J. Physiol, 342, 181–215. Fisher, S., Zucker, S. W., & Carew, T. J. (1997). Use-dependent enhancement of frequency facilitation at L30 inhibitory synapes in Aplysia. Soc. Neurosci. Abstracts Freund, T. F., Martin, K. A. C., Somogyi, P., & Whitteridge, D. (1985). Innervation of cat areas 17 and 18 by physiologically identified X- and Y-type thalamic afferents. II. Identification of postsynaptic targets by GABA immunochemical and Golgi impregnation. J. Comp. Neurol., 242, 275–291. Garey, M. R., & Johnson, D. S. (1979). Computers and intractability. San Francisco: W. H. Freeman. Georgopoulos, A. P., Lurito, J. T., Petrides, M., Schwartz, A. B., & Massey, J. T. (1989). Mental rotation of the neuronal population vector. Science, 243, 234– 236. Gilbert, C. D., & Wiesel, T. N. (1979). Morphology and intracortical projections of functionally characterized neurones in cat visual cortex. Nature, 280, 120–125. Gilbert, C. D., & Wiesel, T. N. (1983). Clustered intrinsic connections in cat visual cortex. J. Neurosci., 3, 1116–1133. Gilbert, C. D., & Wiesel, T. N. (1989). Columnar specificity of intrinsic horizontal and corticocortical connections in cat visual cortex. J. Neurosci., 9, 2432–2442. Gilbert, C. D., & Wiesel, T. N. (1990). The influence of contextual stimuli on the orientation selectivity of cells in primary visual cortex of the cat. Vision Res., 30, 1689–1701. Gopalsamy, K., & He, X. (1994). Stability in asymmetric Hopfield nets with transmission delays. Physica D, 76, 344–358.
Computing with Self-Excitatory Cliques
63
Gray, C. M., Konig, ¨ P., Engel, A. K., & Singer, W. (1989). Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature, 338, 334–337. Gray, C. M., & Singer, W. (1989). Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. Proc. Natl. Acad. Sci. USA, 86, 1698–1702. Harth, E., Csermely, T., Beek, B., & Lindsay, R. D. (1970). Brain functions and neural dynamics. J. Theor. Biol., 26, 93–120. Hebb, D. O. (1949). The organization of behaviour. New York: Wiley. Heller, J., Hertz, J., Kjaer, T., & Richmond, B. (1995). Information flow and temporal coding in primate pattern vision. J. Computational Neuroscience, 2, 175–193. Henry, G. H. (1977). Receptive field classes of cells in the striate cortex of the cat. Brain Research, 133, 1–28. Henry, G. H., Harvey, A. R., & Lund, J. S. (1979). The afferent connections and laminar distribution of cells in the cat striate cortex. J. Comp. Neurol., 187, 725–744. Hirsch, M. W., & Smale, S. (1974). Differential equations, dynamical systems, and linear algebra. New York: Academic Press. Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA, 79, 2554-2558. Hopfield, J. J. (1984). Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. USA, 81, 3088–3092. Hopfield, J. J., & Tank, D. W. (1985). “Neural” computation of decisions in optimization problems. Biol. Cybern., 52, 141–152. Horton, J. C., Dagi, L. R., McCrane, E. P., & de Monasterio, F. M. (1990). Arrangement of ocular dominance columns in human visual cortex. Arch. Ophthalmol., 108, 1025–1031. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol., 160, 106–154. Hubel, D. H., & Wiesel, T. N. (1968). Receptive fields and functional architecture of monkey striate cortex. J. Physiol., 195, 215–243. Hubel, D. H., & Wiesel, T. N. (1977). Ferrier Lecture. Functional architecture of macaque monkey visual cortex. Proc. R. Soc. Lond. B., 198, 1–59. Hummel, R. A., & Zucker, S. W. (1983). On the foundations of relaxation labeling processes. IEEE Trans. Pattern Analysis and Machine Intelligence, PAMI-5, 267– 287. Konig, ¨ P., & Engel, A. (1995). Correlated firing in sensory-motor systems. Current Opinion in Neurobiology, 5, 511–519. Kruger, ¨ J., & Becker, J. D. (1991). Recognizing the visual stimulus from neuronal discharges. Trends in Neuroscience, 14, 282–286. Lehky, S. R., & Sejnowski, T. J. (1990). Neural model of stereoacuity and depth interpolation based on a distributed representation of stereo disparity. J. Neuroscience, 10, 2281–2299. Lowel, ¨ S., & Singer, W. (1992). Selection of intrinsic horizontal connections in the visual cortex by correlated neuronal activity. Science, 255, 209–212. Marcus, C., Waugh, F., & Westervelt, R. (1990). Associative memory in an analog iterated-map network. Phys. Rev. A, 41, 3355–3364.
64
Douglas A. Miller and Steven W. Zucker
Marcus, C., & Westervelt, R. (1989). Dynamics of iterated-map networks. Phys. Rev. A, 40(1), 501–504. Martin, K. A. C. (1984). Neuronal circuits in cat striate cortex. In E. G. Jones & A. Peters (Eds.), Cerebral cortex, Vol. 2: Functional properties of cortical cells (pp. 241–284). New York: Plenum Press. McCormick, D. A. (1990). Membrane properties and neurotransmitter actions. In G. M. Shepherd (Ed.), The synaptic organization of the brain (pp. 32–66). New York: Oxford University Press. McCormick, D. A., Connors, B. W., Lighthall, J. W., & Prince, D. A. (1985). Comparative electrophysiology of pyramidal and sparsely spiny stellate neurons of the neocortex. J. Neurophysiol., 54, 782–806. McClurkin, J., Gawne, T., Optican, L., & Richmond, B. (1991). Lateral geniculate neurons in behaving primates. II. Encoding of visual information in the temporal shape of the response. J. Neurophys., 66, 794–808. McNaughton, B. L., Douglas, R. M., & Goodard, G. V. (1978). Synaptic enhancement in fascia dentata: Cooperativity among coactive afferents. Brain Research, 157, 227–293. Meister, M. (1996). Multineuronal codes in retinal signaling. Proc. Nat. acad. Sci. (USA), 93, 609–614. Michel, A., Farrell, J., & Porod, W. (1988). Stability results for neural networks. In D. Z. Anderson (Ed.), Neural information processing systems (pp. 554–563). New York: American Institute of Physics. Miller, D. A., & Zucker, S. W. (1991). Copositive-plus Lemke algorithm solves polymatrix games. Operations Research Letters, 10, 285–290. Miller, D. A., & Zucker, S. W. (1992). Efficient simplex-like methods for equilibria of nonsymmetric analog networks. Neural Computation, 4, 167–190. Miller, D. A., & Zucker, S. W. (In press). Cliques, computation, and computational tractability. Pattern Recognition, to be published; Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, Venice, May 1997, Pelillo and Hancock (eds), Lecture Notes in Computer Science 1223, Springer-Verlag, New York, 1997. Miller, K. D., Keller, J. B., & Stryker, M. P. (1989). Ocular dominance column development: analysis and simulation. Science, 245, 605–615. Moore, E. F., & Shannon, C. E. (1956). Reliable circuits using less reliable relays. J. Franklin Inst., 262, 191–208, 281–297. Nelken, I. (1988). Analysis of the activity of single neurons in stochastic settings. Biol. Cybern., 59, 201–215. Palm, G. (1980). On associative memory. Biol. Cybern., 36, 19–31. Palm, G. (1981). On the storage capacity of associative memory with randomly distributed storage elements. Biol. Cybern., 39, 125–127. Palm, G. (1982). Neural assemblies: An alternative approach to artificial intelligence. Berlin: Springer-Verlag. Palm, G. (1993). Cell assemblies, coherence and cortico-hippocampal interplay. Hippocampus, 3, 219–225. Palm, G., & Aertsen, A. (Eds.). (1986). Brain theory. Berlin: Springer-Verlag. Peters, A. (1987). Number of neurons and synapses in the primary visual cortex. In E. G. Jones & A. Peters (Eds.), Cerebral cortex, Vol. 6: Further aspects of cortical
Computing with Self-Excitatory Cliques
65
functions including hippocampus (pp. 267–294). New York: Plenum. Peters, A., & Sethares, C. (1991). Organization of pyramidal neurons in area 17 of monkey visual cortex. J. Comp. Neurol., 306, 1–23. Rockland, K. S., & Lund, J. S. (1982). Widespread periodic intrinsic connections in the tree shrew visual cortex. Science, 215, 1532–1534. Rockland, K. S., & Lund, J. S. (1983). Intrinsic laminar lattice connections in primate visual cortex. J. Comp. Neurol., 216, 303–318. Rockland, K. S., Lund, J. S., & Humphrey, A. L. (1982). Anatomical banding of intrinsic connections in striate cortex of tree shrews (tupaia glis). J. Comp. Neurol., 209, 41–58. Schiller, P. H., Finlay, B. L., & Volman, S. F. (1976a). Quantitative studies of singlecell properties of monkey striate cortex. I. Spatio-temporal organization of receptive fields. J. Neurophysiology, 6, 1288–1319. Schiller, P. H., Finlay, B. L., & Volman, S. F. (1976b). Quantitative studies of singlecell properties of monkey striate cortex. II. Orientation specificity and ocular dominance. J. Neurophysiology, 6, 1320–1333. Schiller, P. H., Finlay, B. L., & Volman, S. F. (1976c). Quantitative studies of single-cell properties of monkey striate cortex. III. Spatial frequency. J. Neurophysiology, 6, 1334–1351. Segev, I., Fleshman, J. W., & Burke, R. E. (1989). Compartmental models of complex neurons. In C. Koch & I. Segev (Eds.), Methods in neuronal modeling: From synapses to networks. Cambridge, MA: MIT Press. Sejnowski, T. J. (1981). Skeleton filters in the brain. In G. E. Hinton & J. A. Anderson (Eds.), Parallel models of associative memory. Hillsdale, NJ: Erlbaum. Singer, W. (1990). The formation of cooperative cell assemblies in the visual cortex. J. Exp. Biol., 153, 177–197. Softky, W., & Koch, C. (1992). Cortical cells should fire regularly, but do not. Neural Computation, 4, 643–645. Traub, R., Whittington, M., Stanford, I., & Jefferys, J. (1996). A mechanism for generation of long-range synchronous fast oscillations in the cortex. Nature, 383, 621–624. Treves, A. (1993). Mean-field analysis of neuronal spike dynamics. Network, 4, 259–284. Ts’o, D. Y., Frostig, R., Lieke, E. E., & Grinvald, A. (1990). Functional organization of primate visual cortex revealed by high resolution optical imaging. Science, 249, 417–420. Tsodyks, M., Mitkov, I., & Sompolinsky, H. (1993). Pattern of synchrony in inhomogeneous networks of oscillators with pulse interactions. Phys. Rev. Lett., 71, 1280–1283. Varela, J., Sen, K., Gibson, J., Fost, J., Abbott, L. F., & Nelson, S. (1997). A quantitative description of short-term plasticity at excitatory synapses in layer 2/3 of rat primary visual cortex. J. Neurosci., 17, 7926–7940. Victor, J. (1987). The dynamics of the cat retinal X-cell centre. J. Physiol. (London), 386, 219–246. Waugh, F., Marcus, C., & Westervelt, R. (1991). Reducing neuron gain to eliminate fixed-point attractors in analog associative memory. Phys. Rev. A, 43(6), 3131– 3142.
66
Douglas A. Miller and Steven W. Zucker
Wennekers, T., Sommer, F., & Palm, G. (1995). Iterative retrieval in associative memories by threshold control of different neural models. In H. J. Hermann (ed.), Proc. Workshop on supercomputers in brain research. Singapore: World Scientific. Westheimer, G. (1982). The spatial grain of the perifoveal visual field. Vision Res., 22, 157–162. Westheimer, G. (1990). The grain of visual space. Cold Spring Harbor Symposia on Quantitative Biology, 55, 759–763. Westheimer, G., & McKee, S. P. (1975). Visual acuity in the presence of retinalimage motion. J. Opt. Soc. Am., 65, 847–850. Westheimer, G., & McKee, S. P. (1977). Spatial configurations for visual hyperacuity. Vision Res., 17, 941–947. Willshaw, D. J., Buneman, O. P., & Longuet-Higgins, H. C. (1969). Nonholographic associative memory. Nature, 222, 960–962. Willshaw, D. J., & Longuet-Higgins, H. C. (1970). Associative memory models. In B. Meltzer and D. Michie (Eds.), Machine intelligence Vol. 5, (pp. 351–359). Edinburgh: Edinburgh University Press. Wilson, H. R., Levi, D., Maffei, L., Rovamo, J., & DeValois, R. (1990). The perception of form: Retina to striate cortex. In L. Spillman & J. S. Werner (Eds.), Visual perception (pp. 231–272). New York: Academic Press. Winograd, S., & Cowan, J. D. (1963). Reliable computation in the presence of noise. Cambridge, MA: MIT Press. Worg ¨ otter, ¨ F., & Koch, C. (1991). A detailed model of the primary visual pathway in the cat: Comparison of afferent excitatory and intracortical inhibitory connection schemes for orientation selectivity. J. Neuroscience, 11, 1959–1979. Zucker, S. W., Dobbins, A., & Iverson, L. (1989). Two stages of curve detection suggest two styles of visual computation. Neural Comp., 1, 68–81. Received January 23, 1998; accepted July 1, 1998.
NOTE
Communicated by William W. Lytton
Complex Response to Periodic Inhibition in Simple and Detailed Neuronal Models Corrado Bernasconi Kaspar Schindler Ruedi Stoop Rodney Douglas ¨ ¨ Institute of Neuroinformatics ETH/Uni Zurich, CH-8057 Zurich, Switzerland
Constant current injection with superimposed periodic inhibition gives rise to phase locking as well as chaotic activity in rat neocortical neurons. Here we compare the behavior of a leaky integrate-and-fire neural model with that of a biophysically realistic model of the rat neuron to determine which membrane properties influence the response to such stimuli. We find that only the biophysical model with voltage-sensitive conductances can produce chaotic behavior. 1 Introduction We recently reported on the spike patterns of periodically stimulated neurons from rat neocortical slices (Schindler, Bernasconi, Stoop, Goodman, & Douglas, 1997a). Our motivation was to analyze factors that influence the firing patterns of single cells, because of their presumed role in neural information processing (Rieke, Warland, de Ruyter van Steveninck, & Bialek, 1997). In the experiments, we first injected constant current to regularly spiking layer V pyramidal cells and determined the period of the unperturbed activity. We then perturbed the system by superimposing periodic inhibitory pulses and studied the spiking behavior of the cell as a function of the parameter Ä, the ratio between the period of the perturbation and that of the unperturbed activity. Depending on the choice of Ä, regular entrained spike patterns or irregular behavior were observed. The two regimes corresponded to phase-locked and chaotic orbits produced by iteration of a Poincar´e map (Guevara, Glass, & Shrier, 1981; Alligood, Sauer, & Yorke, 1997) derived from the experiments that can be used to predict the spiking patterns of the cell. In this article we explore the cellular mechanisms that could have an influence on the temporal features of the spiking activity under the described stimulation paradigm. We consider first a simple neuronal model, the leaky integrate-and-fire (I&F) unit, and demonstrate its limitations. We then use simulations of a more detailed biophysical model of the rat cell to investigate the factors determining the particular shape of the experimental response to Neural Computation 11, 67–74 (1999)
c 1999 Massachusetts Institute of Technology °
68
Corrado Bernasconi, Kaspar Schindler, Ruedi Stoop, and Rodney Douglas
inhibition and discuss the relevance of voltage-sensitive conductances for the regularity of the induced spike patterns. 2 I&F Unit Driven with Constant Current and Periodic Inhibition One of the simplest abstractions from the dynamics of a biological spiking cell is the leaky I&F neuron. In spite of its simplicity, this model has been shown to capture many relevant aspects of the temporal dynamics of cortical cells (Knight, 1972; Marsalek, Koch, & Maunsell, 1997). Under the stimulation of an input current I(t), the dynamics of the subthreshold membrane potential V(t) of an I&F model is governed by (Knight, 1972; Marsalek et al., 1997) V(t) I(t) ˙ + , V(t) =− τ C
(2.1)
where C is the membrane capacitance (in the following C = 1 will be assumed) and τ the passive time constant. As soon as a voltage threshold θ is reached, a spike is emitted, and thereafter the voltage is clamped to 0 for a refractory period tr . If an I&F unit is stimulated with a sufficiently large constant current I(t) = I0 > θ/τ , it fires regularly. Integration of equation 2.1 gives the period (time to threshold): ¶ µ θ . T0 = tr − τ log 1 − I0 τ A brief stimulus delivered at time t, at phase φ = t/T0 (mod 1), perturbs these orbits (limit cycle), leading to an interspike interval of length T(φ). The function T(φ)/T0 is called the phase-response curve (PRC) of the system. In the experimental situation, the PRC could be determined by applying stimuli at various places in the interspike interval and measuring the length of the perturbed interval. For the I&F unit stimulated with a single brief current pulse that displaces the membrane potential by q units, the PRC can be calculated explicitly. Equation 2.1 has to be integrated from 0 to t to calculate the voltage after stimulation; then the additional time required to reach threshold from that voltage can be determined. In the case that the pulse is inhibitory, we have T(φ)/T0 =
1 φ+
τ T0
µ log
φT0 −tr τ
−q−I0 τ e− θ −I0 τ
¶
if φ ∈ [0, tr /T0 ] if φ ∈ ]tr /T0 , 1[
.
(2.2)
Since the return to the limit cycle is immediate after the application of the stimulus, we can now use the PRC to infer the effect of periodic inhibition on the sequence of phases {φi }i∈N produced by the system. For a given
Complex Response to Periodic Inhibition
69
stimulation period ts , that is, for the corresponding value of the control parameter Ä = ts /T0 , we have (Guevara et al., 1981; Schindler et al., 1997a): φi+1 = F(φi ) = φi + Ä −
T(φi ) T0
(mod1).
(2.3)
The function F(φi ) is called the Poincar´e map (or the first return map) of the system and expresses the new phase of stimulation as a function of the preceding one. One of the main conclusions from the experiments was that for some values of Ä, the cells and the stimulating oscillator were phase locked, whereas for other values, the spike patterns were chaotic (Schindler et al., 1997a). A necessary condition for chaotic behavior to occur is that the one-dimensional Poincar´e map describing the dynamics be noninjective. If we consider trajectories for which the limit cycle exists (Iτ > θ), the derivative with respect to φ of the Poincar´e map will assume only positive values. As a consequence, the map is monotonic over the entire cycle (see Figure 1A). Chaotic behavior is therefore excluded for the leaky I&F unit. 1 The I&F neuron has passive membrane properties but lacks the active conductances that dominate the dynamics of the neuronal membrane potential of real cells during spiking activity, and that could contribute to more complex behaviors. We therefore analyzed a biophysically more sophisticated neuronal model in search of cellular properties explaining firing patterns going beyond the dynamics of the I&F unit. 3 Periodic Inhibition of a Biophysical Model of a Regularly Spiking Neocortical Cell We performed computer simulations of a biologically realistic cell using NEURON (Hines, 1989). In order to mimic the behavior of the particular cells analyzed in the experiments, we modified the model of a simplified neocortical pyramidal cell described in the literature (Bernander, 1993; Rhodes & Gray, 1994). In particular we had to tune several kinetic parameters of the eight active currents of the model and replace one of them to obtain a good match to the interspike trajectory of a regularly discharging neuron. 2 1 It can be shown that the Poincar´ e map remains injective also for a periodic stimulus constituted by sequences {pi } of inhibitory δ pulses, which abruptly discharge the membrane of finite amounts {qi }. A similar set of pulses can virtually approximate any kind of inhibitory postsynaptic potential. The only condition for consistency is that the sequence of pulse cannot extend beyond the perturbed period (which is guaranteed, for instance, if each of the pulses is applied at a membrane potential lower than that of the first pulse), or, alternatively, that an intervening spike truncates the sequence of pulses (as it is observed in the experiments). The proof of this proposition, based on the same reasoning applied to the case of the single pulse, is obtained by induction on the number of pulses. It is also easy to show that with excitatory pulses, the system can display only periodic behavior. 2 The active currents of the model with the respective conductance densities were: Ina (fast sodium current), Gna = 500 mS/cm2 ; Idr (delayed rectifier K+ current), Gdr =
70
Corrado Bernasconi, Kaspar Schindler, Ruedi Stoop, and Rodney Douglas
A
B
0
0.2
0.4
φi+1
0.6
0.8
1
C
0
0.2
0.4
0.6
φi
0.8
1 0
0.2
0.4
0.6
φi
0.8
1 0
0.2
0.4
φi
0.6
0.8
1
Figure 1: (A) Poincar´e map of the I&F neuron (solid line, obtained from equation 2.2) and of a neocortical rat cell (crosses) from the experiments. The parameters of the I&F unit were: τ = 10 ms, θ = 1, I = 0.103, q = −0.06, Ä = 0. The map is monotonic over the entire cycle. The map of the experimental cell (which was stimulated with DC of 0.7 nA and pulses of 5 ms and −1.5 nA and had an unperturbed interspike interval of 81 ms) was fitted from experimental data. (B) Poincar´e maps (obtained with inhibitory pulses of −1.5 nA and 5 ms; Ä = 0) of the biophysical model of the experimental neocortical cell. To obtain the different curves, we stepwise decreased the M-current conductance density from Gm = 2.6 to 1.2 mS/cm2 . The input currents are adjusted (from 0.5 to 0.7 nA) to maintain the unperturbed periods between 83 and 85 ms. The curve with the largest downward curvature is the one with the intact M-current. With decreasing conductance density, the maps become less curved. (C) Poincar´e map obtained with synaptic stimulation that reproduced experimental data. The dynamics of the conductance was described by the α-like function g(t) = gmax (t/τ )exp(−(t − τ )/τ ) with maximal value gmax = 1.5 pA and time constant τ = 4 ms. The reversal potential of the synapse was −95 mV.
Since the spiking patterns elicited by a periodic stimulus can be predicted by the Poincar´e map (see Figure 1B), we examined the factors influencing its properties. The map was relatively insensitive to manipulation of the fast voltage-dependent currents such as those of the spike mechanism, provided that a robust 1 ms action potential was present. The features of the interspike trajectory are important to the Poincar´e map, and so modifications of the currents underlying the medium-duration afterhyperpolarization (Schwindt et al., 1988) had most interesting effects. 110 mS/cm2 ; Ia (A-type current, taken from Rhodes & Gray, 1994), Ga = 3 mS/cm2 ; Im (Mtype current), Gm = 2.4 mS/cm2 ; Iahp (Ca++ -dependent K+ current), Gahp = 80 mS/cm2 ; Ica (high-threshold calcium current), Gca = 3.2 mS/cm2 ; INa,p (persistent Na+ current), GNa,p = 1 mS/cm2 ; mS/cm2 ; Iar (anomalous rectifier current), Gar = 1 mS/cm2 . Further details of the simulation are given in Schindler et al. (1997a).
Complex Response to Periodic Inhibition
71
A distinctive feature of the experimental rat cell was the fact that an inhibitory pulse applied shortly after a spike reduced the length of the interspike interval. This is in agreement with some previous simulations performed by Lytton and Sejnowski (1991) and had important consequences for the behavior of the system. The shortening of the perturbed interval on early stimulation was related to the partial deactivation of slow K+ conductances by the hyperpolarizing pulse. As a consequence, the K+ outward current decreased enough to allow the cell to reach threshold earlier. This effect was mainly related to the dynamics of the M-current, so the presence of this particular active conductance conferred the characteristic shape to the PRC and the Poincar´e map. The shortening of the interspike interval on early stimulation produced the high initial slope, bringing the map above the identity line. The steepest lengthening of the interspike interval for a pulse applied late in the cycle contributed to the presence of the negative slope of the return map in the second half of the interval. The role of the M-current was also critical to the expression of chaotic spike patterns. In general, a smaller M-type conductance was associated with a less curved Poincar´e map. When we decreased the conductance by more than about 25%, we obtained a monotonic Poincar´e map, which lacks a property necessary for chaotic behavior. This result arises for two reasons. First, there was no longer a significant shortening of the perturbed interval; second, the maximal lengthening of the interval decreased slightly, and so did the downward bending of the Poincar´e map. These changes are partly due to the shorter duration of the unperturbed period T0 caused by the decreased total outward current. In order to operate in the same range of frequencies as in the case of intact M-current, we compensated for the missing K+ conductance by increasing the A-type conductance or the calciumdependent K+ conductance or, alternatively, by lowering the input current. Figure 1B shows the effect of a progressive reduction of the conductance of the M-current and a parallel adjustment of the input current to maintain a constant unperturbed interval. None of the described manipulations restored a noninjective Poincar´e map. In agreement with the experiments, the original, nonmonotonic map was associated with positive Lyapunov exponents, that is, with unstable, chaotic orbits (Alligood et al., 1997; Peinke, Parisi, Roessler, & Stoop, 1992). Figure 2 illustrates two examples of the behavior of the model for parameters expected to be associated with regular and irregular firing. The manipulation of parameters of other types of active currents (e.g., of kinetic parameters of Iahp , Ia , or Ica ), under the constraint of a realistic behavior of the cell and of a physiological choice of the parameters of the currents, could not reverse the effect of a decreased M-type conductance on the return map. In fact, those parameters affected the overall behavior of the neuron (shape of the action potential or of the interspike trajectory, adaptation features, presence of bursts, etc.) more drastically than they affected the Poincar´e map.
membrane potential (mV)
72
Corrado Bernasconi, Kaspar Schindler, Ruedi Stoop, and Rodney Douglas
40 0
-40 -80
input current (nA)
1 -1 0
500 time (ms)
1000 0
500 time (ms)
1000
Figure 2: Traces of the membrane potential for two values of the control parameter Ä. The parameters of the simulation are those of the model with intact M-current of Figure 1B. In the left plot, Ä = 0.2 (mod 1) produces phase-locked trajectories. On the right, periodic stimulation with Ä = 0.5 induces irregular spiking patterns. The transition between different firing modes can be described by bifurcations produced by the Poincar´e map, which belongs to the class of the one-dimensional circle maps (Stoop et al., 1998).
The effect of the inhibitory input was most prominent when the output frequency of the cell was around 10 to 20 Hz. At these frequencies, perturbations interfere incisively with the dynamics of the M-current, which has an (activation) time constant of 40 ms. There must be a match between firing frequency and kinetic parameters of the current for the mechanism to be effective. However, although the role of the M-current is central in the cases considered, other currents (in particular voltage sensitive currents such as the A-current) also contribute to the shaping of the Poincar´e map. The effect of channels with different kinetics might be prevalent at other firing regimes or in other classes of cells with different dynamical properties (e.g, fast spiking cells or intrinsically bursting cells). To explore more realistic types of inhibition than rectangular current pulses, we also investigated the effect of simulated inhibitory synapses (see Figure 1C). The values of the synaptic dynamics were obtained from experiments in which the afferents to the impaled cell were stimulated with a bipolar electrode, while excitatory synapses were blocked pharmacologically (Schindler et al., 1997b). The effect of this relatively long-lasting inhibition was a reduced shortening of the interspike interval on early stimulation and a larger lengthening on late stimulation. Nevertheless, in agreement with the experiments, the obtained Poincar´e map was noninjective. As already noted, the map of an I&F unit stimulated in a similar way does not have this property.
Complex Response to Periodic Inhibition
73
4 Conclusions Due to its simplicity, the leaky integrate-and-fire neuron is commonly used for large-scale simulations of neural systems and hardware implementations. It is therefore important to know which biological phenomena its dynamics can capture and which it cannot. We have shown that with periodic inhibition, leaky integrate-and-fire neurons cannot generate chaotic spike patterns, whereas more biophysically realistic models with voltagesensitive conductances have a substantially enriched dynamics, which permit chaotic spike patterns. In particular, the irregular discharge of pyramidal neurons depends crucially on outward currents of a medium duration such as the M-current. Because chaos can mediate between different periodicities, the ability of a neuron to express such activity may be important in the context of synchronization and desynchronization of discharge in biological neural networks. This aspect is under investigation (Stoop et al., 1998). Aknowledgments This work was supported by the Helmut Horten Stiftung (Madonna del Piano, CH), the Maurice E. Muller ¨ Stiftung (Bern, CH), and the Swiss Priority Programme Biotechnology of the Swiss National Science Foundation. We thank Paul Verschure, Phil Goodman, Giacomo Indiveri, and Peter Konig ¨ for helpful discussions and the referees for useful comments. References Alligood, K., Sauer, T., & Yorke, J. (1997). Chaos, an introduction to dynamical systems. New York: Springer-Verlag. ¨ (1993). Synaptic integration and its control in neocortical pyramidal Bernander, O. cells. Unpublished doctoral dissertation, California Institute of Technology. Guevara, M., Glass, L., & Shrier, A. (1981). Phase locking, period-doubling bifurcations, and irregular dynamics in periodically stimulated cardiac cells. Science, 214, 1350–1353. Hines, M. (1989). A program for simulation of nerve equations with branching geometries. Int. J. Biomed. Comput., 24, 55–68. Knight, B. (1972). Dynamics of encoding in a population of neurons. J. Gen. Phys., 59, 734–766. Lytton, W., & Sejnowski, T. (1991). Simulations of cortical pyramidal neurons synchronized by inhibitory interneurons. J. Neurophysiol., 66(3), 1059–1079. Marsalek, P., Koch, C., & Maunsell, J. (1997). On the relationship between synaptic input and spike output jitter in individual neurons. Proc. Natl. Acad. Sci. U.S.A., 94, 735–740. Peinke, J., Parisi, J., Roessler, O., & Stoop, R. (1992). Encounter with chaos. Berlin: Springer-Verlag.
74
Corrado Bernasconi, Kaspar Schindler, Ruedi Stoop, and Rodney Douglas
Rhodes, P., & Gray, C. (1994). Simulations of intrinsically bursting neocortical pyramidal neurons. Neural Comput., 6, 1086–1110. Rieke, F., Warland, D., de Ruyter van Steveninck, R., & Bialek, B. (1997). Spikes: Exploring the neural code. Cambridge, MA: MIT Press. Schindler, K., Bernasconi, C., Stoop, R., Goodman, P., & Douglas, R. (1997a). Chaotic spike patterns evoked by periodic inhibition of rat cortical neurons. Z. Naturforsch., 52a, 509–512. Schindler, K., Bernasconi, C., Stoop, R., Goodman, P., Douglas, R., & Martin, K. A. C. (1997b). Irregular spike patterns produced by periodic inhibition of a regularly firing rat neocortical neuron. Soc. Neurosci. Abstr., 23, 397.5. Schwindt, P., Spain, W., Foehring, R., Stafstrom, C., Chubb, M., & Crill, W. E. (1988). Multiple potassium conductances and their functions in neurons from cat sensorimotor cortex in vitro. J. Neurophysiol., 59, 424–449. Stoop, R., et al. (1998). Chaotic inhibitory connections generate global periodicity in cortical neural networks. Unpublished manuscript. Received February 27, 1998; accepted June 10, 1998.
NOTE
Communicated by Laurence Abbott
Neuronal Tuning: To Sharpen or Broaden? Kechen Zhang Howard Hughes Medical Institute, Computational Neurobiology Laboratory, Salk Institute for Biological Studies, La Jolla, CA 92037, U.S.A.
Terrence J. Sejnowski Department of Biology, University of California, San Diego, La Jolla, CA 92093, U.S.A. Howard Hughes Medical Institute, Computational Neurobiology Laboratory, Salk Institute for Biological Studies, La Jolla, CA 92037, U.S.A.
Sensory and motor variables are typically represented by a population of broadly tuned neurons. A coarser representation with broader tuning can often improve coding accuracy, but sometimes the accuracy may also improve with sharper tuning. The theoretical analysis here shows that the relationship between tuning width and accuracy depends crucially on the dimension of the encoded variable. A general rule is derived for how the Fisher information scales with the tuning width, regardless of the exact shape of the tuning function, the probability distribution of spikes, and allowing some correlated noise between neurons. These results demonstrate a universal dimensionality effect in neural population coding. 1 Introduction Let the activity of a population of neurons represent a continuous D-dimensional vector variable x = (x1 , x2 , . . . , xD ). Randomness of spike firing implies an inherent inaccuracy, because the numbers of spikes fired by these neurons differ in repeated trials; thus, the true value of x can never be completely determined, regardless of the method for reading out the information. The Fisher information J provides a good measure on encoding accuracy because its inverse is the Cram´er-Rao lower bound on the mean squared error: h i 1 E ε2 ≥ , J
(1.1)
which applies to all possible unbiased estimation methods that can read out variable x from population activity without systematic error (Paradiso, 1988; Seung & Sompolinsky, 1993; Snippe, 1996). The Cram´er-Rao bound can sometimes be reached by biologically plausible decoding methods (Pouget, Neural Computation 11, 75–84 (1999)
c 1999 Massachusetts Institute of Technology °
76
Kechen Zhang and Terrence J. Sejnowski
Zhang, Deneve, & Latham, 1999; Zhang, Ginzburg, McNaughton, & Sejnowski, 1998). Here the square error in a single trial is 2 , ε2 = ε12 + ε22 + · · · + εD
(1.2)
with εi the error for estimating xi . The Fisher information J can be defined by ( "µ ¶2 #)−1 D 1 X ∂ = ln P(n | x, τ ) , E J ∂xi i=1
(1.3)
where the average is over n = (n1 , n2 , . . . , nN ), the numbers of spikes fired by all the neurons within a time interval τ , with the probability distribution P depending on the value of the encoded variable x. The definition in equation 1.3 is appropriate if the full Fisher information matrix is diagonal. This is indeed the case in this article because we consider only randomly placed radial symmetric tuning functions for a large population of neurons so that the distributions of estimation errors in different dimensions are always identical, and uncorrelated. A recent introduction to Fisher information can be found in Kay (1993). 2 Scaling Rule for Tuning Width The problem of how coding accuracy depends on the tuning width of neurons and dimensionality of the space being represented was first studied by Hinton, McClelland, and Rumelhart (1986) and later by Baldi and Heiligenberg (1988), Snippe and Koenderink (1992), Zohary (1992), and Zhang et al. (1998). All of these earlier results involved specific assumptions on the tuning functions, the noise, and the measure of coding accuracy. Here we consider the general case using Fisher information as a measure and show that there is a universal scaling rule. This rule applies to all methods that can achieve the best performance given by the Cram´er-Rao bound, although it cannot constrain the tuning properties of suboptimal methods. The tuning function refers to the dependence of the mean firing rate f (x) of a neuron on the variable of interest x = (x1 , x2 , . . . , xD ). We consider only radial symmetric functions µ
|x − c|2 f (x) = Fφ σ2
¶ ,
(2.1)
which depend on only the Euclidean distance to the center c. Here σ is the tuning width, which scales the tuning function without changing its shape, and F is the mean peak firing rate, given that the maximum of φ is 1. This general formula includes all radial symmetric functions, of which gaussian tuning is the special case: φ(z) = exp(−z/2). Other tuning functions
Neuronal Tuning
77
can sometimes be transformed into a radial symmetric one by scaling or a linear transformation on the variable. If x is a circular variable, the tuning equation, 2.1, is reasonable only for sufficiently narrow tuning functions, because a broad periodic tuning function cannot be scaled uniformly without changing its shape. Suppose the probability for n spikes to occur within a time window of length τ is P(n | x, τ ) = S(n, f (x), τ ).
(2.2)
This general formulation requires only that the probability be a function of the mean firing rate f (x) and the time window τ . These general assumptions are sufficient conditions to prove that the total Fisher information has the form J = ησ D−2 Kφ (F, τ, D),
(2.3)
where η is a number density—the number of neurons whose tuning centers fall into a unit volume in the D-dimensional space of encoded variable, assuming that all neurons have identical tuning parameters and independent activity. We also assume that the centers of the tuning functions are uniformly distributed at least in the local region of interest. Thus, η is proportional to the total number of neurons that are activated. The subscript of Kφ implies that it also depends on the shape of function φ. Equation 2.3 gives the complete dependence of J on tuning width σ and number density η. The factor σ D−2 is consistent with the specific examples considered by Snippe and Koenderink (1992) and Zhang et al. (1998), but the exponent is off by one from the noiseless model considered by Hinton et al. (1986). More generally, when different neuron groups have different tuning widths σ and peak firing rates F, we have E D J = η σ D−2 Kφ (F, τ, D) ,
(2.4)
where the average is over neuron groups, and η is the number density including all groups so that J is still proportional to the total number of contributing neurons. Equation 2.4 follows directly from equation 2.3 because Fisher information is additive for neurons with independent activity. Equations 2.3 and 2.4 show how the Fisher information scales with the tuning width in arbitrary dimensions D. Sharpening the tuning width helps only when D = 1, has no effect when D = 2, and reduces information encoded by a fixed set of neurons for D ≥ 3 (see Figure 1A). Although sharpening makes individual neurons appear more informative, it reduces the number of simultaneously active neurons, a factor that dominates in higher dimensions where neighboring tuning functions overlap more substantially.
Kechen Zhang and Terrence J. Sejnowski
A
Fisher information per neuron per sec
78
6
10
D=4 4
10
D=3
D=2
2
10
D=1 0
10
0
5
10 15 20 Average tuning width
25
1
10
Fisher information per spike
B
0
10
−1
10
D=1
−2
10
−3
10
D=4 −4
10
0
5
10 15 20 Average tuning width
25
Figure 1: The accuracy of population coding by tuned neurons as a function of tuning width follows a universal scaling rule regardless of the exact shape of the tuning function and the exact probability distribution of spikes. The accuracy depends on the total Fisher information, which is here proportional to the total number of both neurons and spikes. (A) Sharpening the tuning width can increase, decrease, or not change the Fisher information coded per neuron, depending on the dimension D of the encoded variable, but (B) sharpening always improves the Fisher information coded per spike and thus energy efficiency for spike generation. Here the model neurons have gaussian tuning functions with random spacings (average in each dimension taken as unity), independent Poisson spike distributions, and independent gamma distributions for tuning widths and peak firing rates (the average is 25 Hz).
Neuronal Tuning
79
2.1 Derivation. To derive equation 2.3, first consider a single variable, say, x1 , from x = (x1 , x2 , . . . , xD ). The Fisher information for x1 for a single neuron is "µ
¶2 # ∂ ln P(n | x, τ ) J1 (x) = E ∂x1 ¶ µ (x1 − c1 )2 |x − c|2 , F, τ , = Aφ 2 σ σ4
(2.5) (2.6)
where the first step is a definition and the average is over the number of spikes n. It follows from equations 2.1 and 2.2 that ∂ ln P(n | x, τ ) = ∂x1 µ µ ¶ ¶ µ 2 ¶ 2(x − c ) |x − c|2 1 1 0 |x − c| , T n, Fφ , τ Fφ σ2 σ2 σ2
(2.7)
where φ 0 (z) = dφ(z)/dz and function T is defined by T(n, z, τ ) =
∂ ln S(n, z, τ ). ∂z
(2.8)
Therefore, averaging over n must yield the form in equation 2.6, with function Aφ depending on the shape of φ. Next, the total Fisher information for x1 for the whole population is the sum of J1 (x) over all neurons. The sum can be replaced by an integral, assuming that centers of tuning functions are uniformly distributed with density η in the local region of interest: Z J1 = η
∞
J1 (x) dx1 · · · dxD Z ∞ Aφ (ξ 2 , F, τ )ξ12 dξ1 · · · dξD = ησ D−2
(2.10)
≡ ησ D−2 Kφ (F, τ, D)D,
(2.11)
−∞
−∞
(2.9)
where new variables ξi = (xi − ci )/σ have been introduced so that |x − c|2 = ξ12 + · · · + ξD2 ≡ ξ 2 , σ2 dx1 · · · dxD = σ D dξ1 · · · dξD .
(2.12) (2.13)
Finally, the Fisher information for all D dimensions is J = J1 /D, because the mean squared error in each dimension is the same. The result is equation 2.3.
80
Kechen Zhang and Terrence J. Sejnowski
2.2 Example: Poisson Spike Model. A Poisson distribution is often used to approximate spike statistics: P(n | x, τ ) = S(n, f (x), τ ) =
(τ f (x))n exp(−τ f (x)). n!
(2.14)
Then equation 2.4 becomes E D J = η σ D−2 F τ kφ (D),
(2.15)
where kφ (D) =
4 D
Z
∞
−∞
(φ 0 (ξ 2 )ξ1 )2 dξ1 · · · dξD , φ(ξ 2 )
(2.16)
with ξ 2 = ξ12 + · · · + ξD2 . For example, if the tuning function φ is gaussian, kφ (D) = (2π)D/2 /D.
(2.17)
One special feature of Poisson spike model is that Fisher information in equation 2.15 is proportional to the peak firing rate F. 3 Fisher Information per Spike The energy cost of encoding can be estimated by the Fisher information per spike: Jspikes = J/Nspikes .
(3.1)
If all neurons have identical tuning parameters, the total number of spikes within time window τ is Z Nspikes = η
∞
τ f (x) dx1 · · · dxD Z ∞ φ(ξ 2 ) dξ1 · · · dξD = ησ D Fτ
(3.2)
≡ ησ D Fτ Qφ (D),
(3.4)
−∞
−∞
(3.3)
where f (x) is mean firing rate given by equation 2.1 and ξ 2 = ξ12 + · · · + ξD2 . More generally, when tuning parameters vary in the population, we have D E Nspikes = η σ D F τ Qφ (D).
(3.5)
Neuronal Tuning
81
For example,
Jspike
® σ D−2 = D® D σ
(3.6)
holds when the neurons have gaussian tuning functions, independent Poisson spike distributions, and independent distributions of peak firing rates and tuning widths. As shown in Figure 1B, sharpening the tuning saves energy for all dimensions. 4 Scaling Rule Under Noise Correlation The example in this section shows that scaling law still holds when firingrate fluctuations of different neurons are weakly correlated, and the difference is a constant factor. Assume a continuous model for spike statistics based on multivariate gaussian distribution, where the average number of spikes ni for neuron i is µ ¶ |x − ci |2 , (4.1) µi = E [ni ] = τ fi (x) = τ Fφ σ2 and different neurons have identical tuning parameters except for the location of the centers. The noise correlation between neurons i and j is ½ 2 £ ¤ Ci if i = j, (4.2) Cij = E (ni − µi )(nj − µj ) = qCi Cj otherwise, where Ci = ψ(µi ) = ψ(τ fi (x)),
(4.3)
with ψ an arbitrary function. For example, ψ(z) ≡ constant and ψ(z) = az are the additive and√multiplicative noises considered by Abbott and Dayan (1999), and ψ(z) = z corresponds to the limit of a Poisson distribution. For large population and weak correlation, we obtain the Fisher information ¶ µ ¶ µ 1 1 D−2 Aφ,ψ (F, τ, D) + 1 + Bφ,ψ (F, τ, D) , (4.4) J = ησ 1−q 1−q ignoring contributions from terms slower than linear with respect to the population size. Here Aφ,ψ (F, τ, D) =
4τ 2 F2 D
4τ 2 F2 Bφ,ψ (F, τ, D) = D
Z
∞
−∞
Z
∞
−∞
µ µ
φ 0 (ξ 2 )ξ1 ψ(ζ )
¶2 dξ1 · · · dξD ,
φ 0 (ξ 2 )ψ 0 (ζ )ξ1 ψ(ζ )
(4.5)
¶2 dξ1 · · · dξD ,
(4.6)
82
Kechen Zhang and Terrence J. Sejnowski
with ξ 2 = ξ12 + · · · + ξD2 and ³ ´ ζ = τ Fφ ξ 2 .
(4.7)
Thus, the only contribution of noise correlation is the constant factor 1/(1 − q), which slightly increases the Fisher information when there is positive correlation (q > 0). This result is consistent with the conclusion of Abbott and Dayan (1999). Notice that now the scaling rule for tuning width remains the same, and the Fisher information is still proportional to the total number of contributing neurons. Equation 2.15 for the √ Poisson spike model can be recovered from equation 4.4 when ψ(z) = z with a large time window and high firing rates so that the contribution from ψ 0 or Bφ,ψ can be ignored. The only difference is an additional proportional constant 1/(1 − q). 5 Hierarchical Processing In hierarchical processing, the total Fisher information cannot increase when transmitted from population A to population B (Pouget, Deneve, Ducom, & Latham, 1998, 1999). This is because decoding a variable directly from population B is indirectly decoding from population A and therefore must be subject to the same Cram´er-Rao bound. Assuming a Poisson spike model with fixed noise correlation (cf. the end of section 4), we have NB D D−2 E NA D D−2 E F ≥ F , σ σ A B 1 − qA 1 − qB
(5.1)
where the averages are over all neurons of total numbers NA and NB in the two populations. This constrains allowable tuning parameters in the hierarchy. 6 Concluding Remarks The issue of how tuning width affects coding accuracy was raised again recently by the report of progressing sharpening of tuning curves for interaural time difference (ITD) in the auditory pathway (Fitzpatrick, Batra, Stanford, & Kuwada, 1997). In a hierarchical processing system, the total information cannot be increased at a later stage by altering tuning parameters because of additional constraints such as inequality 5.1. (See the more detailed discussion by Pouget et al., 1998.) For a one-dimensional feature such as ITD, more information can be coded per neuron for a sharper tuning curve, provided that all other factors are fixed, such as peak firing rate and noise correlation. For two-dimensional features, such as the spatial representation by hippocampal place cells, coding accuracy should be insensitive to the tuning width (Zhang et al., 1998).
Neuronal Tuning
83
In three and higher dimensions, such as the multiple visual features represented concurrently in the ventral stream of primate visual system, more information can be coded per neuron by broader tuning. For energy consumption, narrower tuning improves the information coded per spike, provided that the tuning width stays large enough compared with the spacing of tuning functions. Therefore, it is advantageous to use relatively narrow tuning for one- and two-dimensional features, but there is a trade-off between coding accuracy and energy expenditure for features of three and higher dimensions. The scaling rule compares different system configurations or the same system under different states, such as attention. For example, contrary to popular intuition, sharpening visual receptive fields should not affect how accurately a small, distant target can be localized by the visual system, because the example here is two-dimensional. The results presented here are sufficiently general to apply to neural populations in a wide range of biological systems.
Acknowledgments We thank Alexandre Pouget, Richard S. Zemel, and the reviewers for helpful comments and suggestions.
References Abbott, L. F., & Dayan, P. (1999). The effect of correlated variability on the accuracy of a population code. Neural Computation, 11, 91–101. Baldi, P., & Heiligenberg, W. (1988). How sensory maps could enhance resolution through ordered arrangements of broadly tuned receivers. Biological Cybernetics, 59, 313–318. Fitzpatrick, D. C., Batra, R., Stanford, T. R., & Kuwada, S. (1997). A neuronal population code for sound localization. Nature, 388, 871–874. Hinton, G. E., McClelland, J. L., & Rumelhart, D. E. (1986). Distributed representations. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing (Vol. 1, pp. 77–109). Cambridge, MA: MIT Press. Kay, S. M. (1993). Fundamentals of statistical signal processing: Estimation theory. Englewood Cliffs, NJ: Prentice Hall. Paradiso, M. A. (1988). A theory for the use of visual orientation information which exploits the columnar structure of striate cortex. Biological Cybernetics, 58, 35–49. Pouget, A., Deneve, S., Ducom, J.-C., & Latham, P. E. (1999). Narrow versus wide tuning curves: What’s better for a population code?. Neural Computation, 11, 85–90. Pouget, A., Zhang, K.-C., Deneve, S., & Latham, P. E. (1998). Statistically efficient estimation using population code. Neural Computation, 10, 373–401.
84
Kechen Zhang and Terrence J. Sejnowski
Seung, H. S., & Sompolinsky, H. (1993). Simple models for reading neuronal population codes. Proceedings of the National Academy of Sciences USA, 90, 10749–10753. Snippe, H. P. (1996). Parameter extraction from population codes: A critical assessment. Neural Computation, 8, 511–539. Snippe, H. P., & Koenderink, J. J. (1992). Discrimination thresholds for channelcoded systems. Biological Cybernetics, 66, 543–551. Zhang, K.-C., Ginzburg, I., McNaughton, B. L., & Sejnowski, T. J. (1998). Interpreting neuronal population activity by reconstruction: Unified framework with application to hippocampal place cells. Journal of Neurophysiology, 79, 1017–1044. Zohary, E. (1992). Population coding of visual stimuli by cortical neurons tuned to more than one dimension. Biological Cybernetics, 66, 265–272. Received January 13, 1998; accepted May 27, 1998.
NOTE
Communicated by Richard Zemel
Narrow Versus Wide Tuning Curves: What’s Best for a Population Code? Alexandre Pouget Sophie Deneve Jean-Christophe Ducom Georgetown Institute for Computational and Cognitive Sciences, Georgetown University, Washington, DC 20007-2197, U.S.A.
Peter E. Latham Department of Neurobiology, University of California at Los Angeles, Los Angeles, CA 90095-1763, U.S.A.
Neurophysiologists are often faced with the problem of evaluating the quality of a code for a sensory or motor variable, either to relate it to the performance of the animal in a simple discrimination task or to compare the codes at various stages along the neuronal pathway. One common belief that has emerged from such studies is that sharpening of tuning curves improves the quality of the code, although only to a certain point; sharpening beyond that is believed to be harmful. We show that this belief relies on either problematic technical analysis or improper assumptions about the noise. We conclude that one cannot tell, in the general case, whether narrow tuning curves are better than wide ones; the answer depends critically on the covariance of the noise. The same conclusion applies to other manipulations of the tuning curve profiles such as gain increase. 1 Introduction It is widely assumed that sharpening tuning curves, up to a certain point, can improve the quality of a coarse code. For instance, attention is believed to improve the code for orientation by sharpening the tuning curves to orientation in the visual area V4 (Spitzer, Desimone, & Moran, 1988). This belief comes partly from a seminal paper by Hinton, McClelland, and Rumelhart (1986), which showed that there exists an optimal width for which the accuracy of a population code is maximized, suggesting that sharpening is beneficial when the tuning curves have a width larger than the optimal one. This result, however, was derived for binary units and does not readily generalize to continuous units. A recent attempt to show experimentally that, for continuous tuning curves, sharper is better relied on the center-of-mass estimator to evaluate Neural Computation 11, 85–90 (1999)
c 1999 Massachusetts Institute of Technology °
86
A. Pouget, S. Deneve, J.-C. Ducom, & P. E. Latham
the quality of the code (Fitzpatrick, Batra, Stanford, & Kuwada, 1997). These authors measured the tuning curves of auditory neurons to interaural time difference (ITD), a cue for localizing auditory stimuli. They argued that narrow tuning curves are better than wide ones—in the range they observed experimentally—in the sense that the minimum detectable change (MDC) in ITD is smaller with narrow tuning curves when using a center-of-mass estimator. Their analysis, however, suffered from two problems: (1) they did not consider a biologically plausible model of the noise, and (2) the MDC obtained with a center of mass is not, in the general case, an objective measure of the information content of a representation, because center of mass is not an optimal readout method (Snippe, 1996). A better way to proceed is to use Fisher information, the square root of which is inversely proportional to the smallest achievable MDC independent of the readout method (Paradiso, 1988; Seung & Sompolinsky, 1993; Pouget, Zhang, Deneve, & Latham, 1998). (Shannon information would be another natural choice, but it is simply, and monotonically, related to Fisher information in the case of population coding with a large number of units; see Brunel & Nadal, 1998. It thus yields identical results when comparing codes.) To determine whether sharp tuning curves are indeed better than wide ones, one can simply plot the MDC obtained from Fisher information as a function of the width of the tuning curves. Fisher information is defined as ¸ · ∂2 (1) I = E − 2 log P(A|θ) , ∂θ where P(A|θ) is the distribution of the activity conditioned on the encoded variable θ and E[·] is the expected value over the distribution P(A|θ ). As we show next, sharpening increases Fisher information when the noise distribution is fixed, but sharpening can also have the opposite effect: it can decrease information when the distribution of the noise changes with the width. The latter case, which happens when sharpening is the result of computation in a network, is the most relevant for neurophysiologists. Consider first the case in which the noise distribution is fixed. For instance, for a population of N neurons with gaussian tuning curves and independent gaussian noise with variance σ 2 , Fisher information reduces to I=
N X fi0 (θ)2 , σ2 i=1
(2)
where fi (θ) is the mean activity of unit i in response to the presentation angle, θ , and fi0 (θ) is its derivative with respect to θ. Therefore, as the width of the tuning curve decreases, the derivative increases, resulting in an in-
Narrow Versus Wide Tuning Curves
87
crease of information. This implies that the smallest achievable MDC goes up with the width of tuning, as shown in Figure 1A, because the MDC is inversely proportional to the square root of the Fisher information. This is a case where narrow tuning curves are better than wide ones. Note, however, that the optimal tuning curve for Fisher information has zero width (or, more precisely, a width on the order of 1/N, where N is the number of neurons), unlike what Hinton et al. found for binary tuning curves. Note also that for the same kind of noise, the MDC measured with center of mass shows the opposite trend—wide is better—confirming that the MDC obtained with the center of mass does not reflect the information content of the representation.1 Consider now a case in which the noise distribution is no longer fixed, such as in the two-layer network illustrated in Figure 1B. The network has the same number of units in both layers, and the output layer contains lateral connections, which sharpen the tuning curves. This case is particularly relevant for neurophysiologists since this type of circuit is quite common in the cortex. In fact, some evidence suggests that a similar network is involved in tuning curve sharpening in the primary visual cortex for orientation selectivity (Ringach, Hawken, & Shapley, 1997). Do the output neurons contain more information than the input neurons just because they have narrower tuning curves? The answer is no, regardless of the details of the implementation, because processing and transmission cannot increase information in a closed system (Shannon & Weaver, 1963). Sharpening is done at the cost of introducing correlated noise among neurons, and the loss of information in the output layer can be traced to those correlations (Pouget & Zhang, 1996; Pouget et al., 1998). This is a case where wide tuning curves (the ones in the input layer) are better than narrow ones (the ones in the output layer). That wide tuning curves contain more information than narrow ones in this particular architecture can be easily missed if one assumes the wrong noise distribution. Unfortunately, it is difficult to measure precisely the joint distribution of the noise or even its covariance matrix. It is therefore often assumed that the noise is independent among neurons when dealing with real data. Let’s examine what happens if we assume independent noise for the output units of the network depicted in Figure 1B. We consider the case in which the output units are deterministic; the only source of noise is in the input activities, and the output tuning curves have the same width as the input tuning curves. We have shown (Pouget et al., 1998) that in this case, the network performs a close approximation to maximum likelihood and the noise in the output units is gaussian with variance fi0 (θ)2 /I1 , where I1 is
1 Fitzpatrick et al. (1997) reported the opposite result. They found sharp tuning curves to be better than wide ones when using a center-of-mass estimator. This is because the noise model they used is different from ours and biologically implausible.
88
A. Pouget, S. Deneve, J.-C. Ducom, & P. E. Latham
B Activity
30 25 20
Orientation
15
Output
10
Input
5 0
0
20
40
60
80
Width (deg)
Activity
Minimum Detectable Change(deg)
A
Orientation
Figure 1: (A) For a fixed noise distribution, the minimum detectable change (MDC) obtained from Fisher information (solid line) increases with the width. Therefore, in this case, narrow tuning curves are better, in the sense that they transmit more information about the presentation angle. Note that using a center-of-mass estimator (dashed line) to compute the MDC leads to the opposite conclusion: that wide tuning curves are better. This is a compelling demonstration that the center of mass is not a proper way to evaluate information content. (B) A neural network with 10 input units and 10 output units, fully connected with feedforward connections between layers and lateral connections in the output layer. We show only one representative set of connections for each layer. The lateral weights can be set in such a way that the tuning curves in the output layer are narrower than in the input layer (see Pouget et al., 1998, for details). Because the information in the output layer cannot be greater than the information in the input layer, sharpening tuning curves in the output layer can only decrease (or at best preserve) the information. Therefore, the wide tuning curves in the input layer contain more information about the stimulus than the sharp tuning curves in the output layer. In this case, wide tuning curves are better.
the Fisher information in the input layer. Using equation 2 for independent gaussian noise we find that the information in the output layer, denoted I2 , is given by: I2 =
N X i=1
N X fi0 (θ)2 = I1 = NI1 . fi0 (θ)2 /I1 i=1
The independence assumption would therefore lead us to conclude that the information in the output layer is much larger than in the input layer, which is clearly wrong.
Narrow Versus Wide Tuning Curves
89
These simple examples demonstrate that a proper characterization of the information content of a representation must rely on an objective measure of information, such as Fisher information, and detailed knowledge of the noise distribution and its covariance matrix. (The number of variables being encoded is also critical, as shown by Zhang and Sejnowski, 1999) Using estimators such as the center of mass, or assuming independent noise, is not guaranteed to lead to the right answer. Therefore, attention may sharpen (Spitzer et al., 1988) tuning curves (and/or increase their gain; McAdams & Maunsell, 1996), but whether this results in a better code is impossible to tell without knowledge of the covariance of the noise across conditions. The emergence of multielectrode recordings may soon make it possible to measure these covariance matrices. Acknowledgments We thank Rich Zemel, Peter Dayan, and Kechen Zhang for their comments on an earlier version of this article. References Brunel, N., & Nadal, J. P. (1998). Mutual information, Fisher information and population coding. Neural Computation, In press. Fitzpatrick, D. C., Batra, R., Stanford, T. R., & Kuwada, S. (1997). A neuronal population code for sound localization. Nature, 388, 871–874. Hinton, G. E., McClelland, J. L., & Rumelhart, D. E. (1986). Distributed representations. In D. E. Rumelhart, J. L. McClelland, & the PDP Research Group (Eds.), Parallel distributed processing (Vol. 1, pp. 318–362). Cambridge, MA: MIT Press. McAdams, C. J., & Maunsell, J. R. H. (1996). Attention enhances neuronal responses without altering orientation selectivity in macaque area V4. Society for Neuroscience Abstracts, 22. Paradiso, M. A. (1988). A theory of the use of visual orientation information which exploits the columnar structure of striate cortex. Biological Cybernetics, 58, 35–49. Pouget, A., & Zhang, K. (1996). A statistical perspective on orientation selectivity in primary visual cortex. Society for Neuroscience Abstracts, 22, 1704. Pouget, A., Zhang, K., Deneve, S., & Latham, P. E. (1998). Statistically efficient estimation using population coding. Neural Computation, 10, 373–401. Ringach, D. L., Hawken, M. J., & Shapley, R. (1997). Dynamics of orientation tuning in macaque primary visual cortex. Nature, 387, 281–284. Seung, H. S., & Sompolinsky, H. (1993). Simple model for reading neuronal population codes. Proceedings of National Academy of Sciences, USA, 90, 10749– 10753. Shannon, E., & Weaver, W. (1963). The mathematical theory of communication. Urbana: University of Illinois Press.
90
A. Pouget, S. Deneve, J.-C. Ducom, & P. E. Latham
Snippe, H. P. (1996). Parameter extraction from population codes: A critical assessment. Neural Computation, 8, 511–530. Spitzer, H., Desimone, R., & Moran, J. 1988. Increased attention enhances both behavioral and neuronal performance. Science, 240, 338–340. Zhang, K., & Sejnowski, T. J. (1999). Neuronal tuning: To sharpen or broaden?. Neural Computation, 11, 75–84. Received March 16, 1998; accepted June 25, 1998.
LETTER
Communicated by Michael Shadlen
The Effect of Correlated Variability on the Accuracy of a Population Code L. F. Abbott Volen Center and Department of Biology, Brandeis University, Waltham, MA 024549110, U.S.A.
Peter Dayan∗ Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, U.S.A.
We study the impact of correlated neuronal firing rate variability on the accuracy with which an encoded quantity can be extracted from a population of neurons. Contrary to widespread belief, correlations in the variabilities of neuronal firing rates do not, in general, limit the increase in coding accuracy provided by using large populations of encoding neurons. Furthermore, in some cases, but not all, correlations improve the accuracy of a population code. 1 Introduction In population coding schemes, the activities of a number of neurons jointly encode the value of a quantity. A frequently touted advantage of population coding is that it suppresses the effects of neuronal variability. The observation of correlations in the trial-to-trial fluctuations of simultaneously recorded neurons (Gawne & Richmond, 1993; Zohary, Shadlen, & Newsome, 1994; Lee, Port, Kruse, & Georgopoulos, 1998) has raised some doubt as to whether this advantage is actually realized in real nervous systems. The dramatic effects of correlated variability can be seen by examining its impact on the average of N neuronal firing rates. When the fluctuations of individual neurons about their mean rates are uncorrelated, the variance of the average decreases like 1/N for large N. In contrast, correlated fluctuations cause the variance of the average to approach a fixed limit as the number of neurons increases. While illustrative, this example is not conclusive because the value of an encoded quantity can be extracted from a population of neurons by methods that do not require averaging their firing rates. Statements in the literature suggest that correlated variability can
∗ Present address: Gatsby Computational Neuroscience Unit, University College, London, Alexandra House, 17 Queen Square, London WCIN 3AR U.K.
Neural Computation 11, 91–101 (1999)
c 1999 Massachusetts Institute of Technology °
92
L. F. Abbott and Peter Dayan
either decrease or increase (Snippe & Koenderink, 1992; Shadlen & Newsome, 1994; Shadlen, Britten, Newsome, & Movshon, 1996; Reike, Warland, de Ruyter van Stevenick, & Bialek, 1996; Oram, Foldiak, Perett, & Sengpiel, 1998; Lee et al., 1998) the accuracy of a population code. The purpose of this article is to clarify this situation by addressing two questions: (1) Does correlation necessarily increase or decrease the accuracy with which the value of an encoded quantity can be extracted from a population of N neurons? (2) Does this accuracy approach a fixed limit as N increases? This issue of correlated variability was first addressed by Johnson (1980b), who discussed circumstances under which correlation is either helpful or harmful for discrimination. Snippe and Koenderink (1992) studied the effect of correlated variability on optimal linear discrimination and also found some cases in which correlation improved discrimination and others in which discrimination was degraded by correlation. We will study the effects of correlation on population coding accuracy by computing the Fisher information (Cox & Hinckley, 1974; Paradiso, 1988; Seung & Sompolinsky, 1993). The inverse of the Fisher information is the minimum averaged squared error for any unbiased estimator of an encoded variable. It thus sets a limit on the accuracy with which a population code can be read out by an unbiased decoding method. Two simple examples illustrate the subtleties involved in analyzing the effects of correlation. Consider a set of N neurons with firing rates ri , for i = 1, 2, . . . , N, which have mean values fi , identical variances σ 2 , and correlated variabilities so that £ ® ¤ (ri − fi )(rj − fj ) = σ 2 δij + c(1 − δij ) ,
(1.1)
with correlation coefficient c satisfying 0 ≤ c < 1. The angle brackets on the left side of this equation denote an average over trials, and δii = 1 for all i while δij = 0 if i 6= j. In this case, the variance of the average of the rates, R=
N 1 X ri , N i=1
(1.2)
is σ2 = R
σ2 [1 + c(N − 1)] . N
(1.3)
This illustrates two negative effects of correlation. For fixed N, the variance increases as a function of the degree of correlation c, and beyond N ≈ 1/c the variance approaches a fixed limit σ 2 → cσ 2 , as discussed in the R opening paragraph. Correlation among the activities of neurons in area MT of monkeys that are viewing moving random dot displays has been estimated at about c = 0.1 − 0.2 (Zohary et al., 1994; Shadlen et al., 1996).
Effect of Correlated Variability
93
This has led to the suggestion that coding accuracy will not improve for populations of more than about 100 neurons (Shadlen & Newsome, 1994). The second example may seem a bit contrived but is nevertheless illustrative. Consider the sign-alternating sum N 1 X (−1)i ri . R˜ = N i=1
(1.4)
The variance of this quantity (for even N) is σR2˜ =
σ2 (1 − c) . N
(1.5)
For this variable, positive correlation always decreases the variance, and the variance is proportional to 1/N whether or not correlation is present. One reason to think that correlation need not always be harmful is that it generally reduces the entropy of the variability in a neural population, suggesting that it should therefore increase the accuracy of a population code. Our results on population coding generally concur with this entropy analysis. For the cases we consider, the lower limit on the averaged squared decoding error provided by the Fisher information is proportional to 1/N for large N, similar to the behavior of equation 1.5, not equation 1.3. For additive or multiplicative noise with uniform correlations, the dependence on the degree of correlation c also resembles that of equation 1.5, and thus correlation improves population coding accuracy. We also consider correlations of limited range for which coding accuracy can display both increasing and decreasing behavior (Snippe & Koenderink, 1992). 2 The Model We consider a population code in which N neurons respond to a stimulus with firing rates that depend on a variable x that parameterizes some stimulus attribute (Johnson, 1980a,b; Georgopoulos, Schwartz, & Kettner, 1986; Paradiso, 1988; Baldi & Heiligenberg, 1988; Snippe & Koenderink, 1992; Seung & Sompolinsky, 1993; Salinas & Abbott, 1994; Snippe, 1996; Sanger, 1996). The activity of neuron i, averaged over trials that use the stimulus x, is fi (x), and its activity on any given trial is ri = fi (x) + ηi .
(2.1)
We interpret this as the number of spikes fired by the neuron over a fixed time period. We do not discuss encoding that involves the fine-scale temporal structure of spike trains. The random terms ηi for i = 1, 2, . . . , N are generated from a gaussian probability distribution with zero mean and covariance matrix Q(x). We consider three different models of variability. In
94
L. F. Abbott and Peter Dayan
the additive noise model (Johnson, 1980b), the covariance matrix is identical to equation 1.1: Qij = σ 2 [δij + c(1 − δij )] .
(2.2)
For multiplicative noise, the variability in the firing rate is still described by equation 2.1, but the covariance matrix is scaled by the average firing rates, Qij (x) = σ 2 [δij + c(1 − δij )] fi (x) fj (x) .
(2.3)
This produces variances that increase as a function of firing rate and larger correlations for neurons with overlapping tuning curves, as seen in the data (Lee et al., 1998). We also consider a model in which the correlations can have an even more dramatic dependence on the distance between tuning curves. This is the limited-range correlation model (Snippe & Koenderink, 1992), with the correlation matrix written as Qij = σ 2 ρ |i−j| ,
(2.4)
where the parameter ρ (with 0 < ρ < 1) determines the range of the correlations between different neurons in the population. The parameter ρ can be expressed in terms of a correlation length L by writing ρ = exp(−1/L),
(2.5)
where 1 is the spacing between the peaks of adjacent tuning curves. We use the notation Q to denote the matrix with elements Qij , and r and f(x) to denote the vectors of firing rates with elements ri and fi (x), respectively. In the additive and limited-range cases, Q does not depend on x, while for multiplicative noise it does. The average firing rates f(x) are the tuning curves of the neurons in the population. We imagine that the tuning curves are arranged to cover a range of x values, with different tuning curves localized to different ranges of x. We assume that the coverage is dense and roughly uniform (we define these terms below), but otherwise leave the exact nature of these tuning curves relatively unrestricted. 3 Fisher Information The Fisher information provides a useful measure of the accuracy of a population code. Through the Cram´er-Rao bound, the Fisher information limits the accuracy with which any unbiased decoding scheme can extract an estimate of an encoded quantity from the activity of a population of neurons. The average value of an unbiased estimate is equal to the true value, x, of the encoded quantity, but the estimate will typically differ from x on a trialto-trial basis. For an unbiased estimate, the average squared decoding error
Effect of Correlated Variability
95
is equal to the variance of these trial-to-trial deviations. The Cram´er-Rao bound states that the average squared decoding error for an unbiased estimate is greater than or equal to 1/IF (x), where IF (x) is the Fisher information. Although in some cases, an biased decoding scheme may outperform a biased method, no biased estimate can do better than the Cram´er-Rao lower bound. To compute the Fisher information, we need to know the conditional probability distribution P[r|x], which determines the probability that a given response r is evoked by the stimulus x. The Fisher information is given in terms of P[r|x] by Z IF (x) = −
drP[r|x]
d2 log P[r|x] . dx2
(3.1)
The maximum likelihood estimator, which chooses for an estimate the value of x that maximizes P[r|x], asymptotically saturates the Cram´er-Rao bound as N → ∞. Thus, the bound sets a realizable limit, making it a good measure of the accuracy of a population code (see Paradiso, 1988; Seung & Sompolinsky, 1993; and Pouget, Zhang, Deneve, & Latham, 1998, for discussions of the use of maximum likelihood (ML) techniques and Fisher information for population codes in the absence of correlation). The psychophysical measure of discriminability d0 that quantifies how accurately discriminations can be made between two slightly different values x and x + 1x based on r is related to the Fisher information by the formula p d0 = 1x IF (x) .
(3.2)
The larger the Fisher information, the better the discriminability and the smaller the minimum unbiased decoding error. When the random η terms are generated from a gaussian probability distribution as discussed above, · ¸ 1 1 exp − [r − f(x)]T Q−1 (x)[r − f(x)] , (3.3) P[r|x] = p 2 (2π)N det Q(x) where T stands for the transpose operation. This equation does not give zero probability for negative firing rates, but we assume that the means and variances are such that this has a small effect. Substituting equation 3.3 into equation 3.1, we find (see, for example, Kay, 1993), i 1 h IF (x) = f 0 (x)T Q−1 (x)f 0 (x) + Tr Q0 (x)Q−1 (x)Q0 (x)Q−1 (x) 2
(3.4)
where Tr stands for the trace operation, Q0 (x) =
dQ(x) dx
and
f 0 (x) =
df(x) . dx
(3.5)
96
L. F. Abbott and Peter Dayan
When Q is independent of x, as it is for additive noise and limited-range correlations, this reduces to IF (x) = f 0 (x)T Q−1 f 0 (x) .
(3.6)
Equations 3.4 and 3.6 are the basis for all our results. To apply them, we need the inverses of the covariance matrices, which are, in the additive case, [Q−1 ]ij =
δij (Nc + 1 − c) − c 2 σ (1 − c)(Nc + 1 − c)
;
(3.7)
in the multiplicative case, [Q−1 (x)]ij =
δij (Nc + 1 − c) − c ; fi (x) fj (x)σ 2 (1 − c)(Nc + 1 − c)
(3.8)
and in the case of correlations with limited range, · ¸ ¢ ρ ¡ 1 + ρ2 δij − δi+1,j + δi−1,j . [Q ]ij = 2 σ (1 − ρ 2 ) 1 + ρ2 −1
(3.9)
4 Results 4.1 Additive Noise. The Fisher information in the additive case is computed by substituting the correlation matrix (2.2) into equation 3.6 and doing the necessary algebra. The result depends on two sums, 1 X ¡ 0 ¢2 fi (x) F1 (x) = N i
à and
F2 (x) =
1 X 0 f (x) N i i
!2 .
(4.1)
These have been scaled to be of order one for the case of uniform tuning curve placement. In terms of these quantities, IF (x) =
cN2 [F1 (x) − F2 (x)] + (1 − c)NF1 (x) . σ 2 (1 − c)(Nc + 1 − c)
(4.2)
As N tends to infinity, IF (x) →
N[F1 (x) − F2 (x)] , σ 2 (1 − c)
(4.3)
which grows with N and c provided that F1 (x) − F2 (x) > 0. Note that aside from the factor of F1 (x) − F2 (x), the inverse of this expression matches the variance of equation 1.5. For large N, a uniform array of tuning curves
Effect of Correlated Variability
97
should generate functions F1 (x) and F2 (x) that are insensitive to the values of either x or N (indeed, this is our definition of uniform tuning curve placement). Tuning curve arrays that are symmetric with respect to the sign flip x → −x (that is, for every neuron with tuning curve f (x) there is another neuron with tuning curve f (−x)) have F2 (x) = 0. In the additive noise case, the inverse of the Fisher information, which determines the minimum unbiased decoding error, decreases as 1/N for large N, and also decreases as a function of c, the degree of correlation. The Fisher information diverges, and the minimal error goes to zero as c approaches one. As c → 1, any slight difference in the tuning curves can be exploited to calculate the noise exactly and remove it. In their article in this issue, Zhang and Sejnowski note an interesting scaling property of the Fisher information that also appears in our results. They considered the simultaneous encoding of D variables by a population of neurons and studied the effect of changing tuning curve width on encoding accuracy. If the widths of the tuning curves of the encoding population are scaled by a parameter w, we expect F1 to scale like wD−2 . The factor of wD reflects the number of responding neurons, while the factor w−2 is due to the squared derivative. For simplicity, we take F2 = 0. Then the Fisher information satisfies IF ∝ NwD−2 /σ 2 in agreement with the results of Zhang and Sejnowski. The Fisher information we have computed increases as a function of c and N unless F1 (x)−F2 (x) = 0 or F1 (x)−F2 (x) → 0 for large N. By the CauchySchwartz inequality, F1 (x) ≥ F2 (x). For large N, F1 (x) − F2 (x) could go to zero if both F1 (x) → 0 and F2 (x) → 0. We define the tuning curve coverage as being dense if F1 (x) 6→ 0 for any x, since this implies that as more neurons are added, a fixed fraction respond significantly to x. By the condition for equality in the Cauchy-Schwartz inequality, the other alternative, F1 (x) = F2 (x), requires fi0 (x) to be independent of i or, equivalently, fi (x) = p(x) + qi
(4.4)
for any function p and numbers qi . Thus, the Fisher information will fail to grow as a function of c and N only if there is an additive separation of dependency between the value x and the index i. This means that correlation is harmful only when all the neurons share the same tuning dependence on x. This is not normally the case since neurons almost always have some variability in their stimulus preferences. Of course, we must assume that the mechanism that reads out the encoded quantity takes advantage of this variability and does not simply perform an averaging operation. 4.2 Multiplicative Noise. When Q(x) is given by equation 2.3, the Fisher information defined by equation 3.4 depends on the logarithmic derivatives
98
L. F. Abbott and Peter Dayan
of the average firing-rate tuning curves g0i (x) =
d ln fi (x) 1 dfi (x) = . fi (x) dx dx
(4.5)
In particular, it depends on the sums 1 X ¡ 0 ¢2 gi (x) G1 (x) = N i
à and
G2 (x) =
1 X 0 g (x) N i i
!2 (4.6)
and is given by cN2 [G1 (x) − G2 (x)] + (1 − c)NG1 σ 2 (1 − c)(Nc + 1 − c) 2 [N c(2 − c) + 2N(1 − c)2 ]G1 (x) − c2 N2 G2 (x) . + (1 − c)(Nc + 1 − c)
IF (x) =
(4.7)
For large N, this approaches the limit IF (x) →
N[G1 (x) − G2 (x)] N[(2 − c)G1 (x) − cG2 (x)] + . σ 2 (1 − c) (1 − c)
(4.8)
The Fisher information for multiplicative noise contains one term that depends on the noise variance σ 2 and another term that, surprisingly, is independent of σ 2 . This second term arises because, with multiplicative noise, the encoded variable can be estimated from second-order quantities, not merely from measurements of the firing rates themselves. The Fisher information of equation 4.8 is proportional to N and is an increasing function of c provided that G1 (x) > G2 (x). Since G1 (x) ≥ G2 (x) by the Cauchy-Schwartz inequality, the only way to modify this behavior is if G1 (x) = G2 (x). This condition holds only if g0i (x) is independent of i or, equivalently, if fi (x) = p(x)qi + r(x) + si
(4.9)
for any functions p and r and numbers qi and si . This is multiplicative separability rather than the additive separability of equation 4.4. As in the case of additive noise, the Fisher information with multiplicative noise increases with correlation and grows linearly with N unless a contrived set of tuning curves is used. 4.3 Limited-Range Correlations. In both of the cases we have studied thus far, the accuracy of the population code, quantified by the Fisher information, increases as a function of both N and c. Although the linear
Effect of Correlated Variability
99
dependence of the Fisher information on N appears quite general, there are cases in which introducing correlation decreases rather than increases IF (Johnson, 1980b; Snippe & Koenderink, 1992). One example is provided by the limited-range correlations described by the matrix of equation 2.4. The Fisher information for this case is IF (x) =
N(1 − ρ)F1 (x) N1−2/D ρF3 (x) + , σ 2 (1 + ρ) σ 2 (1 − ρ 2 )
(4.10)
where F1 is as given above, D is the number of encoded variables, and (provided that x is away from the edges of the range covered by the population tuning curves), F3 (x) = N2/D−1
N ¡ X
0 (x) − fi0 (x) fi+1
¢2
.
(4.11)
i=1
The power of N in the definition F3 (x) is chosen so that it is independent of N for uniform tuning curve coverage. As N gets large, the distance between neighboring tuning curves decreases as N−1/D , and the difference between their derivatives is proportional to this factor. For a fixed value of N, the Fisher information in equation 4.10 is a nonmonotonic function of the parameter ρ that determines the range and degree of the correlations. The first term in equation 4.10 is a decreasing function of ρ and hence of L, the correlation length introduced in equation 2.5, while the second term has the opposite dependence. For a fixed N value, the first term dominates for small L, and the second dominates for large L. For any finite value of D, the first term in equation 4.10 will dominate for large N, so as N → ∞, IF (x) →
N(1 − ρ)F1 (x) . σ 2 (1 + ρ)
(4.12)
Note that this limit is approached more rapidly for small D than for large D. The expression in equation 4.12 is a decreasing function of ρ, so in this case, unlike the additive and multiplicative cases, increasing correlation decreases the Fisher information. However, the Fisher information still grows linearly with N for any ρ < 1. The only way to prevent the Fisher information from growing with N is to force ρ nearer and nearer to 1 as N → ∞. For example, if ρ = c1/N , for 0 < c < 1, the Fisher information tends to a constant as N → ∞. 5 Discussion We have studied how correlations between the activities of neurons within a coding population affect the accuracy with which an encoded quantity
100
L. F. Abbott and Peter Dayan
can be determined or discriminated (Johnson, 1980b). We find that, generically, correlations between units do not prevent the Fisher information from growing linearly with the number of encoding neurons, and correlations can either improve or degrade decoding accuracy depending on the form of the correlation matrix. Only in the limit as the correlations get very close to 1 can this behavior change in some circumstances. Since our results are based on the Fisher information, they apply only to unbiased estimates. However, biased estimates would presumably be used only to improve accuracy further, and thus the increase in accuracy with N would not be destroyed by using a biased estimate. Thus, optimal population-based estimates do not suffer from the limitations that correlation imposes on estimates of average firing rates. Although averaging can be used to obtain more accurate firingrate estimates from a group of neurons, it does not improve the accuracy of a population decoding procedure. There are nevertheless possible lacunae in our analysis. We considered only relatively simple noise models. We also used noise with gaussian statistics. Poisson noise would be an obvious alternative and would entail slightly different calculations. Finally, we did not consider the computational complexity or biological implementation of the optimal decoding algorithms, although a good point of departure would be the work of Pouget et al. (1998) showing how to perform ML inference using a recurrent network in the case without correlations. Correlations could make the implementation of an optimal decoding procedure more difficult. The most relevant requirement for retaining the improved accuracy provided by large populations of encoding neurons is that the neurons should have different selectivities to the quantity they are jointly encoding. In particular, their tuning curves must not be additively or multiplicatively separable. Tuning functions that are commonly adopted in modeling work and seen in biological systems do not appear to have these problematic features. Acknowledgments We are grateful to Peter Latham, Alex Pouget, Sebastian Seung, Michael Shadlen, and Haim Sompolinsky for discussions and comments. Research was supported by the National Science Foundation (DMS-9503261) and the W. M. Keck Foundation for L. F. A, and by the National Institute of Mental Health (MH-55541) and the National Science Foundation (IBM-9634339) for P. D. References Baldi, P., & Heiligenberg, W. (1988). How sensory maps could enhance resolution through ordered arrangements of broadly tuned receivers. Biological Cybernetics, 59, 313–318.
Effect of Correlated Variability
101
Cox, D. R., & Hinckley, D. V. (1974). Theoretical statistics. London: Chapman & Hall. Gawne, T. J., & Richmond, B. J. (1993). How independent are the messages carried by adjacent inferior temporal cortical neurons? Journal of Neuroscience, 13, 2758–2771. Georgopoulos, A. P., Schwartz, A. B., & Kettner, R. E. (1986). Neuronal population coding of movement direction. Science, 243, 1416–1419. Johnson, K. O. (1980a). Sensory discrimination: Decision process. Journal of Neurophysiology, 43, 1771–1792. Johnson, K. O. (1980b). Sensory discrimination: Neural processes preceding discrimination decision. Journal of Neurophysiology, 43, 1793–1815. Kay, S. M. (1993). Fundamentals of statistical signal processing: Estimation theory. Englewood Cliffs, NJ: Prentice-Hall. Lee, D., Port, N. L., Kruse, W., & Georgopoulos, A. P. (1998). Variability and correlated noise in the discharge of neurons in motor and parietal areas of the primate cortex. Journal of Neuroscience, 18, 1161–1170. Oram, M. W., Foldiak, P., Perett, D. I., & Sengpiel, F. (1998). The “ideal homunculus”: Decoding neural population signals. Trends in Neuroscience, 21, 259–265. Paradiso, M. A. (1988). A theory for the use of visual orientation information which exploits the columnar structure of striate cortex. Biological Cybernetics, 58, 35–49. Pouget, A., Zhang, K., Deneve, S., & Latham, P. (1998). Statistically efficient estimation using population coding. Neural Computation, 10, 2–8. Reike, F., Warland, D., de Ruyter van Steveninck, R., & Bialek, W. (1996). Spikes: Exploring the neural code. Cambridge MA: MIT Press. Salinas, E., & Abbott, L. F. (1994). Vector reconstruction from firing rates. Journal of Computational Neuroscience, 1, 89–107. Sanger, T. D. (1996). Probability density estimation for the interpretation of neural population codes. Journal of Neurophysiology, 76, 2790–2793. Seung, H. S., & Sompolinsky, H. (1993). Simple models for reading neuronal population codes. Proc. Natl. Acad. Sci. USA, 90, 10749–10753. Shadlen, M. N., Britten, K. H., Newsome, W. T., & Movshon, J. A. (1996). A computational analysis of the relationship between neuronal and behavioral responses to visual motion. Journal of Neuroscience, 16, 1486–1510. Shadlen, M. N., & Newsome, W. T. (1994). Noise, neural codes and cortical organization. Current Opinion in Neurobiology, 4, 569–579. Snippe, H. P. (1996). Parameter extraction from population codes: A critical assessment. Neural Computation, 8, 511–530. Snippe, H. P., & Koenderink, J. J. (1992). Information in channel-coded systems: Correlated receivers. Biological Cybernetics, 67, 183–190. Zohary, E., Shadlen, M. N., & Newsome, W. T. (1994). Correlated neuronal discharge rate and its implications for psychophysical performance. Nature, 370, 140–143. Received December 19, 1997; accepted May 5, 1998.
LETTER
Communicated by Jack Cowan
A Neural Network Model of Temporal Code Generation and Position-Invariant Pattern Recognition Dean V. Buonomano Michael Merzenich Keck Center for Integrative Neuroscience, University of California, San Francisco, San Francisco, CA 94143, U.S.A.
Numerous studies have suggested that the brain may encode information in the temporal firing pattern of neurons. However, little is known regarding how information may come to be temporally encoded and about the potential computational advantages of temporal coding. Here, it is shown that local inhibition may underlie the temporal encoding of spatial images. As a result of inhibition, the response of a given cell can be significantly modulated by stimulus features outside its own receptive field. Feedforward and lateral inhibition can modulate both the firing rate and temporal features, such as latency. In this article, it is shown that a simple neural network model can use local inhibition to generate temporal codes of handwritten numbers. The temporal encoding of a spatial patterns has the interesting and computationally beneficial feature of exhibiting position invariance. This work demonstrates a manner by which the nervous system may generate temporal codes and shows that temporal encoding can be used to create position-invariant codes. 1 Introduction Experimental (McClurkin, Optican, Richmond, & Gawne,1991; Middlebrooks, Clock, Xu, & Green, 1994; Bialek, Rieke, Ruyter van Steveninck, & Warland, 1991) and theoretical (Hopfield, 1995; Thorpe & Gautrais, 1997) studies have suggested that the nervous system may encode information in the temporal structure of neuron spike trains, generally referred to as temporal coding. For example, McClurkin et al. (1991) have shown that by taking into account the temporal structure of neuronal responses to Walsh patterns, there is more information about the stimuli than there is in the firing rate alone. However, in addition to showing that there is information in the temporal structure of spike trains, at least two additional issues relating to temporal encoding must be addressed: (1) How does sensory information come to be temporally encoded? (2) How is temporally coded information used or “decoded” by the nervous system? Here we focus on the first question and consider the potential advantages of encoding information in the temporal domain. Neural Computation 11, 103–116 (1999)
c 1999 Massachusetts Institute of Technology °
104
Dean V. Buonomano and Michael Merzenich
In principle, any of the many neuronal properties that affect the balance of excitation and inhibition can produce significant changes in the temporal structure of neuronal responses. If the temporal structure is to contain information about a given stimulus, it should be reproducible and stimulus specific, and should be amenable to stimulus generalization. Local inhibitory interactions, which include both feedforward and lateral inhibition, may provide a neural mechanism that satisfies these conditions. Local inhibition is an almost ubiquitous feature of neuronal circuits and typically has been thought to be a means of preventing cells from firing or of modulating the average firing rate of neurons. However, inhibition can also shape the temporal structure of responses by modulating when a neuron will reach threshold. As a result of local inhibition, both the firing rate and temporal structure of neuronal responses can be significantly altered by neighboring neurons and may contain information not only about their own receptive fields but also about the local spatial structure of stimuli. Changes in the temporal structure of neuronal responses can be manifested in various degrees of complexity. The simplest feature that may be changed, from both an encoding and decoding perspective, is spike latency, which we will consider here. One possible advantage of temporal encoding is that the amount of information that can be potentially transmitted on a per-spike basis is larger than that transmitted by a rate code. An alternate or additional advantage of temporal coding is that once spatial information is temporally encoded, it can potentially represent spatial patterns in a position-invariant manner. Position invariance has proved to be an extremely challenging problem, from the perspective of understanding how the brain solves it and in developing artificial systems capable of invariant pattern recognition. In their simplest forms, conventional neural networks do not exhibit position invariance because information is stored in the spatial pattern of synaptic strengths. Figure 1 provides a schematic illustration of why conventional neural networks are not generally well suited to solve position invariance. If a unit is to behave as a + detector, the connection strengths of the weight matrix are distributed in a fashion that spatially reflects the + symbol. If the + is shifted to different positions on the input layer, the + detector may develop responses to other stimuli. To circumvent this problem, various biologically plausible models have been proposed that are capable of exhibiting different forms of invariant pattern recognition. One approach has been to develop large-scale multilayer networks, in which each layer exhibits position invariance to higher-order features (Fukushima, 1988). A second approach has been to develop networks capable of dynamically changing the local connectivity (Olshausen, Anderson, & Van Essen, 1993; Konen, Maurer & von der Malsburg, 1994), or the local gain of the network (Salinas & Abbott, 1997), thus essentially accomplishing online translation and scaling of images. Here we develop an alternative hypothesis based on temporal coding.
Neural Network Model
105
Figure 1: Schematic of the position-invariant pattern recognition problem. (A) In a conventional one-layer network, the creation of a “plus” detector essentially consists of connecting the spatial arrangement of units activated by the + to a plus-detector output unit. (B) If the same plus detector is also going to detect a + in a different position, such as the lower left corner, the units activated by that + will also be connected to the plus detector. In the process, other patterns, such as a square (bold outline), will also activate the plus detector.
2 Methods Figure 2 schematizes the type of feedforward, surround inhibition used in the model. Figure 3 shows a simple network that captures the basic principles of our model. Below we explain in detail the more complex network used for handwritten digit recognition. Each unit from the input layer (the “retina”) provides an excitatory input to the topologically equivalent position in the feature detector layer (analogous to L-IV of V1), and inhibition to the neighboring units in a surround inhibition pattern. Each point in the feature detector layer had four types of line detectors (vertical, horizontal, and two diagonals). Each feature detector unit was activated if five appropriately aligned input units were on. The feature detector layer provided input to the “cortical” layer (analogous f to L-II/III of V1). The voltage (Vi ) of a cortical unit in position i, and of
106
Dean V. Buonomano and Michael Merzenich
Figure 2: Schematic of the circuit used for feedforward, center surround inhibition. Units from the input layer provide excitatory input to the topologically equivalent unit in the next layer and inhibitory connections to the neighboring units.
orientation f , is given by f
f
f
Vi (t) = Vi (t − 1) + KEx Ii −
F INH X X l6= f
j
Ijl · Wij +
EX X k6=i
f
Ik · Wik .
(2.1)
f
Ii represents the binary input from the unit in position i and of orientation f from the feature detector layer. The third term represents the inhibition, which is implemented in a cross-orientation manner; for example, each vertical unit receives inhibition from the horizontal and two diagonal input layers. The weights, Wij , are a function of the distance from unit i and of the difference in orientation preference between the units: Wij = KF · exp(−ki − jk) · KInh ,
(2.2)
where KF = 1 when the orientation of i and j differs by 90 degrees and KF = 0.66 when they differ by 45 degrees. The fourth term in equation 2.1 represents iso-orientation excitation; a vertical unit will excite neighboring
Neural Network Model
107
Figure 3: A simple retinal model with inhibition can create position-invariant temporal codes. (A) The first column of each row represents the input pattern— either a + or a T at two different positions. The subsequent columns represent frames of the voltage of each element in time. Voltage is represented on gray scale. A unit in white represents those that have reached threshold. Since we are interested in onset latency, what happens after a unit reaches threshold is not relevant, and it is assumed that each element stays on. The time step at which threshold was reached defines the latency of that unit. The latency histogram produced by each image is represented in the last column. The yaxis represents the number of activated units at a given latency. (B) Schematic diagram of a network that could use latency histograms for position-invariant pattern recognition (empty and filled bars represent + and T, respectively). The latency code is used as spatial code in which each unit corresponds to a latency and serves as the input to the recognition network.
vertical units that are vertically aligned, but not vertical units that are horizontally aligned with it. For excitatory weights Wij = C · 0.04, where C is 1.0, 0.67, or 0.0, when j is positioned above or below, diagonal to, or lateral to
108
Dean V. Buonomano and Michael Merzenich
i, respectively, for a vertical unit. Such iso-orientation topography has been reported experimentally (Fitzpatrick, 1996). Threshold was arbitrarily set at 1, and V(0) = 0. One important parameter is the radius of the of the feedforward inhibition. If the radius is too small in relation to the size of the image, all local interactions are likely to be expressing local noise rather than the relevant structure. Conversely, a large radius will tend to “normalize” all responses according to overall activity, and not express any local structure. In the current simulations, an inhibition radius of 3 was found to be optimal. Within a fairly wide range, changes in the excitation KEx and inhibition KInh constants did not dramatically affect performance as long as inhibition was strong enough to modulate the latency. In the simulations presented here, the values of KEx and KInh were 0.26 and 0.002, respectively.
2.1 Recognition Network. The second component of the model consisted of a recognition network, which was necessary in order to determine whether the temporal codes generated could actually be used for positioninvariant pattern recognition. For this purpose, it is necessary to decode the temporal code. Decoding temporal codes is critical not only when the nervous system may use internally generated temporal codes, but in temporal processing in general, an important and difficult problem faced by the brain and artificial networks processing time-varying signals, such as speech. Presenting a realistic model of temporal processing is far beyond the scope of this article. Indeed, it is because the decoding of complex temporal codes can easily become intractable that we chose to analyze latency of the first spike. To decode temporal information—that is, transform a temporal code into a spatial code—we assumed the presence of delay lines (e.g., Tank & Hopfield, 1987; Waibel, 1989), in which hypothetical delays are used to generate elements sensitive to temporal features, specifically latency. In practice, each of the latency histograms represents the output of the delay line model. Each bin represents the output of a single element tuned to a given delay (schematized in Figure 3b). The next step was to determine whether once the latency codes are mapped spatially, they can be used for digit recognition. For this purpose, the output of the delay line network was used in a conventional backpropagation network (Rumelhart & McClelland, 1986). Since each orientation sublayer generates a temporal code, and only 10 different latencies were considered, the input to the backpropagation network consisted of a single vector of length 40. The backpropagation network contained 16 hidden units and 10 output units (one for each numeral). The handwritten digit database was obtained from the National Institute of Standards and Technology. The samples used here were from the Handprinted Character Database, subdirectory f13/fl3/data/f0035, which
Neural Network Model
109
comprise characters from a single writer. The raw images were used; that is, no scaling or normalization was performed. 3 Results Figure 3 illustrates a simple network that captures the fundamental properties of our model, in which temporal codes of images are generated with the use of local inhibition. These codes can be then used by a second network for position-invariant pattern recognition. In this simplified network, a singlelayer network of linear-threshold units was used. Each “pixel” of the input stimulus excites a unit in the corresponding position of the network and inhibits neighboring units (see Figure 2). During each time step, the sum of excitation and inhibition was computed. If the sum reaches threshold, a spike occurs. In the simulations considered here, only the first spike is relevant; once a unit has fired, it remains on. The latency is defined as the time step at which threshold was reached. In the simulation shown in Figure 3A, two symbols (T and +), each composed of a single vertical and horizontal line, were presented to the network, each at two different locations. For the +, the four extremes reached threshold first and fired at t = 2. These four units fired first because each receives inhibition from only one inhibitory input. The center unit will be the last to fire at t = 8, because it is inhibited by four neighboring units. In contrast for the T, three extreme units will fire first, followed by their neighbors, until the innermost unit fires. The plots on the right of Figure 3A show the latency histograms of the network. Note that the latency histograms are distinct for the T and + and are independent of position. Figure 3B illustrates how the latency distributions could be used for position-invariant pattern recognition. Using a traditional neural network architecture, the latency histograms serve as the input patterns, and each output unit becomes a pattern detector. An implicit step in this process is to transform the temporal code into a spatial code. For example, this can be accomplished using tapped delay lines. To determine whether neural networks that generate temporal codes can be used to recognize real-world patterns, we developed a network and tested it with handwritten digits placed in different locations on the input layer (see Figure 4). The network consisted of an input layer, a feature detection layer, and a “cortical” layer. The feature detection and cortical layer each contained a complete topographic representation of four different orientations. The feature detection and cortical layer can be best visualized as each containing four distinct sublayers, one for each orientation: vertical, horizontal, and two diagonal sublayers. Lateral interactions take place in the cortical layer in the form of cross-orientation inhibition and iso-orientation excitation. Figure 5 shows the result of a simulation. The input pattern activates the appropriate units on the vertical, horizontal, and diagonal feature detector layer. The feature detector layer projects to the cortical layer, in which the
110
Dean V. Buonomano and Michael Merzenich
Figure 4: Handwritten digits used to test position-invariant pattern recognition. Digits were obtained from a National Institute of Standards and Technology database in a 32 × 32 pixel format. Each digit was placed at random location on a 64 × 64 input layer in order to test for position invariance.
total inhibitory and excitatory input is computed for each unit on each time step. For visualization purposes, the cells of the same orientation preference are shown as a separate sublayer. This is similar to looking down on V1 and looking at all horizontal, vertical, and diagonal cells separately. Each of the four cortical sublayers generates a latency histogram, which represents how many units fired at each time step (only the first spike is relevant). To determine whether these temporal codes latency histograms are sufficient to code for all digits in a position-independent manner and if it could generalize across samples, we presented 100 handwritten digits to the network (see Figure 4). Each 32 × 32 digit image was placed at a random location on the 64 × 64 input layer. Each of the 100 sets of four latency histograms served as an input vector to the recognition network, which was a standard backpropagation network. Figure 6A (upper panel) shows an example of the 50 latency histograms used for testing. The latency histograms for each of the sublayers are placed in a single row. The recognition network was
Neural Network Model
111
Figure 5: Simulation of digit recognition. An image is presented to the input layer, which is preprocessed by a feature detector layer. The feature detector layer has four types of orientation detectors: vertical, horizontal, and two diagonal lines. For visualization, each type of detector is presented as a separate sublayer. The feature detector layer then projects to a “cortical” layer, which also contains a representation of each orientation. Units in the feature detector layer inhibit units in the cortical layer in a cross-orientation fashion and excite them in an iso-orientation fashion. The voltage in the cortical units is shown at t = 10. When and how many cells reached threshold at a given time step is shown in the latency histograms on the right.
trained on 50 latency histograms (5 of each digit), and then tested on the remaining 50 patterns. The lower panel in Figure 6A shows the output of the recognition network in response to the 50 test latency histograms. In this run, 48 of 50 digits were correctly classified. The average performance was 93.4 ± 0.16%. It is important to determine that the performance of the network is de-
112
Dean V. Buonomano and Michael Merzenich
Figure 6: (A) Latency histograms for the 50 digits used to test network performance represented in a gray-scale code (upper panel). A black point means that no units fired at that latency in response to a given digit; white means that a maximal number of units fired at that latency. The lower panel represents and the output of the backpropagation network in response to the 50 test digits, after training on the 50-digit learning set. (B) Control histograms in which temporal information is collapsed (upper panel) and output of the backpropagation network trained on the control stimulus set.
pendent on temporal information rather than on spatial information. Since there are four different types of feature detectors, there is information contained in the total number of spikes from each feature detector sublayer irrespective of temporal structure. For example, since the number 1 essentially corresponds to the vertical feature detector, it is unlikely that temporal information is contributing to its recognition. To examine the contribution of the latency code, temporal information was removed by collapsing each of the four latency histograms into a single time bin and training the same recognition network on the collapsed input vectors. The upper panel in Figure 6B shows the input patterns for the 50 test stimuli when the latency histograms are collapsed across each of the four cortical sublayers. The lower panel in Figure 6B shows the output of the pattern recognition networks.
Neural Network Model
113
In this example, 33 of 50 of the digits were correctly classified. Note that, as expected, the exemplars of digit 1 were well classified based solely on spatial information. On average, performance was below 70% in the absence of temporal information. In neural network models, it is generally important to determine how noise affects the performance of the network. Both extrinsic and intrinsic noise sources should be considered. Extrinsic noise refers to the noise or variability of the stimulus set. Since we have used a real-world stimulus set, the model presented here clearly performs well with the inherent variability generated by a single writer (see Figure 4). To provide some insight as to the performance of the network in the presence of intrinsic noise, we assumed that all elements of the cortical network exhibited some level of spontaneous activity. Assuming the time steps in our simulations correspond to approximately 1 ms, we examined performance in the presence of 1 Hz and 10 Hz spontaneous activity. At 1 Hz noise, there was a small drop in the performance to 87.44 ± 0.36%. At 10 Hz, there was a large drop in performance to 60.3 ± 0.77%. We should stress that it is difficult to compare the effect of noise on simple artificial networks with that of biological networks. Sources of intrinsic noise have various biological sources, including synaptic and membrane potential variability. Even when these data are available, it is difficult to apply them to simple models such as that presented here in which arbitrary discrete time steps are used. Additionally, there is generally a trade-off between noise levels and the size of the network; thus, the behavior of small networks is generally more sensitive to noise. 4 Discussion The results described here show that by using temporal information generated by local inhibition, it was possible to create a network that classified handwritten digits in a position-invariant fashion. The temporal codes generated for each pattern were used to train the recognition network; half the stimuli were used for training and half for testing. After training, the network generalized appropriately to the test stimuli, comprising both different digit exemplars and positions. The ability to recognize different handwritten exemplars indicates that the temporal codes are sufficiently specific to code for the 10 different digits, yet robust enough to generalize over the intrinsic variability of the digits. Our stimulus set was from a single writer; stimulus sets from multiple writers will decrease the performance of the simple network presented here. Good performance of the network was obtained despite a simple implementation; specifically, only four feature detectors were used. There are a few intrinsic limitations of using simple feature detectors with 180-degree symmetry. For example, the network cannot distinguish between the same image rotated by 180 degrees, since precisely the same units will be activated. Nevertheless, the network correctly classified most instances of 6
114
Dean V. Buonomano and Michael Merzenich
and 9 because of distinct local features intrinsic to the handwritten samples. (See the upper left panel of Figure 6. More vertical elements are activated by 9. Additionally, most are activated on time step 3, whereas more vertical elements are activated at time step 4 for the 6.) More sophisticated implementations of the model could be custom designed for specific pattern recognition problems by incorporating more and higher-order sets of feature detectors. It is the presence of local interactions in general, rather than the specific characteristics of the connectivity, that underlies the generation of temporal codes. In other words, the model does not rely on the specific assumptions of cross-orientation inhibition and iso-orientation excitation. We propose that one of the functions of local interactions in neural circuits may be to generate temporal codes. Such local circuit interactions would represent a simple, widespread, and biologically plausible mechanism by which the nervous system could encode information by extending it into the temporal domain. Indeed, the temporal structure of neurons has been reported to contain a significant amount of information (McClurkin et al., 1991; Middlebrooks et al., 1994; Bialek et al., 1991). In the current model, we have simplified the problem of both generating and analyzing temporal codes by focusing on the latency of the first spike (see also Thorpe and Gautrais, 1997). We suspect that the same circuitry will generate more complex temporal codes that are likely to increase the performance of the network and make it less sensitive to intrinsic noise; however, more enhance the richness of the temporal code, but would require a more sophisticated decoding stages would also likely become necessary. Furthermore, analysis of latency is reasonable since it may be one of the most important temporal parameters (Richmond, Optican, & Spitzer, 1990; Tov´ee, Rolls, Treves, & Bellis, 1993; Gawne, Kjaer, & Richmond, 1996). Here we have shown that temporal coding may emerge from simple and well-established network characteristics. Furthermore, we have suggested that position invariance may be a reason for which it is beneficial to encode information in the temporal domain. Our model, which relies on local circuit interactions, establishes a biologically plausible mechanism to generate temporal codes that have been proposed to contribute to information processing (Hopfield, 1995). Thorpe and Gautrais (1997) have also proposed that different spike latencies, in their model generated as a function of contrast, could be used to generate temporal codes that could be used for pattern recognition. We should emphasize two potential limitations of the hypothesis presented here. First, the mechanisms proposed here cannot be solely responsible for position invariance; clearly, position information is not collapsed across large portions of retinal position in the early stages of visual processing. However, the temporal codes generated by lateral interactions could to contribute to position invariance, particularly on small scales in a multistage process. Second, other stimulus features, such as contrast, that modu-
Neural Network Model
115
late spike latency could easily confound temporal codes, although temporal codes generated at higher levels of visual processing or normalization mechanisms could be used to overcome this problem. One of the predictions that emerges from our hypothesis is that the response latency or temporal structure of neurons to simple images such as + and T (see Figure 1) should be different. Since the differences in temporal structure arise from feedforward and lateral interactions, they are likely to be more robust in higher-order versus primary visual areas. In addition to the problem of temporal encoding, a critical issue that remains to be addressed, if the nervous system uses temporal codes, is that of decoding. We did not address this issue here, and a simple version of delay lines was used in decoding the temporal patterns. However, delay lines are unlikely to account for temporal decoding, and temporal processing in general, particularly for more complex temporal patterns. Networks that rely on local circuit dynamics and short-term forms of plasticity may provide a more biological mechanism to decode temporal information (e.g., Buonomano & Merzenich, 1995), but future experimental and theoretical research must focus on how the brain decodes temporal information, in addition to temporal encoding of information.
Acknowledgments The work was supported by ONR grant N00014-96-1-0206. We thank C. deCharms, M. Kvale, H. Mahncke, Jennifer Raymond, F. Theunissen, and members of the Lisberger lab for helpful discussions and comments of an earlier version of this article.
References Bialek, W., Rieke, F., Ruyter van Steveninck, R. R., & Warland, D. (1991). Reading a neural code. Science, 252, 1854–1857. Buonomano, D. V., & Merzenich, M. M. (1995). Temporal information transformed into a spatial code by a neural network with realistic properties. Science, 267, 1028–1030. Fitzpatrick, D. (1996). The functional organization of local circuits in visual cortex—Insights from the study of tree shrew striate cortex. Cerebral Cortex, 6, 329–341. Fukushima, K. (1998). Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural Networks, 1, 119–130. Gawne, T. J., Kjaer, T. W., & Richmond, B. J. (1996). Latency—Another potential code for feature binding in striate cortex. J. Neurophysiol., 76, 1356–1360. Hopfield, J. J. (1995). Pattern recognition computation using action potential timing for stimulus representation. Nature, 376, 33–36.
116
Dean V. Buonomano and Michael Merzenich
Konen, W. K., Maurer T., & von der Malsburg, C. (1994). A fast dynamic link matching algorithm for invariant pattern recognition. Neural Networks, 7, 1019–1030. McClurkin, J. W., Optican, L. M., Richmond, B. J., & Gawne, T. J. (1991). Concurrent processing and complexity of temporally encoded neuronal messages in visual perception. Science, 253, 675–677. Middlebrooks, J. C., Clock, A. E., Xu, L., & Green, D. M. (1994). A panoramic code for sound location by cortical neurons. Science, 264, 842–844. Olshausen, B. A., Anderson, C. H., & Van Essen, D. V. (1993). A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. J. Neurosci., 13, 4700–4719. Richmond, B. J., Optican, L. M., & Sptizer, H. (1990). Temporal encoding of twodimensional patterns by single units in primate visual cortex. I. Stimulusresponse relations. J. Neurophysiol., 64, 351–368. Rumelhart, D. E., & McClelland, J. L. (1986). Parallel distributed processing. Cambridge, MA: MIT Press. Salinas, E., & Abbott L. F. (1997). Invariant responses from attention gain fields. J. Neurophysiol., 77, 3267–3272. Tank, D. W., & Hopfield, J. J. (1987). Neural computation by concentrating information in time. Proc. Natl. Acad. Sci., 84, 1896–1900. Thorpe, S. J., & Gautrais, J. (1997). Rapid visual processing using spike asynchrony. In M. C. Mozer, M. I. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems, 9 (pp. 901–907). Cambridge, MA: MIT Press. Tov´ee, M. J., Rolls, E. T., Treves, A., & Bellis, R. P. (1993). Information encoding and the responses of single neurons in the primate temporal visual cortex. J. Neurophysiol., 70, 640–654. Waibel, A. (1989). Modular construction of time-delay neural networks for speech recognition. Neural Comp., 1, 39–46.
Received October 10, 1997; accepted May 7, 1998.
LETTER
Communicated by Misha Tsodyks
Probabilistic Synaptic Transmission in the Associative Net Bruce Graham David Willshaw Centre for Cognitive Science, University of Edinburgh, Edinburgh EH8 9LW, Scotland, U. K.
The associative net model of heteroassociative memory with binary-valued synapses has been extended to include recent experimental data indicating that in the hippocampus, one form of synaptic modification is a change in the probability of synaptic transmission. Pattern pairs are stored in the net by a version of the Hebbian learning rule that changes the probability of transmission at synapses where the presynaptic and postsynaptic units are simultaneously active from a low, base value to a high, modified value. Numerical calculations of the expected recall response of this stochastic associative net have been used to assess the performance for different values of the base and modified probabilities. If there is a cost incurred with generating the difference between these probabilities, then a difference of about 0.4 is optimal. This corresponds to the magnitude of change seen experimentally. Performance can be greatly enhanced by using multiple cue presentations during recall.
1 Introduction Comparison between the mammalian hippocampus and neural network models of associative memory has a long history (Bennett, Gibson, & Robinson, 1994; Marr, 1971; McNaughton & Morris, 1987; Treves & Rolls, 1994). The models are necessarily simpler than the hippocampus in both architecture and operation, but are arguably still relevant to the neurobiological system (McNaughton & Morris, 1987; Graham & Willshaw, 1995). Patterns are stored in these memory models by local Hebbian learning (Hebb, 1949). Such learning rules in principal may correspond to the phenomena of longterm potentiation (LTP) and long-term depression (LTD) in the nervous system (for recent reviews of LTP and LTD, see Bliss & Collingridge, 1993; Larkman & Jack, 1995; Malenka, 1994). Recall of information from the memories involves threshold setting on the activities of output units (or neurons). The threshold-setting mechanisms used in the models (e.g., k-winners-takeall) could conceivably be implemented by local circuits involving inhibitory interneurons in the hippocampus (Bennett et al., 1994; Graham & Willshaw, 1995; Marr, 1971). Neural Computation 11, 117–137 (1999)
c 1999 Massachusetts Institute of Technology °
118
Bruce Graham and David Willshaw
The nervous system contains many forms of noise that may disrupt the operation of associative memory networks. For example, the principal neurons in the hippocampus are only sparsely interconnected with each other (Amaral, Ishizuka, & Claiborne, 1990). The effect of this partial connectivity, which is a potential source of noise, has been studied extensively in neural network models (Canning & Gardner, 1988; Gardner-Medwin, 1976; Marr, 1971; Sompolinsky, 1987). One feature of biological nervous systems that has not been included in these models is the probabilistic nature of transmission of a signal from a pre- to a postsynaptic neuron (for an overview of synaptic transmission, see Redman, 1990). In this article, we examine the effects of probabilistic transmission on the operation of the associative net model of heteroassociative memory (Willshaw, Buneman, & Longuet-Higgins, 1969; Willshaw, 1971). Recent advances in experimental techniques have begun to quantify the transmission probabilities at individual hippocampal synapses. Fewer than half of the action potentials arriving at a presynaptic terminal may elicit a postsynaptic response (Allen & Stevens, 1994; Hessler, Shirke, & Malinow, 1993; Rosenmund, Clements, & Westbrook, 1993). Long-term potentiation of mammalian central synapses was first detected in the hippocampus (Bliss & Lomo, 1973; Bliss & Gardner-Medwin, 1973), and a change in the probability of transmission is at least a component of LTP (Bekkers & Stevens, 1990; Malinow & Tsien, 1990). Recent experiments have demonstrated LTP in which it may be the only component (Bolshakov & Siegelbaum, 1995; Siegelbaum & Bolshakov, 1996; Stevens & Wang, 1994) (though for competing explanations, see Collingridge, 1994; Malinow & Mainen, 1996). Some experiments have identified two populations of synapses: one with a very low probability of transmission (< 0.1) and the other with a higher probability (Hessler et al., 1993; Rosenmund et al., 1993). This could be due to LTP being implemented as a step increase in probability (Hessler et al., 1993). Induction of LTD may result in a step decrease in transmission probability and can be the reversal of LTP (Stevens & Wang, 1994). The associative net model can be extended naturally to include probabilistic transmission between the computing units. The model consists of a set of input units that send feedforward connections to a set of output units. Unit activity is binary (zero for inactive and one for active). In the standard net, the connection weights are also binary. Initially, all weights are zero. Pairs of binary patterns are stored in the net by altering a connection weight from zero to one if the input and output units are active for the same pattern pair. This is a clipped Hebbian learning rule that includes synaptic potentiation but not depression. To create a stochastic associative net, we treat the connection weights as probabilities of synaptic transmission. In the standard deterministic net, an unmodified synapse has a connection weight of zero, allowing no transmission of a signal from the input unit to the output unit. After learning, a modified synapse has a weight of one, so such a synapse will always transmit a signal from the input to the output
Probabilistic Synaptic Transmission in the Associative Net
119
unit. The aim of this work is to investigate the effect on associative memory performance when the transmission probabilities are not zero and one. For example, the unmodified probability might be 0.2, leading to a small chance of the synapse transmitting a signal, even though it has not been modified by Hebbian learning. A modified synapse will have a higher probability of transmitting a signal, say 0.8, but conversely will retain a small probability of not doing so. The notion of a weight in this model is different from that in most artificial neural networks in which transmitted signals are multiplied by the connecting weight to give a scaled signal. The experimental data outlined above yield a variety of probabilities of transmission. We consider the effect on memory performance of both the actual values of the modified and unmodified probabilities and the difference between them. A change in the probability of transmission at a biological synapse requires the use of energy. It is obviously desirable to make the minimum change necessary for learning and thus use as little energy as possible. If it is assumed that the cost in energy usage is directly proportional to the magnitude of change made to the transmission probability, then our results indicate that a rather small difference in probability (0.4 or less) is optimal. The best values of the modified and unmodified probabilities also depend on net connectivity and the threshold-setting strategy used during pattern recall. Some of this work has been presented in abstract form (Graham & Willshaw, 1997a). 2 The Stochastic Associative Net 2.1 Net Configuration and Operation. The stochastic associative net consists of NB output units, each of which receives connections from a random fraction, 0 < Z ≤ 1, of NA input units. Unit activity is binary, so that a unit is either inactive (0) or active (1). Input (output) patterns consist of MA ¿ NA (MB ¿ NB ) active units. During pattern recall, the transmission of an active signal from an input unit to an output unit occurs with a probability whose value is the synaptic weight. In this net, the weights are two-valued. Initially all connections have a low, base probability of transmission, Pb . As pairs of binary patterns are stored in the net, synaptic weights are changed to a high, modified probability, Pm , if the input and output units are active for the same pattern pair. We are concerned with the effect on memory performance of the specific values of Pb and Pm , as well as the difference between them, Pd = Pm − Pb . Once a number of pattern pairs has been stored, one of the input patterns is presented on the input units as a cue, and a recall strategy is used to retrieve the associated output pattern. The basis of pattern recall is the dendritic sum, which is the weighted sum of the inputs to an output unit. In this stochastic net, the dendritic sum of an output unit is a random variable whose mean value is determined by how many active inputs an output unit is connected to with base or modified probabilities of transmission.
120
Bruce Graham and David Willshaw
The dendritic sum of an output unit may be different each time the same input cue is presented. The recall strategy must decide, on the basis of all of the dendritic sums, which output units to make active. In previous work on the deterministic associative net (Graham & Willshaw, 1995, 1997b), we used a simple winners-take-all (WTA) strategy that selects the MB output units with the highest dendritic sums to be active. We use this strategy here, which we call the basic WTA. Given that the dendritic sums are random variables, an output unit will gain a better estimate of its mean dendritic sum if the input cue is presented several times, allowing the output unit to obtain an average dendritic sum over all of the presentations. This may help recall performance, and such a multiple cueing strategy is tried in combination with the basic WTA. Multiple cues have been used previously to improve recall from the deterministic net when the input cues are noisy (Budinich, Graham, & Willshaw, 1995). When the net is only partially connected, each output unit is connected to only a fraction (Z < 1) of the active input units during recall. This distorts the dendritic sums and reduces recall performance. Greatly improved recall is obtained if the dendritic sums are normalized by the number of active inputs the output unit is connected to, before the WTA threshold is applied. We have called this the normalized WTA (Graham & Willshaw, 1995). In the hippocampus, the number of active inputs could possibly be measured by inhibitory interneurons (Marr, 1971) or by a separate synaptic mechanism, possibly involving NMDA channels (Graham & Willshaw, 1995). We compare the performance of the basic and normalized WTA strategies for a partially connected stochastic net. The two strategies are equivalent for a fully connected net. 2.2 Recall Performance. Two different measures are used to assess recall performance. First, the ability to recall a single output pattern is measured by the overlap of the recalled pattern with the target output pattern. Specifically, we use the correlation between the recalled and target patterns, which ranges from 0 for no overlap to 1 for perfect recall (Graham & Willshaw, 1995). Second, we consider net capacity, defined to be the number of pattern pairs that can be stored in the net so that every output pattern can be reliably recalled (at most one unit in error). 2.3 Numerical Calculations of Net Recall. The results presented in this article were generated by numerical calculations of expected recall performance. For the deterministic net, expressions for the probability distributions of the dendritic sums of output units that should be inactive (low units) or active (high units) have been obtained (Buckingham, 1991; Buckingham & Willshaw, 1993; Graham & Willshaw, 1997b). These expressions have been extended here to include probabilistic transmission and multiple cueing. The full details are given in appendix A. The WTA response to an input cue is calculated using these probability distributions by finding the
Probabilistic Synaptic Transmission in the Associative Net
121
threshold, T, which gives (NB − MB )P(Dl ≥ T) + MB P(Dh ≥ T) = MB ,
(2.1)
where P(Dl ≥ T) (P(Dh ≥ T)) is the probability that a low (high) output unit has a dendritic sum greater than the threshold. The number of false-positive and false-negative errors in the response is given by E = (NB − MB )P(Dl ≥ T) + MB (1 − P(Dh ≥ T)).
(2.2)
These numerical calculations give very good agreement with the recall response of computer simulations of networks. 3 Results Results have been obtained for a net with NA = 8000 input units and NB = 1024 output units. The stored patterns are sparse with, generally, MA = 240 and MB = 30 (approximately 3% activity). With deterministic transmission (Pb = 0 and Pm = 1), the net has a capacity of 3576 pattern pairs. This net is large enough to support sparse pattern activity, which is most information efficient (Willshaw et al., 1969), in concert with partial connectivity, resulting in a biologically reasonable configuration. 3.1 Fully Connected Net. When this stochastic associative net is fully connected (Z = 1), the only source of noise during recall, apart from that due to the other stored patterns, is the probabilistic transmission at the synapses. Examples of the recall overlap achieved for different numbers of stored pattern pairs when Pm = 0.9 and Pb is varied are shown in Figure 1a. Even for a large difference in probabilities (e.g., Pd = 0.8), recall performance is considerably impaired compared to the standard deterministic net. Regardless, many patterns can be stored and recalled without error for quite small differences in the probabilities of transmission (e.g., more than 500 for Pd = 0.4). Recall performance can be improved by presenting the input cue many times so that each output unit can calculate an average dendritic sum before the WTA threshold is applied. Figure 1b shows the effect of multiple cueing on pattern overlap. Performance increases rapidly with the number of cue presentations and is nearly maximal after 100 presentations. The effect on net capacity is shown by Figure 2. For one cue presentation, a probability difference of Pd = 0.8 provides a capacity of 1729, approximately 4.5 times the capacity of 387 when Pd = 0.2. When the input cue is presented 1000 times, the capacity is 3204 when Pd = 0.8, only slightly greater than the capacity of 2964 when Pd = 0.2. Not only is recall improved with multiple cueing, but the relative performance for small probability differences is also greatly increased. If there is a cost involved with each cue presentation,
122
Bruce Graham and David Willshaw
(a)
(b)
Overlap
1.0 0.8
Det 1 10 100 1000
0.6 0.4 0.2 0.0 500
1500
2500
3500
4500
Number of Patterns
500
1500
2500
3500
4500
Number of Patterns
Figure 1: Recall overlap as a function of the number of stored pattern pairs. (a) Different probability differences. The dashed line is the deterministic net. The solid lines have Pm = 0.9, with Pb from 0.1 to 0.8 in increments of 0.1, from right to left. (b) Multiple cueing. Input cue is presented 1, 10, 100, or 1000 times. Pb = 0.1 and Pm = 0.9.
we can ascertain whether there is an optimum number by dividing the capacity by the number of presentations to produce what we will call the Np capacity (see Figure 2b, solid lines). For probability differences greater than 0.1, a single presentation is optimal (maximal Np capacity); for Pd = 0.1, five presentations are optimal. In a biological system it could be that there is a start-up cost associated with cue presentation, so that extra presentations after the first may be relatively less expensive. If we assume that the cost increases only with the log of the number of presentations (see Figure 2b, dashed lines), then a single presentation is still optimal for Pd > 0.3, 10 presentations is optimal for Pd = 0.3, about 100 for Pd = 0.2, and 1000 for Pd = 0.1. Of major interest is the effect of different values of Pb and Pm on recall performance. In particular, the same difference in probabilities can be achieved with different values of Pb and Pm . Three schemes for setting the probability difference have been compared: High Pm : Pb varied, Pm fixed at 0.9. Low Pb : Pb fixed at 0.1, Pm varied. Both: both Pb and Pm varied, equidistant from 0.5. Comparing recall capacity when the probability difference, Pd , is set in these different ways shows that consistently better performance is achieved when Pm is high, regardless of Pb (the High Pm scheme; see Figure 3). The decrease in capacity with decreasing Pd is nonlinear. An optimal probability difference can be obtained by assuming there is a cost incurred that is proportional to the difference. We then optimize the relative capacity, which is the net capacity divided by the probability difference (we will call this the
Probabilistic Synaptic Transmission in the Associative Net
(b) 2000
Np Capacity
Capacity
(a) 3500 3000 2500 2000 1500 1000 500 0
123
1
10
100
1000 500 0
1000
linear log
1500
Number of Cue Presentations
1
10
100
1000
Number of Cue Presentations
Figure 2: Capacity and Np capacity as a function of the number of cue presentations for different probability differences. Pm = 0.9; Pb ranges from 0.1 to 0.8 going from top to bottom. (b) Solid lines are for linear cost, dashed lines for log cost.
(a)
(b)
1500 1000
High Pm Low Pb Both
Pd Capacity
Capacity
2000
500 0
3000 2500 2000 1500 1000 500 0
(d) 10000
Pd Capacity
Capacity
(c) 3000 2500 2000 1500 1000 500 0 0.0
0.2
0.4
0.6
Probability Difference
0.8
8000 6000 4000 2000 0 0.0
0.2
0.4
0.6
0.8
Probability Difference
Figure 3: Capacity and Pd capacity for either (a, b) 1 or (c, d) 10 cue presentations, as functions of the probability difference.
Pd capacity). The optimum depends on the actual probability values and the number of cue presentations. This is demonstrated by the examples given in Figure 3. For the High Pm scheme, the optimal difference is around 0.4 for one cue presentation (see Figure 3b), reducing to 0.1 for 10 cue presentations (see Figure 3d). The effects of the input coding rate and net size are illustrated by Figure 4. The sensitivity of the capacity to Pd increases with decreasing MA
124
Bruce Graham and David Willshaw
(b) 5000
120 240 480 960
Pd Capacity
Capacity
(a) 3000 2500 2000 1500 1000 500 0
4000 3000 2000 1000 0
3000 2500 2000 1500 1000 500 0 0.0
(d)
Pd Capacity
Capacity
(c)
0.2
0.4
0.6
Probability Difference
0.8
6000 5000 4000 3000 2000 1000 0 0.0
0.2
0.4
0.6
0.8
Probability Difference
Figure 4: Capacity and Pd capacity for one cue presentation to a fully connected net with input coding rates, MA , of 120, 240, 480, and 960 and (a, b) NA = 8000 or (c, d) NA set to 4000, 8000, 16,000, and 32,000, respectively. Pm = 0.9 and Pb varied.
due to the smaller sample size of active inputs available to each output unit. For high Pd and the input layer fixed at NA = 8000, the capacity decreases with increasing MA (see Figure 4a), as has been shown for the deterministic net (Graham & Willshaw, 1997b). However, for low Pd (< 0.3), large MA may actually provide greater capacity than a lower value of MA . This increased sensitivity is also reflected in the optimum probability difference. The optimum of around 0.2 for MA = 960 increases to 0.5 for MA = 120 (see Figure 4b). Nonetheless, this is only a 2.5 times change for an eightfold change in MA . The sensitivity of capacity to the sample size seen by an output unit is clearly illustrated when the input layer size, NA , is varied with MA to maintain a constant input coding rate of 3%. For a given Pd , the larger is NA (and hence MA ), the higher is the capacity (see Figure 4c). Once again, the optimum probability difference also decreases with increasing sample size, being around 0.2 for NA = 32000 (MA = 960) (see Figure 4d). 3.2 Partially Connected Net. Partial connectivity introduces another noise component during recall. The performance of the basic and normalized WTA recall strategies when each output unit receives connections from a random selection of 60% of the input units (Z = 0.6) is shown in Figure 5. Using the basic WTA, the performance of the deterministic net is degraded and is not much better than the stochastic net with a high probability dif-
Probabilistic Synaptic Transmission in the Associative Net
(a)
125
(b)
Overlap
1.0 0.8 0.6 0.4 0.2 0.0 500
1500
2500
3500
Number of Patterns
4500
500
1500
2500
3500
4500
Number of Patterns
Figure 5: Recall overlap as a function of the number of stored pattern pairs when the net has 60% connectivity. (a) Basic WTA. (b) Normalized WTA. The dashed lines are the deterministic net. The solid lines have Pm = 0.9, with Pb from 0.1 to 0.8 in increments of 0.1, from right to left in each graph. One cue presentation.
ference (see Figure 5a). The use of normalized WTA significantly improves recall performance. Figure 5b shows that the overlap for different numbers of stored patterns is now very similar to that obtained for the fully connected net (see Figure 1a). For the basic WTA, there is a significant change in the best values for Pb and Pm , compared to the fully connected net. Now the Low Pb scheme (Pb = 0.1) consistently provides better performance (see Figure 6a). The optimal probability difference is also now higher than for the fully connected net (0.5 for the Low Pb scheme; see Figure 6b). However, when using normalized WTA, the situation is similar to the fully connected net, with the High Pm scheme (Pm = 0.9) providing the best performance (see Figures 6c and 6d). The effect of connectivity on the best values for Pb and Pm is shown by Figure 7. For connectivities less than 80% (Z < 0.8) the Low Pb scheme provides the highest capacity when using basic WTA (see Figure 7a). For normalized WTA, the High Pm scheme always performs better (see Figure 7b). 4 Discussion We have introduced synaptic transmission probabilities to the associative net model of heteroassociative memory to create a stochastic net. In this net, the weight of a synaptic connection specifies the probability with which the binary activity of an input unit is transmitted to the output unit. This is different from most other neural net models in which input unit activity is multiplied by the connection weight to provide the final input signal to an output unit. However, probabilistic transmission accords with neurobiology, and recent experimental evidence suggests that one form of LTP in the mammalian hippocampus involves only a change in the probability of
126
Bruce Graham and David Willshaw
(a)
(b)
High Pm Low Pb Both
800 600 400
Pd Capacity
Capacity
1000
200 0
1500 1250 1000 750 500 250 0
(d) 2500
Pd Capacity
Capacity
(c) 1500 1250 1000 750 500 250 0 0.0
0.2
0.4
0.6
0.8
2000 1500 1000 500 0 0.0
Probability Difference
0.2
0.4
0.6
0.8
Probability Difference
Figure 6: Capacity and Pd capacity of the partially connected net as functions of probability difference. Z = 0.6; one cue presentation. (a, b) Basic WTA. (c, d) Normalized WTA.
Capacity
(a) 1200 1000 800 600 400 200 0 0.0
(b)
High Pm Low Pb Both
0.2
0.4
0.6
Connectivity (Z)
0.8
1.0 0.0
0.2
0.4
0.6
0.8
1.0
Connectivity (Z)
Figure 7: Capacity as a function of connectivity when the probability difference is 0.4. (a) Basic WTA. (b) Normalized WTA.
transmission (Bolshakov & Siegelbaum, 1995; Stevens & Wang, 1994). The associative net uses a clipped Hebbian learning rule that stores patterns via a step increase in the synaptic weight, or probability of transmission. We are concerned with the effect on associative memory performance of different values of the probability of transmission before (base: Pb ) and after (modified: Pm ) learning. Our results show that with probabilistic transmission, the associative
Probabilistic Synaptic Transmission in the Associative Net
127
net still functions as an associative memory but necessarily with impaired performance compared to the standard model with deterministic synapses. Even so, performance may approach that of the standard model if the net is only partially connected or input cues are presented many times (see Figures 1b and 5a). 4.1 Values of the Base and Modified Probabilities. In different circumstances, different values of the base and modified probabilities of transmission are optimal. When the net is fully connected, or normalized WTA threshold setting is used for recall, a high value of the modified probability, Pm , provides the best recall performance, regardless of the value of the base probability, Pb . The reason is the dominance of all the dendritic sums by modified synapses when the net is near capacity. During recall, a high output unit will be connected to all the active inputs by modified synapses, so its dendritic sum is purely determined by Pm . A low unit will be connected to the active inputs by both base and modified synapses, but near capacity over half the connections are likely to be modified. The recall performance of the net is determined by the signal-to-noise ratio between the dendritic sums of high and low units (Dayan & Willshaw, 1991). Essentially, the means of the two types of sum should be well separated, and their variances should be small. The difference between the means depends on only the probability difference and not on the absolute values of the base and modified probabilities (see appendix B). However, the variance of both sums is largely determined by Pm . These variances are maximal when Pm is 0.5 and decrease when Pm increases or decreases. For a given probability difference, Pd , the variance will be minimized by choosing Pm as high (close to 1) as possible. When using basic WTA threshold setting with a partially connected net, the results show that now the lowest possible value of Pm provides the best recall performance for a given Pd . The variance in the dendritic sums of both low and high units is increased by variation in the number of active inputs each output unit is connected to. Again, near capacity, both low and high unit dendritic sums are dominated by modified synapses, so performance is best when the variance contributed by these synapses is minimized. The variance is now determined by the product of the connectivity level (Z) and the modified transmission probability and is maximal when ZPm = 0.5 (see appendix B). So for connectivity around 0.5 or less, the variance is maximal when Pm = 1 and decreases as Pm decreases. Hence, the lowest possible value of Pm is best. 4.2 The Optimum Probability Difference. While memory capacity decreases as the probability difference, Pd , is reduced, the energy used to make the required changes during learning is also reduced. The trade-off between energy usage and performance is likely to be important in a neurobiological system. We have assumed that the cost in energy is directly proportional
128
Bruce Graham and David Willshaw
to the magnitude of change made to the transmission probability when a synapse is modified. In these circumstances, it is possible to identify an optimum probability difference, Pd = Pm − Pb . If Pb and Pm may be freely chosen, the optimum is quite small—around 0.4 for a single cue presentation to a fully connected net. Higher values, approaching the deterministic net (Pd = 1), are optimal if the value of either Pm or Pb is poorly chosen (e.g., Pb =0.1 with a fully connected net; see Figure 3b). Factors that decrease the sample size of active inputs seen by an output unit, such as small input coding rates or partial connectivity, also increase the optimum Pd (see Figures 4b, 4d, and 6b, respectively). The optimum difference is reduced to 0.1 or less if multiple cue presentations are used during recall. This introduces the extra cost of generating the multiple cue presentations. In a biological nervous system, this could equate to an input neuron generating multiple action potentials. If each action potential is generated independently, then the cost is proportional to the number of action potentials. However, if a burst of action potentials is produced by a slowly activating depolarizing current, it is likely that the cost of each succeeding action potential in the burst is proportionally less than the cost of the first. For a linear increase in cost with the number of presentations, the most cost-effective configuration for our fully connected system is to use a single cue presentation during recall with a probability difference of 0.4 for learning. While the optimum probability difference of 100 cue presentations is 0.04, this configuration has only about one-tenth of the relative (Np ) capacity of the optimal single cue configuration (data not shown). However, if we assume the cost of cue presentations increases only with the log of the number of cues, then the maximum relative capacity increases with the number of presentations, and for 100 cues with Pd = 0.04, it is about 3.5 times the single cue capacity with Pd = 0.4. 4.3 Relationship to Neurobiology. Given the abstract nature of our model, the results are at best indicative of the effect of probabilistic transmission on the operation of parts of the mammalian nervous system, such as the hippocampus. The presentation of an input cue during recall may be interpreted as a set of input neurons firing single action potentials that arrive synchronously at the output neurons. The model takes no account of noise due to asynchronous arrival of the spikes and the different times taken for the resulting postsynaptic signals to travel along the dendritic tree of an output neuron. Initial work on quantifying this spatiotemporal noise indicates that it may have a significant impact on net capacity (Graham & Willshaw, 1997c). Similarly, multiple cue presentations correspond to multiple presynaptic action potentials from the input neurons. Again all spikes are assumed to be synchronous, and no account is taken of frequencydependent effects such as paired-pulse facilitation or synaptic depression. Nonetheless, this interpretative framework leads to the following conclusions.
Probabilistic Synaptic Transmission in the Associative Net
129
The above treatment of the costs involved in altering the probability of transmission and the multiple presentation of input cues is entirely arbitrary. However, for a biological system, it points to a trade-off between the cost of generating an action potential and the cost of transmitter release at a chemical synapse. It may be more efficient for a neuron to produce a burst of action potentials and have highly unreliable synapses than to produce single action potentials that reach reliable synapses. Either mode of operation should ensure transmission of a signal to the postsynaptic neuron. Lisman (1997) provides an interesting discussion of the possible roles of burst firing. Current experimental measurements of the probability of transmission at mammalian hippocampal synapses vary over the full range. After the induction of LTP at synapses between CA3 and CA1 pyramidal cells in hippocampal slices from 2- to 3-week old rats, Stevens & Wang (1994) saw an increase in transmission probability from 0.4 to 0.8, while Bolshakov and Siegelbaum (1995) measured an increase from 0.58 to 0.92. Without inducing LTP, Hessler et al. (1993) and Rosenmund et al. (1993) detected two populations of synapses: one with a very low probability of transmission (0.06 and 0.09, respectively) and the other with a moderate probability (0.37 and 0.54, respectively). All of these measurements yield a probability difference of around 0.3 to 0.4. Such a difference can be optimal for our associative memory model. The net used here is at least an order of magnitude smaller than sections of the mammalian hippocampus. A major determinant of memory performance is the sample size of active inputs seen by an output unit (see Figure 4). In the CA3 region of the hippocampus of Sprague-Dawley rats, 330,000 principal neurons interconnect with approximately 2% of their neighbors (Amaral et al., 1990; Boss, Turlejski, Stanfield, & Cowan, 1987), possibly forming an autoassociative memory network. Activity levels are assumed to be low. If 3% of the neurons are active, then each neuron receives input from a population of around 200 cells. This is of the same order of magnitude as the sample sizes (MA ) considered here. Thus, the effects of probabilistic transmission will still be present in nets of the size of the mammalian hippocampus.
4.4 Other Models of Synaptic Transmission and Modification. The size of the postsynaptic response to a presynaptic action potential may be affected by presynaptic factors other than just the probability of transmitter release. A presynaptic neuron may make more than one synaptic contact with a postsynaptic cell, and each synapse may contain more than one site of transmitter release. The number and size of quanta of transmitter released from each site may also vary. An extensive review of quantal transmission is given in Redman (1990). Our model corresponds to a single synaptic contact between each input and output neuron, with each synapse containing only a single site of
130
Bruce Graham and David Willshaw
transmitter release. An action potential may cause only a single quantum of transmitter to be released, and there is no variance in the size of quanta. In the mammalian hippocampus, most excitatory afferents make only a single connection with a postsynaptic pyramidal cell (Sorra & Harris, 1993). However, a significant proportion of afferents (20%) may make up to four synaptic contacts, and synapses may contain more than one release site (Sorra & Harris, 1993). Also, recent experimental data show considerable variability in the postsynaptic response to single inputs, due to either the release of multiple quanta of transmitter or variation in the quantal size (Stricker, Field, & Redman, 1996b). Changes in all these parameters may take place during the induction of LTP (Isaac, Hjelmstad, Nicoll, & Malenka, 1996; Stricker, Field, & Redman, 1996a), leading to an increase in synaptic potency (the average size of the postsynaptic response when transmitter is released). Changes that result in an increase in synaptic potency match more closely the increases in multiplicative weights during unconstrained Hebbian learning in associative memory models such as the Hopfield net (Hopfield, 1982). Bennett et al. (1994) have included variability in the amplitude of the postsynaptic response in an autoassociative neural net model of the CA3 region of the hippocampus. Like the associative net model, unit activity was binary, and the connection weights were also binary. Variations in the postsynaptic effect of an active input were modeled by the addition of gaussian noise to the mean amplitude of 1. This accounts for variations in the number of quanta of transmitter released and in the size of individual quanta. The effect of this noise was to decrease memory capacity, but also to improve recall from noisy input cues, due to the dynamics of multistep recall in this autoassociative model. The noise kept more units active during the early stages of recall. Without the noise, all unit activity died away quickly if the initial cue was too different from the stored pattern. Such noise could be incorporated into our model, where it should cause a decrease in capacity in proportion to the level of noise for the single-step recall of our heteroassociative memory. Other neural network models of associative memory have incorporated random processes in the activity of the units or in the effect of their synapses. Most widely studied are thermodynamic network models of autoassociative memory (for overviews, see Amit, 1989; Peretto, 1992). In these models the output activity of a unit is a probabilistic function of the weighted sum of the inputs. The probabilistic function is most often in the form of a sigmoid, the slope of which is inversely related to the so-called temperature. This is essentially equivalent to the noise due to all synapses’ having the same probability of transmission (Peretto, 1992). Such noise may help prevent the network’s becoming stuck in local minima during multistep recall. Burnod and Korn (1989) have demonstrated that random activity in a population of inhibitory neurons can shape such a sigmoid stochastic activity function in the postsynaptic cell to which they connect. Their model assumed that
Probabilistic Synaptic Transmission in the Associative Net
131
each inhibitory neuron formed synapses with multiple release sites with a certain probability of transmission at each site. 5 Summary An associative net with probabilistic transmission at the synapses still functions as an associative memory. Only very small changes in the probability of transmission need to be made during learning to enable many patterns to be stored and accurately recalled. Depending on the relative costs, there is a trade-off between the size of the probability difference used in learning and the number of cue presentations used during pattern recall to achieve the most efficient stochastic associative net. Appendix A: Probability Distributions of Dendritic Sums The WTA recall response can be calculated numerically using expressions for the distributions of the dendritic sums of low- and high-output units. This appendix gives details of the probability distributions of the basic and normalized sums. It is assumed that all patterns are sparse and the activity of individual units is approximately independent (1 ¿ MA ¿ NA and 1 ¿ MB ¿ NB ). Also, the number of stored pattern pairs must be much smaller than the possible number of independent pattern pairs. The distributions for the deterministic net have been determined previously (Buckingham, 1991; Buckingham & Willshaw, 1993; Graham & Willshaw, 1997b). These expressions are extended here to include the probabilistic transmission through synapses and summing due to multiple cue presentations. A.1 Basic WTA. The probability distribution of a dendritic sum is the sum of several different probability distributions that account for (1) how many times the output unit was active during pattern storage (unit usage, r), (2) the number of active input units it is connected to, and (3) the number of modified synapses it has. For a low output unit (one that should be inactive), the probability that an arbitrary synapse was modified during pattern storage is ρ[r] = 1 − (1 − αA )r ,
(A.1)
where r is the number of times the unit was active during storage and αA = MA /NA is the level of input pattern activity. For a high unit (one that should be active) a good approximation for this probability is µ[r + 1] ' g + sρ[r] = 1 − s(1 − αA )r ,
(A.2)
where g and s are the probabilities that a particular active input in the cue pattern is genuine (belongs to the stored pattern) or spurious, respectively
132
Bruce Graham and David Willshaw
(g + s = 1) (Buckingham & Willshaw, 1993). In this article, we have considered only noise-free input cues, so s = 0 and µ[r + 1] = 1. We will consider first the probability of the dendritic sum of a low unit. Suppose the unit is connected to mc = mb + mm active inputs by mb base synapses and mm modified synapses. The dendritic sum consists of independent components from the base and modified synapses, d∗ = db + dm .
(A.3)
The components are binomially distributed with db = Bin(mb , Pb ) and dm = Bin(mm , Pm ). Thus the expectation and variance of the dendritic sum are E[d∗ ] = E[db ] + E[dm ] = mb Pb + mm Pm
(A.4)
V[d∗ ] = V[db ] + V[dm ] = mb Pb (1 − Pb ) + mm Pm (1 − Pm ).
(A.5)
If the input cue is presented Np times and the output unit sums these dendritic sums, the total dendritic sum, D∗ , has mean and variance given by E[D∗ ] = Np (mb Pb + mm Pm )
(A.6)
V[D∗ ] = Np (mb Pb (1 − Pb ) + mm Pm (1 − Pm )),
(A.7)
as each of the Np individual sums is independent of the others. For Pb and Pm not very close to zero or one, the distribution of D∗ is approximately gaussian, D∗ = Gauss(E[D∗ ], V[D∗ ]). This is the dendritic sum distribution only if mc , mb , and mm are known. The overall distribution can be gained only by summing over all possible values of mc and mm (and hence mb ). For a given mc , the number of modified connections is binomially distributed and for a low unit is given by mm = Bin(mc , ρ[r]). The number of connections, mc , is approximately binomial, with mc = Bin(MA , Z), where Z is the level of connectivity. Finally, the unit usage is also binomially distributed, r = Bin(R, αB ), where R is the total number of pattern pairs stored, and αB = MB /NB is the level of output pattern activity. Thus, the probability that the basic dendritic sum of a lowoutput unit should have a particular value x is P(Dl = x) =
R X
Bin(R, αB )
r=0
×
mc X mm =0
MA X
Bin(MA , Z)
mc =0
Bin(mc , ρ[r])Gauss(E[D∗ ], V[D∗ ], x).
(A.8)
Probabilistic Synaptic Transmission in the Associative Net
133
The probability distribution for a high unit, Dh , is obtained by replacing ρ[r] with µ[r + 1] in equation A.8. For noise-free input cues, µ[r + 1] = 1, mm = mc and the above expressions reduce to E[D∗ ] = Np mc Pm
(A.9)
V[D∗ ] = Np mc Pm (1 − Pm )
(A.10)
P(Dl = x) = MA R X X Bin(R, αB ) Bin(MA , Z)Gauss(E[D∗ ], V[D∗ ], x).
(A.11)
r=0
mc =0
A.2 Normalized WTA. A normalized dendritic sum is D0 = D/a, where the input activity, a, is the number of active input units an output unit is connected to. The distributions of normalized sums can be approximated by the basic distributions for the situation where every unit has the mean input activity, am = MA Z. In this case mc has only the single value, MA Z, and the low-unit normalized distribution is given by P(Dl = x) = M R AZ X X Bin(R, αB ) Bin(MA Z, ρ[r])Gauss(E[D∗ ], V[D∗ ], x). r=0
(A.12)
mm =0
The high-unit distribution is again given by replacing ρ[r] with µ[r + 1] in equation A.12. For the noise-free case, this simplifies to P(Dl = x) =
R X
Bin(R, αB )MA ZGauss(E[D∗ ], V[D∗ ], x).
(A.13)
r=0
Appendix B: Signal-to-Noise Ratio of Dendritic Sums The expressions for the mean and variance of high and low output unit dendritic sums given in appendix A can be further simplified to make transparent the roles of the base and modified probabilities of transmission in memory recall performance. We will consider the dendritic sums of a single output unit due to patterns to which it should respond high and low, respectively. This allows us to ignore differences in unit usage between units. A further simplification is to assume that the probability that the output unit receives a signal from an active input is the probability that they are connected, Z, multiplied by the probability of transmission, Pb or Pm .
134
Bruce Graham and David Willshaw
For high patterns, the unit is connected to active inputs only by modified connections. Under the above assumptions, the dendritic sums due to these patterns are approximately binomially distributed, with dh = Bin(MA , ZPm ). Thus, the mean and variance of the sums are E[dh ] = MA ZPm .
(B.1)
V[dh ] = MA ZPm (1 − ZPm ).
(B.2)
Assume that the unit is connected to each low pattern by mb base synapses and mm modified synapses (mb + mm = MA ). Then the distribution of dendritic sums due to these patterns is the sum of two binomial distributions, db = Bin(mb , ZPb ) and dm = Bin(mm , ZPm ). The mean and variance are E[dl ] = mb ZPb + mm ZPm
(B.3)
V[dl ] = mb ZPb (1 − ZPb ) + mm ZPm (1 − ZPm ).
(B.4)
The memory performance of such a unit is maximized when the signal-tonoise ratio between these two types of dendritic sums is maximized (Dayan & Willshaw, 1991). This will occur when the difference between the means is maximized and the sum of the variances is minimized. The difference between the means is E[dh ] − E[dl ] = MA ZPm − mb ZPb − mm ZPm = mb Z(Pm − Pb ) = mb ZPd ,
(B.5)
and the sum of the variances is V[dh ] + V[dl ] = MA ZPm (1 − ZPm ) + mb ZPb (1 − ZPb ) + mm ZPm (1 − ZPm ) = (MA + mm )ZPm (1 − ZPm ) + mb ZPb (1 − ZPb ).
(B.6)
For a given probability difference, Pd , the difference in means is independent of the values of Pm and Pb . The sum of the variances is dominated by Pm and will be minimized when the contribution due to Pm is minimized. This will occur when ZPm is close to zero or one. So the preferred value of Pm depends on the connectivity level, Z. Acknowledgments The Medical Research Council provided financial support for our work under Program grant PG 9119632. Many thanks to Robin Lester for initial discussions on probabilistic synaptic transmission.
Probabilistic Synaptic Transmission in the Associative Net
135
References Allen, C., & Stevens, C. (1994). An evaluation of causes for unreliability of synaptic transmission. Proc. Nat. Acad. Sci., 91, 10380–10383. Amaral, D., Ishizuka, N., & Claiborne, B. (1990). Neurons, numbers and the hippocampal network. In J. Storm-Mathisen, J. Zimmer, & O. Ottersen (Eds.), Progress in brain research (pp. 1–11). Amsterdam: Elsevier Science. Amit, D. J. (1989). Modeling brain function: The world of attractor neural networks. Cambridge University Press. Bekkers, J., & Stevens, C. (1990). Presynaptic mechanism for long-term potentiation in the hippocampus. Nature, 346, 724–729. Bennett, M., Gibson, W., & Robinson, J. (1994). Dynamics of the CA3 pyramidal neuron autoassociative memory network in the hippocampus. Phil. Trans. Roy. Soc. Lond. B, 343, 167–187. Bliss, T., & Collingridge, G. (1993). A synaptic model of memory: Long-term potentiation in the hippocampus. Nature, 361, 31–39. Bliss, T., & Gardner-Medwin, A. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the unanaesthetized rabbit following stimulation of the perforant path. J. Physiol., 232, 357–374. Bliss, T., & Lomo, T. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. J. Physiol., 232, 331–356. Bolshakov, V., & Siegelbaum, S. (1995). Regulation of hippocampal transmitter release during development and long-term potentiation. Science, 269, 1730– 1734. Boss, B., Turlejski, K., Stanfield, B., & Cowan, W. (1987). On the numbers of neurons in fields CA1 and CA3 of the hippocampus of Sprague-Dawley and Wistar rats. Brain Res., 406, 280–287. Buckingham, J. (1991). Delicate nets, faint recollections: A study of partially connected associative network memories. Unpublished doctoral dissertation, University of Edinburgh. Buckingham, J., & Willshaw, D. (1993). On setting unit thresholds in an incompletely connected associative net. Network, 4, 441–459. Budinich, M., Graham, B., & Willshaw, D. (1995). Multiple cueing of an associative net. Int. J. Neural Systems, Supplementary Issue, 171. Burnod, Y., & Korn, H. (1989). Consequences of stochastic release of neurotransmitters for network computation in the central nervous system. Proc. Nat. Acad. Sci., 86, 352–356. Canning, A., & Gardner, E. (1988). Partially connected models of neural networks. J. Phys. A: Math. Gen., 21, 3275–3284. Collingridge, G. (1994). A question of reliability. Nature, 371, 652–653. Dayan, P., & Willshaw, D. (1991). Optimising synaptic learning rules in linear associative memories. Biol. Cybern., 65, 253–265. Gardner-Medwin, A. (1976). The recall of events through the learning of associations between their parts. Proc. Roy. Soc. Lond. B, 194, 375–402.
136
Bruce Graham and David Willshaw
Graham, B., & Willshaw, D. (1995). Improving recall from an associative memory. Biol. Cybern., 72, 337–346. Graham, B., & Willshaw, D. (1997a). An associative memory model with probabilistic synaptic transmission. In J. Bower (Ed.), Computational neuroscience: Trends in research, 1997 (pp. 315–319). New York: Plenum Press. Graham, B., & Willshaw, D. (1997b). Capacity and information efficiency of the associative net. Network, 8, 35–54. Graham, B., & Willshaw, D. (1997c). A model of clipped Hebbian learning in a neocortical pyramidal cell. In W. Gerstner, A. Germond, M. Hasler, & J.-D. Nicoud (Eds.), Artificial neural networks—ICANN ’97 (pp. 151–156). Berlin: Springer-Verlag. Hebb, D. (1949). The organization of behavior. New York: Wiley. Hessler, N., Shirke, A., & Malinow, R. (1993). The probability of transmitter release at a mammalian central synapse. Nature, 366, 569–572. Hopfield, J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proc. Nat. Acad. Sci., 79, 2554–2558. Isaac, J., Hjelmstad, G., Nicoll, R., & Malenka, R. (1996). Long-term potentiation at single fiber inputs to hippocampal CA1 pyramidal cells. Proc. Nat. Acad. Sci., 93, 8710–8715. Larkman, A., & Jack, J. (1995). Synaptic plasticity—Hippocampal LTP. Curr. Opin. Neurobiol., 5, 324–334. Lisman, J. (1997). Bursts as a unit of neural information: Making unreliable synapses reliable. TINS, 20, 38–43. Malenka, R. (1994). Synaptic plasticity in the hippocampus: LTP and LTD. Cell, 78, 535–538. Malinow, R., & Mainen, Z. (1996). Long-term potentiation in the CA1 hippocampus. Science, 271, 1604–1605. Malinow, R., & Tsien, R. (1990). Presynaptic enhancement shown by whole-cell recordings of long-term potentiation in hippocampal slices. Nature, 346, 177– 180. Marr, D. (1971). Simple memory: A theory for archicortex. Phil. Trans. Roy. Soc. Lond. B, 262, 23–81. McNaughton, B., & Morris, R. (1987). Hippocampal synaptic enhancement and information storage within a distributed memory system. TINS, 10, 408–415. Peretto, P. (1992). An introduction to the modeling of neural networks. Cambridge: Cambridge University Press. Redman, S. (1990). Quantal analysis of synaptic potentials in neurons of the central nervous system. Physiol. Rev., 70, 165–198. Rosenmund, C., Clements, J., & Westbrook, G. (1993). Nonuniform probability of glutamate release at a hippocampal synapse. Science, 262, 754–757. Siegelbaum, S., & Bolshakov, V. (1996). Long-term potentiation in the CA1 hippocampus–response. Science, 271, 1605–1606. Sompolinsky, H. (1987). The theory of neural networks: The Hebb rules and beyond. In J. van Hemmen & I. Morgenstern (Eds.), Heidelberg colloquium on glassy dynamics (pp. 485–527). Berlin: Springer-Verlag. Sorra, K., & Harris, K. (1993). Occurrence and three-dimensional structure of
Probabilistic Synaptic Transmission in the Associative Net
137
multiple synapses between individual radiatum axons and their target pyramidal cells in hippocampal area CA1. J. Neurosci., 13, 3736–3748. Stevens, C., & Wang, Y. (1994). Changes in reliability of synaptic function as a mechanism for plasticity. Nature, 371, 704–707. Stricker, C., Field, A., & Redman, S. (1996a). Changes in quantal parameters of EPSCs in rat CA1 neurons in vitro after the induction of long-term potentiation. J. Physiol., 490, 443–454. Stricker, C., Field, A., & Redman, S. (1996b). Statistical analysis of amplitude fluctuations in EPSCs evoked in rat CA1 pyramidal neurones in vitro. J. Physiol., 490, 419–441. Treves, A., & Rolls, E. (1994). Computational analysis of the role of the hippocampus in memory. Hippocampus, 4, 374–391. Willshaw, D. (1971). Models of distributed associative memory. Unpublished doctoral dissertation, University of Edinburgh. Willshaw, D., Buneman, O., & Longuet-Higgins, H. (1969). Non-holographic associative memory. Nature, 222, 960–962. Received August 21, 1997; accepted May 28, 1998.
LETTER
Communicated by Geoffrey Goodhill
A Stochastic Self-Organizing Map for Proximity Data Thore Graepel Klaus Obermayer Department of Computer Science, Technical University of Berlin, Berlin, Germany
We derive an efficient algorithm for topographic mapping of proximity data (TMP), which can be seen as an extension of Kohonen’s selforganizing map to arbitrary distance measures. The TMP cost function is derived in a Baysian framework of folded Markov chains for the description of autoencoders. It incorporates the data by a dissimilarity matrix D and the topographic neighborhood by a matrix H of transition probabilities. From the principle of maximum entropy, a nonfactorizing Gibbs distribution is obtained, which is approximated in a mean-field fashion. This allows for maximum likelihood estimation using an expectationmaximization algorithm. In analogy to the transition from topographic vector quantization to the self-organizing map, we suggest an approximation to TMP that is computationally more efficient. In order to prevent convergence to local minima, an annealing scheme in the temperature parameter is introduced, for which the critical temperature of the first phase transition is calculated in terms of D and H. Numerical results demonstrate the working of the algorithm and confirm the analytical results. Finally, the algorithm is used to generate a connection map of areas of the cat’s cerebral cortex.
1 Introduction Exploratory data analysis and visualization have received a lot of attention, since electronic data processing has made available large amounts of data from different sources all over the world. With respect to unsupervised learning, researchers have focused on analysis methods for data, which are given as vectors in a space that is assumed to be Euclidean. Examples for this kind include principal component analysis (PCA) (Jolliffe 1986; Tipping & Bishop, 1997), independent component analysis (ICA) (Bell & Sejnowski, 1995), vector quantization (VQ) (MacQueen, 1967), latent variable models (Bishop, Svens´en, & Williams, 1997), or self-organizing maps (SOM) (Kohonen, 1982; Ritter, Martinetz, & Schulten, 1992). Often, however, data items are not given as points in a Euclidean data space, but one has to restrict oneself to the set of pairwise proximities as measured in particular in empirical sciences like biochemistry, economics, linguistics, or psychology. Here, two Neural Computation 11, 139–155 (1999)
c 1999 Massachusetts Institute of Technology °
140
Thore Graepel and Klaus Obermayer
strategies for data analysis have been pursued for some time: pairwise clustering, which detects cluster structure in dissimilarity data (Hofmann & Buhmann, 1994; Duda & Hart, 1973), and metric multidimensional scaling (MMDS), which deals with the embedding of pairwise proximity data in a Euclidean space for the purpose of visualization (Borg & Lingoes, 1987). Recently, both approaches were combined by Hofmann and Buhmann (1997), who restrict the mean-fields from the clustering cost function to squared Euclidean distances between data points and cluster centers in the embedding space. The coupling between clusters, however, is maintained only at finite temperatures. Here we present another approach to combine clustering and the visualization of proximity data, which is based on an extension of Kohonen’s (1982) SOM. Data items characterized by mutual dissimilarities are mapped in a many-to-one fashion (clustering) to a set of neurons with predefined neighborhood relations (visualization), according to their similarities. To this extent, we first derive a general cost function for probabilistic autoencoders in the spirit of Luttrell (1994), but with arbitrary distortion measures. This cost function is then minimized by deterministic annealing, which shares the robustness properties of maximum entropy inference. Then the same approximation that leads from Luttrell’s (1991) topographic vector quantization (TVQ) to Kohonen’s (1982) SOM for the Euclidean case is introduced in order to achieve higher computational efficiency. The article is structured as follows. In section 2 we derive the cost function for topographic mapping of proximity data (TMP) and discuss its properties. We then derive an optimization algorithm based on deterministic annealing using a mean-field approximation (see Graepel & Obermayer, 1998, for a kernel-based method leading to similar equations in a different context). In section 3 we analytically determine the critical temperature of the first/phase-transition during the annealing as a function of the dissimilarity matrix D of the data and the coupling matrix H, which determines the neighborhood relations of the set of neurons. In section 4, we apply the algorithm to a toy example and discuss the effects of the approximations made. As an example of how the algorithm works in practice, we finally use the TMP algorithm to group the areas in the cat’s cerebral cortex based on their corticocortical connectivity patterns. 2 Derivation of the Topographic Mapping for Proximity Data 2.1 Two-stage Folded Markov Chain. According to Luttrell (1994) the cost functional for a probabilistic autoencoder with two stages as shown in Figure 1 can be expressed as EFMC =
XXX i,j r,r0 s,s0
0
0
0
0
P0 (i)P1 (r|i)P2 (s|r)δ(s |s)P˜2 (r |s )P˜1 (j|r )d(i, j) , (2.1)
Stochastic Self-Organizing Map for Proximity Data
141
where i and j are data items, whose dissimilarity is given by d(i, j). P0 (i) is the probability of item i, and P1 (r|i) and P2 (s|r) are probabilistic encoders. Their 0 0 0 corresponding probabilistic decoders are P˜1 (j|r ) and P˜2 (r |s ), respectively. A probabilistic encoder is related to its corresponding decoder by Bayes’ theorem, 0
P1 (r |j)P0 (j) 0 P˜ 1 (j|r ) = 0 P1 (r ) 0
0
0
P2 (s |r )P1 (r ) 0 0 . P˜2 (r |s ) = 0 P2 (s )
(2.2) 0
Inserting equations 2.2 into equation 2.1, cancelling P1 (r ), performing the 0 sum over s , and assuming P2 (s|r) to be given yields EFMC (P1 ) =
X
0
P0 (i)P1 (r|i)P2 (s|r)P1 (r |j)P0 (j)
i,r,s,r0 ,j
0
P2 (s|r ) d(i, j) . P2 (s)
(2.3)
This equation does not depend on the decoders anymore, but instead we had to introduce P2 (s), X P2 (s|r)P1 (r|i)P0 (i) , (2.4) P2 (s) = i,r
as a consequence of Bayes’ theorem. Let us now choose the hitherto probabilistic encoder P1 (r|i) to be deterministic. In this case, we can express the encoder in terms of a stochastic matrix M = (mir )i=1,...,D, r=1,...,N ∈ RD×N , whose elements are binary asP def signment variables P1 (r|i) = mir , r mir = 1, ∀i, and may take only values from the set {0, 1}. In order to make contact with the SOM literature (Kodef
honen, 1982), P we denote the second encoder P2 (s|r) = hrs , subject to the constraints s hrs = 1, ∀r. In the literature on Kohonen’s SOM, hrs is called a neighborhood function in the space of neurons and determines the coupling between neurons r and s due to their spatial arrangement in a neural lattice. The data are given by a dissimilarity matrix D = (dij )i,j=1,...,D ∈ RD×D . With these notational conventions, we then arrive at the following cost function for the topographic mapping of D data items onto N neurons, ETMP (M) =
D N X mir hrs mjt hts 1X dij , PD PN 2 i,j=1 r,s,t=1 k=1 u=1 mku hus
(2.5)
via their dissimilarity values dij . The factor 1/2 has been introduced for computational convenience. Let us consider three special cases of the above cost function ETMP :
142
Thore Graepel and Klaus Obermayer
i
i
-
r P1 (rji)
j
P2 (sjr)
s
?
0
d(i; j )
-
(s js) 0
s j
0
P~1(j jr ) 0
r
?
P~2(r js ) 0
0
Figure 1: Illustration of a probabilistic autoencoder in the form of a two-stage folded Markov chain. A data item i is encoded by probabilistic encoders P1 (r|i) 0 0 0 and P2 (s|r) and recovered by corresponding decoders P˜ 2 (r |s ) and P˜ 1 (j|r ) leading to data item j. The resulting distortion is measured as d(i, j).
1. If the dissimilarity values dij are taken as the squared Euclidean distance |E xi − xEj |2 of data points xEi in a Euclidean space, ETMP is equivalent to the TVQ cost function introduced by Luttrell (1991). 2. If, however, the second encoder or neighborhood matrix is taken to be hrs = δrs , then we recover a cost function that is equivalent to Hofmann and Buhmann’s (1997) for pairwise clustering. The normalP izing denominator in equation 2.5 then becomes D k=1 mks and was introduced by Hofmann and Buhmann (1997) based on heuristic arguments about cluster coherency. In our derivation, it appears as a natural consequence of Bayes’ theorem applied to a probabilistic autoencoder. 3. P In the special case of a one-to-one mapping, that is, N = D and i mir = 1, ∀r, we recover a form equivalent to the C measure, which was introduced by Goodhill and Sejnowski (1997) as a unifying objective function for topographic one-to-one mappings. 2.2 EM Algorithm and Deterministic Annealing. In order to obtain a robust optimization scheme, we apply the principle of maximum entropy
Stochastic Self-Organizing Map for Proximity Data
143
(Jaynes, 1957) and obtain a Gibbs distribution, P(M) =
1 exp(−βETMP (M)) , ZP
(2.6)
where β is the inverse temperature and ZP the partition function, which can be interpreted as the likelihood of the dissimilarity data. The summation in the partition function is over all “legal” assignment matrices {M}. Since the cost function ETMP (M) depends on the assignment variables mir in a nonlinear fashion, this probability distribution does not factorize and, as a consequence, it is difficult to calculate averages with respect to it. Following Saul and Jordan (1996) and Hofmann and Buhmann (1997), we make a parameterized ansatz for a probability distribution Q(M, E ), Ã ! N D X X 1 exp −β mir eir , Q(M, E ) = ZQ i=1 r=1
(2.7)
which factorizes, and choose the partial assignment costs E = {eir } in such aPway as to minimize the Kullback-Leibler (KL) divergence KL(Q|P) = {M} Q ln(Q/P). Note that this approach implicitly assumes that assignments of data items to neurons are independent in the sense that hmir mjr i = hmir ihmjr i, an assumption that is most likely to be valid in the case D À N. We obtain N ∂hmkr i ∗ ! ∂hETMP (M)i X − e = 0 , ∀k, v , ∂ekv ∂ekv kr r=1
(2.8)
from which the mean-fields e∗kr can be calculated as detailed in the appendix. We note that the cost function (see equation 2.5) is invariant under the substitution dij = dji ← (dij + dji )/2. We also make the simplifying assumption of zero self-dissimilarity, dii = 0, and we neglect terms of order O(1/D). The optimal mean-fields e∗kr are then given by e∗kr =
N X
D X
PN
t=1 hmjt ihts PD PN s=1 j=1 l=1 u=1 hmlu ihus à ! PN D 1X u=1 hmiu ihus × dkj − dij , P PN 2 i=1 D l=1 u=1 hmlu ihus
hrs
(2.9)
where exp(−βe∗kr ) . hmkr i = PN ∗ s=1 exp(−βeks )
(2.10)
144
Thore Graepel and Klaus Obermayer
The self-consistent equations (2.9 and 2.10) can be solved by fixed-point iteration at any given value of the temperature parameter β. This constitutes an EM algorithm (Dempster, Laird, & Rubin, 1977), where the missing variables mkr are estimated in the E step, equation 2.10, and the recalculation of the mean-fields, equation 2.9, corresponds to the M step. Since we are interested in globally optimal solutions, we employ deterministic annealing in β. The annealing scheme starts from high temperature, β < β ∗ (see equation 3.7), where the unique maximum of the likelihood is found using the expectation-maximization (EM) algorithm. This maximum is then tracked through higher values of β. At sufficiently high β, the solution is expected to correspond to a good minimum of the original cost function. 2.3 SOM Approximation for TMP. According to Luttrell (1991), Kohonen’s (1982) SOM can be considered an approximation to what Luttrell calls TVQ (Luttrell, 1991; Graepel, Burger, & Obermayer, 1997). TVQ corresponds to the optimization of the cost functional (see equation 2.1) for the case that d(i, j) = |E xi − xEj |2 for data vectors xEi and the SOM approximation uses the E s | instead of the minix−w nearest-neighbor winning rule rwin = argmin Ps |E E t | for data points xE x−w mum distortion prescription rwin = argmins t hst |E E s . Let us now introduce an equivalent approximation and weight vectors w to TMP. The E-step, equation 2.10, can be seen as a softmax function with respect to the mean-fields e∗kr . Leaving out the convolution with hrs thus leads to a new prescription for the calculation of the mean fields, e∗kr
=
D X
PN
t=1 hmjt ihtr PD PN j=1 l=1 u=1 hmlu ihur à ! PN D 1X u=1 hmiu ihur × dkj − dij , P PN 2 i=1 D l=1 u=1 hmlu ihur
(2.11)
the SOM approximation. This approximation is computationally more efficient than the exact update given in equation 2.9. However, it has the drawback that the iteration scheme, equations 2.10 and 2.11, no longer performs an exact maximum likelihood estimate. However, the robustness of the SOM algorithm, being based on the same approximation (Burger, Graepel, & Obermayer, 1998), and our numerical results demonstrate the usefulness of the approximation. 3 Critical Temperature of the First-Phase Transition For soft clustering it is known that during the annealing process in β, the cluster representation undergoes a series of splittings (Rose, Gurewitz, & Fox, 1990, 1992; Buhmann & Kuhnel, ¨ 1993). In the topographic case, weight vectors in data space split along the principal axis of the data according to
Stochastic Self-Organizing Map for Proximity Data
145
the eigenstructure of the coupling matrix H (Graepel et al., 1997). Although there exists no Euclidean data space in our dissimilarity approach, we can examine the critical behavior of the neuron assignments with decreasing temperature. Let us consider the case of infinite temperature, β = 0. Using hmkr i0 = 1/N in equation 2.10, the mean-fields e0kr for β = 0 are given by e0kr
à ! D D 1 X 1 X = dij . dkj − D j=1 2D i=1
(3.1)
We now linearize the right-hand side of equation 2.9 around e0kr by performing a Taylor expansion in e∗mv − e0mv : e∗kr − e0kr =
¸ N · D X X ¡∗ ¢ ∂e∗kr emv − e0mv + · · · . ∗ ∂emv e0 m=1 v=1
(3.2)
kr
Evaluation of this expression yields e∗kr − e0kr = β
N D X X
¡ ¢ 1km 0rv e∗mv − e0mv
(3.3)
m=1 v=1
with 1km
D D D 1 1 X 1 X 1 X = dim + dkj − dkm − 2 dij D D i=1 D j=1 D i,j=1
and 0rv =
X s
¸ · 1 . hrs hvs − N
(3.4)
(3.5)
Equation 3.3 can be decoupled by transforming the shifted mean-fields e∗kr − e0kr into the eigenbases of 1 and 0. Denoting the transformed meanfields e˜κρ , we arrive at e˜κρ = β10 e˜κρ .
(3.6)
Assuming hrs = hsr , this equation has only nonvanishing solutions for βλ1 λ0 = 1, where λ1 and λ0 are eigenvalues of 1 and 0, respectively. This means that the fixed-point state from equation 3.1 first becomes unstable during the increase of β at β∗ =
1 , 0 λ1 max λmax
(3.7)
146
Thore Graepel and Klaus Obermayer
0 where λ1 max and λmax denote the largest eigenvalues of 1 and 0, respectively. The instability, which is also referred to as the automatic selection of feature dimensions (Kohonen, 1988), is characterized by the corresponding eigen0 0 vectors v1 max and vmax . While vmax determines the mode in neuron space that first becomes unstable (for details, see Graepel et al., 1997), v1 max can be identified as the principal coordinate from classical metric multidimensional scaling (Gower, 1966). It is instructive to consider a special case of 1 to understand its meaning. Assume that the dissimilarity matrix D repxi − xEj |2 /2 of D data vectors resents the squared Euclidean distances dij = |E xE with zero mean in an S-dimensional Euclidean space. Then it is easy to show from equation 3.4 that
1km =
1 xEk · xEm . D
(3.8)
In this case the D × D matrix 1 can at maximum have rank S. From singular value decomposition, it can be seen that the nonzero eigenvalues of 1 are the same as those of the covariance matrix C of the data. Since the eigenvectors of C correspond to the principal axes in data space, and its eigenvalues are the associated variances, the maximum variance in data space determines the critical temperature, whereby the instability occurs along the principal axis. In the general case of dissimilarities, we conclude that 1 can be interpreted as determining the width or pseudovariance of the ensemble of dissimilarity items. The results of this section can be extended to the TMP with SOM approximation. The matrix 0 as given in equation 3.5 is modified by omitting one convolution with hrs , resulting in SOM 0rv
¸ · 1 . = hvr − N
(3.9)
4 Numerical Simulations 4.1 Toy Example: Noisy Spiral. In this section, we examine the ability of TMP to generate a topographic representation of a one-dimensional noisy spiral (see Figure 2, left) in a three-dimensional Euclidean space using distance data only. One hundred data points xE were generated via x = sin(θ) + nx y = cos(θ) + ny θ z = + nz , π
(4.1)
where θ = [0, 4π] and nE is gaussian noise with zero mean and standard deviation σnE = 0.3. The dissimilarity matrix D was calculated from the
Stochastic Self-Organizing Map for Proximity Data
147
10
5
20
4
30
z
data points
3 2 1 0
40 50 60 70
−1 80
1 1
0
y
90
0 −1
100
−1
x
20
40
60
80
100
data points
Figure 2: Plots of a noisy spiral (left) and corresponding dissimilarity matrix (right). One hundred data points were generated according to equation 4.1 with θ = [0, 4π] and σnE = 0.3. The dissimilarity matrix was obtained as dij = |Exi −Exj |2 /2 and was plotted such that the rows from top down and the columns from left to right correspond to the data points in order of their generation along the spiral with increasing θ.
squared Euclidean distances between the data points, dij = |E xi − xEj |2 /2, and is depicted in Figure 2 (right). The neighborhood matrix H was chosen such that it reflects the topology of a chain of 10 neurons, with the coupling strength decreasing as a gaussian function of distance, hrs = exp(−|r − s|2 /2σh2 )/c .
(4.2)
σh = 0.5 and hrs is normalized to unit probability over all neurons by c. Note that this choice of σh corresponds to a very narrow neighborhood. Without annealing, it would lead to topological defects in the representation. We applied TMP both with and without the SOM approximation, choosing an exponential annealing schedule (Graepel et al., 1997) according to β t+1 = 1.1 β t with β 0 = 0.1. As can be seen from Figure 3, both variants of TMP converge to the same final value of the average cost function at low temperature. The first split occurs close to β ∗ = 0.71, the value predicted by equation 3.7, indicated as a vertical line in the plot. Due to the weak coupling, the SOM approximation induces only a slightly earlier transition at βS∗ = 0.70 in accordance with equation 3.9. TMP detects the reduced dimensionality of the spiral and correctly forms groups along the spiral. Figure 4 shows the assignment matrix of the data points in order of their generation in the spiral to the chain of neurons. At high temperature (see Figure 4, left) the assignments
148
Thore Graepel and Klaus Obermayer
140 exact TMP update with SOM approx.
120 100
<E>
80 60 40 20 0 −1 10
0
10
1
β
10
2
10
Figure 3: Plot of the average assignment cost hEi as a function of the temperature parameter β for TMP with and without SOM approximation applied to the dissimilarity matrix of the noisy spiral from Figure 2. The topology of the neurons was that of a chain given by equation 4.2 with σh = 0.5. β was varied according to β t+1 = 1.1 β t with β 0 = 0.1. The convergence criterion for the EM t −7 , ∀i, r. The average cost hEi was calcualgorithm was given by |et+1 ir − eir | < 10 lated using equation 2.5 with the binary assignment variables mir replaced by their averages hmir i. The vertical line indicates the value β ∗ = 0.71 as calculated from equation 3.7.
are fuzzy, but the emerging topography is visible immediately after the phase transition. The diagonal structure of the assignment matrix at low temperature (see Figure 4, right) indicates the topography of the map, while the small defects stem from the gaussian noise on the data. 4.2 Topographic Map of the Cat’s Cerebral Cortex. Let us now consider an example, that cannot in any sense be interpreted as representing a Euclidean space. The input data consist of a matrix of connection strengths between cortical areas of the cat. The data were collected by Scannell, Blake-
10
10
20
20
30
30
data points
data points
Stochastic Self-Organizing Map for Proximity Data
40 50 60
40 50 60
70
70
80
80
90
90
100
100
2
4
6
neurons
8
10
149
2
4
6
8
10
neurons
Figure 4: Plot of the average assignments hmir i at high temperature β = 0.81 (left) and low temperature β = 186.21 (right) for TMP without SOM approximation applied to the noisy spiral. Dark corresponds to high probability of assignment. Data and parameters as in Figure 3.
more, and Young (1995) from text and figures of the available anatomical literature, and the connections are assigned dissimilarity values d as follows: self-connection (d = 0), strong and dense connection (d = 1), intermediate connection (d = 2), weak connection (d = 3), and absent or unreported connection (d = 4). Scannell et al. (1995) analyze these data as ordinal data, but we make the stronger assumption that the dissimilarity values represent a ratio scale. Since the true values of the connection strength are not known, this is a very crude approximation. However, it serves well for demonstration purposes and shows the robustness of the described method. Although the original matrix d0ij was not completely symmetrical due to differences between afferent and efferent connections, the application of TMP is equivalent to the substitution dij = (d0ij + dji0 )/2. Since the original matrix was nearly symmetrical, this introducesP only a small mean square deviation per dissimilarity from the true matrix, i,j (dij − d0ij )2 /D2 ≈ 0.1). The topology was chosen as a two-dimensional map of 5×5 neurons coupled in accordance with equation 4.2 with two-dimensional index vectors and σh = 0.4. Figure 5 shows the dissimilarity matrix sorted according to the TMP assignment results. The dominant block diagonal structure reflects the fact that areas assigned to the same neuron are very similar. Additionally, it can be seen that areas assigned to neurons far apart in the lattice are less similar to each other than those assigned to neighboring neurons. Figure 6 displays the areas as assigned to neurons on the map by TMP. Four coherent regions on the map can be seen to represent four cortical
150
Thore Graepel and Klaus Obermayer
10
data points
20 30 40 50 60 10
20
30
40
50
60
data points Figure 5: Dissimilarity matrix of areas of the cat’s cerebral cortex. The areas are sorted according to their neuron assignments from top down and from left to right. The horizontal and vertical lines show groups of areas as assigned to neurons. Dark means similiar. The topology of the neurons was that of a 5 × 5 lattice as given by equation 4.2 with σh = 0.4. The annealing scheme was β t+1 = 1.05 β t with β 0 = 2.5 < 2.7265 = β ∗ . The convergence criterion for the t −10 EM algorithm was |et+1 , ∀i, r. ir − eir | < 10
systems: visual, auditory, somatosensory, and frontolimbic. The visual areas 20b and PS are an exception and occupy a neuron that is not part of the main visual region. Their position is justified, however, by the fact that these areas have many connections to the frontolimbic system. In general, it is observed that primary areas such as areas 17 and 18 for the visual system, areas 1, 2, 3a, 3b, and SII for the somatosensory system, and areas AI and AII for the auditory system are placed at corners or at an edge of the map. Higher areas with more crosstalk are found more centrally located on the map. An example is EPp, the posterior part of the posterior ectosylvian gyrus, a visual
Stochastic Self-Organizing Map for Proximity Data
35 ER Hipp
PSb Amyg
RS
36
DP
Tem
AI AII AAF P
151
IL Sb
Ia LA PL
PFCv PFCdm CGp
20b PS
Ig
PFCr PFCdl CGa
VP V
SSF EPp
6l 6m
SIV POA
PMLS VLS 20a
ALLS
7 5bl
5al 5m SSAo
4g 5am 5bm SSAi
17 18 AMLS 21a 21b
PLLS DLS AES
19 ALG SVA
4
3a 3b 1 2 SII
Figure 6: Connection map of the cat’s cerebral cortex. The map shows 65 cortical areas mapped to a lattice of 5 × 5 neurons. The four cortical systems— frontolimbic (——), visual (— —), auditory (· · ·), and somatosensory (- -)—have been mapped to coherent regions except for the visual areas 20b and PS, which occupy a neuron apart from the main visual region. Parameters as in Figure 5.
and auditory association area (Scannell et al., 1995) represented at the very center of the map with two direct visual neighbors. In summary, the map in Figure 6 is a plausible visualization of the connection patterns found in the cat’s cerebral cortex. It is clear, however, that the rather arbitrary and coarse topography of the 5 × 5 square map cannot fully express the rich cortical structures. Prior knowledge about the connection patterns, if available, could be encoded in the topology of the neurons to improve the representation.
152
Thore Graepel and Klaus Obermayer
5 Conclusion We proposed a robust algorithm for TMP, which extends the applicability of topographic mapping algorithms, such as Kohonen’s SOM, beyond the standard Euclidean data space used. The deterministic annealing scheme ensures fast convergence and thus leaves the neighborhood matrix H free to encode the desired spatial relations of the neurons onto which the data items are to be mapped. Besides the potential use of TMP as a more flexible alternative to multidimensional scaling, we envision its application to problems such as the generation of topographic maps of symbol strings, which is useful for large-scale data mining, such as in the World Wide Web (Kohonen, Kaski, Lagus, & Honkela, 1996). However, since the number of dissimilarities scales quadratically with the number of data items, it is computationally expensive to determine the pairwise dissimilarities and to process them. As a consequence, a mechanism for active data selection, for example, based on the expected information gain (MacKay, 1992), would be useful for determining those proximity values most important for the representation of the data. Combined with an estimation procedure for missing data based on the EM algorithm (Tresp, Ahmad, & Neuneier, 1994), this would lead to a system capable of performing data mining and visualization of proximity data on an even larger scale. Appendix: Derivation of Mean-Field Equations Using the relations mkr m P P kr = P P l u mlu hus l6=k u mlu hus + hrs
(A.1)
and, from 1/(a + b) = 1/a − b/(a(a + b)), 1 u mlu hus
P P l
1 P m h + u mku hus us lu u
= P
P
= P
1 P
l6=k
l6=k
−
X w
u mlu hus
hws P
l6=k
P
u mlu hus (
m P kw P l6=k
u mlu hus
+ hws )
.
We obtain for the derivative of the averaged cost function hETMP i, À X ∂ ¿ mir mjt 1X ∂hETMP i P P = hrs hts dij ∂ekv 2 r,s,t ∂ekv l u mlu hus i,j
(A.2)
Stochastic Self-Organizing Map for Proximity Data
153
* + X ∂hmkr i mjt 1X P P = hrs hts dkj 2 r,s,t ∂ekv l6=k u mlu hus + hrs j6=k * + X ∂hmkt i mir P P + dik ∂ekv l6=k u mlu hus + hts i6=k * + ∂hmkr i 1 P P + δrt dii ∂ekv l6=k u mlu hus + hrs * + XXX ∂hmkw i mjt mir − hws P P dij. (A.3) ∂ekv l6=k u mlu hus + hrs i6=k j6=k w With dii = 0 and dij = dji we obtain * + X ∂hmkr i X X mjt ∂hETMP i P P = hrs hts ∂ekv ∂ekv s,t l6=k u mlu hus + hrs r j6=k * + XX m iw hws P P dij × dkj − l6=k u mlu hus + hws i6=k w
(A.4)
and, comparing equations A.4 and 2.8 optimal mean-fields e∗kr , equation 2.9. Acknowledgments This project was funded by the Technical University of Berlin via the Forschungsinitiativprojekt FIP 13/41. References Bell, A. J., & Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7, 1129–1159. Bishop, C. M., Svens´en, M., & Williams, C. K. I. (1997). GTM: The generative topographic mapping. Neural Computation, 10, 215–234. Borg, I., & Lingoes, J. (1987). Multidimensional similarity structure analysis. Berlin: Springer-Verlag. Buhmann, J. M., & Kuhnel, ¨ H. (1993). Vector quantization with complexity costs. IEEE Transactions on Information Theory, 39, 1133–1145. Burger, M., Graepel, T., & Obermayer, K. (1998). An annealed self-organizing map for source channel coding. In Advances in neural information processing systems, 10 (pp. 430–436). Cambridge, MA: MIT Press. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, 39, 1–22.
154
Thore Graepel and Klaus Obermayer
Duda, R. O., & Hart, P. E. (1973). Pattern classification and scene analysis. New York: Wiley. Goodhill, G. J., & Sejnowski, T. J. (1997). A unifying objective function for topographic mappings. Neural Computation, 9, 1291–1303. Gower, J. C. (1966). Some distance properties of latent root and vector methods used in multivariate analysis. Biometrika, 9, 325–328. Graepel, T., Burger, M., & Obermayer, K. (1997). Phase transitions in stochastic self-organizing maps. Physical Review E, 56, 3876–3890. Graepel, T., & Obermayer, K. (1998). Fuzzy topographic kernel clustering. In W. Brauer (Ed.), Proceedings of the 5th GI Workshop Fuzzy Neuro Systems ’98 (pp. 90–97). Hofmann, T., & Buhmann, J. (1994). Central and pairwise data clustering by competitive neural networks. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 104–111). San Mateo, CA: Morgan Kaufmann. Hofmann, T., & Buhmann, J. (1997). Pairwise data clustering by deterministic annealing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19, 1–14. Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical Review, 106, 620–630. Jolliffe, I. (1986). Principal component analysis. Berlin: Springer-Verlag. Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological Cybernetics, 43, 59–69. Kohonen, T. (1988). Self-organization and associative memory (3rd ed.). Berlin: Springer-Verlag. Kohonen, T., Kaski, S., Lagus, K., & Honkela, T. (1996). Very large two-level som for the browsing of newsgroups. In C. v. d. Malsburg, J. C. Vorbruggen, ¨ W. v. Seelen, & B. Sendhoff (Eds.), Artificial neural networks–ICANN ’96 (pp. 833– 838). Berlin: Springer-Verlag. Luttrell, S. P. (1991). Code vector density in topographic mappings: Scalar case. IEEE Transactions on Neural Networks, 2, 427–436. Luttrell, S. P. (1994). A Bayesian analysis of self-organizing maps. Neural Computation, 6, 767–794. MacKay, D. J. C. (1992). Information-based objective functions for active data selection. Neural Computation, 4, 586–603. MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In L. M. LeCam & J. Neyman (Eds.), Proceedings of the Fifth Berkeley Symposium on Mathematical Statistic and Probability (pp. 281–297). Berkeley: University of California Press. Ritter, H. J., Martinetz, T., & Schulten, K. J. (1992). Neural computation and selforganizing maps: An introduction. Reading, MA: Addison-Wesley. Rose, K., Gurewitz, E., & Fox, G. C. (1990). Statistical mechanics and phase transitions in clustering. Physical Review Letters, 65, 945–948. Rose, K., Gurewitz, E., & Fox, G. C. (1992). Vector quantization by deterministic annealing. IEEE Transactions on Information Theory, 38, 1249–1257. Saul, L. K., & Jordan, M. I. (1996). Exploiting tractable substructures in intractable networks. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances
Stochastic Self-Organizing Map for Proximity Data
155
in neural information processing systems, 8 (pp. 486–492). Cambridge, MA: MIT Press. Scannell, J. W., Blakemore, C., & Young, M. P. (1995). Analysis of connectivity in the cat cerebral cortex. Journal of Neuroscience, 15, 1463–1483. Tipping, M. E., & Bishop, C. M. (1997). Mixtures of principal component analysers (Tech. Rep. NCRG/97/003). Birmingham: Aston University. Tresp, V., Ahmad, S., & Neuneier, R. (1994). Training neural networks with deficient data. In J. D. Cowan, G. Tessauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 128–135). San Mateo, CA: Morgan Kaufman. Received December 1, 1997; accepted March 27, 1998.
LETTER
Communicated by Anthony Bell
High-Order Contrasts for Independent Component Analysis Jean-Fran¸cois Cardoso Ecole Nationale Sup´erieure des T´el´ecommunications, 75634 Paris Cedex 13, France
This article considers high-order measures of independence for the independent component analysis problem and discusses the class of Jacobi algorithms for their optimization. Several implementations are discussed. We compare the proposed approaches with gradient-based techniques from the algorithmic point of view and also on a set of biomedical data. 1 Introduction Given an n × 1 random vector X, independent component analysis (ICA) consists of finding a basis of Rn on which the coefficients of X are as independent as possible (in some appropriate sense). The change of basis can be represented by an n × n matrix B and the new coefficients given by the entries of vector Y = BX. When the observation vector X is modeled as a linear superposition of source signals, matrix B is understood as a separating matrix, and vector Y = BX is a vector of source signals. Two key issues of ICA are the definition of a measure of independence and the design of algorithms to find the change of basis (or separating matrix) B optimizing this measure. Many recent contributions to the ICA problem in the neural network literature describe stochastic gradient algorithms involving as an essential device in their learning rule a nonlinear activation function. Other ideas for ICA, most of them found in the signal processing literature, exploit the algebraic structure of high-order moments of the observations. They are often regarded as being unreliable, inaccurate, slowly convergent, and utterly sensitive to outliers. As a matter of fact, it is fairly easy to devise an ICA method displaying all these flaws and working on only carefully generated synthetic data sets. This may be the reason that cumulant-based algebraic methods are largely ignored by the researchers of the neural network community involved in ICA. This article tries to correct this view by showing how high-order correlations can be efficiently exploited to reveal independent components. This article describes several ICA algorithms that may be called Jacobi algorithms because they seek to maximize measures of independence by a technique akin to the Jacobi method of diagonalization. These measures of independence are based on fourth-order correlations between the entries of Y. As a benefit, these algorithms evades the curse of gradient descent: Neural Computation 11, 157–192 (1999)
c 1999 Massachusetts Institute of Technology °
158
Jean-Fran¸cois Cardoso
they can move in macroscopic steps through the parameter space. They also have other benefits and drawbacks, which are discussed in the article and summarized in a final section. Before outlining the content of this article, we briefly review some gradient-based ICA methods and the notion of contrast function. 1.1 Gradient Techniques for ICA. Many online solutions for ICA that have been proposed recently have the merit of a simple implementation. Among these adaptive procedures, a specific class can be singled out: algorithms based on a multiplicative update of an estimate B(t) of B. These algorithms update a separating matrix B(t) on reception of a new sample x(t) according to the learning rule y(t) = B(t)x(t),
¢ ¡ B(t + 1) = I − µt H(y(t)) B(t),
(1.1)
where I denotes the n × n identity matrix, {µt } is a scalar sequence of positive learning steps, and H: Rn → Rn×n is a vector-to-matrix function. The stationary points of such algorithms are characterized by the condition that the update has zero mean, that is, by the condition, EH(Y) = 0.
(1.2)
The online scheme, in equation 1.1, can be (and often is) implemented in an off-line manner. Using T samples X(1), . . . , X(T), one goes through the following iterations where the field H is averaged over all the data points: 1. Initialization. Set y(t) = x(t) for t = 1, . . . , T. P 2. Estimate the average field. H = T1 Tt=1 H(y(t)). 3. Update. If H is small enough, stop; else update each data point y(t) by y(t) ← (I − µH)y(t) and go to 2. The algorithm stops for a (arbitrarily) small value of the average field: it solves the estimating equation, T 1X H(y(t)) = 0, T t=1
(1.3)
which is the sample counterpart of the stationarity condition in equation 1.2. Both the online and off-line schemes are gradient algorithms: the mapping H(·) can be obtained as the gradient (the relative gradient [Cardoso & Laheld, 1996] or Amari’s natural gradient [1996]) of some contrast function, that is, a real-valued measure of how far the distribution Y is from some ideal distribution, typically a distribution of independent components. In
High-Order Contrasts for Independent Component Analysis
159
particular, the gradient of the infomax—maximum likelihood (ML) contrast yields a function H(·) in the form H(y) = ψ(y)y† − I,
(1.4)
where ψ(y) is an n × 1 vector of component-wise nonlinear functions with ψi (·) taken to be minus the log derivative of the density of the i component (see Amari, Cichocki, & Yang, 1996, for the online version and Pham & Garat, 1997, for a batch technique). 1.2 The Orthogonal Approach to ICA. In the search for independent components, one may decide, as in principal component analysis (PCA), to request exact decorrelation (second-order independence) of the components: matrix B should be such that Y = BX is “spatially white,” that is, its covariance matrix is the identity matrix. The algorithms described in this article take this design option, which we call the orthogonal approach. It must be stressed that components that are as independent as possible according to some measure of independence are not necessarily uncorrelated because exact independence cannot be achieved in most practical applications. Thus, if decorrelation is desired, it must be enforced explicitly; the algorithms described below optimize under the whiteness constraint approximations of the mutual information and of other contrast functions (possibly designed to take advantage of the whiteness constraint). One practical reason for considering the orthogonal approach is that offline contrast optimization may be simplified by a two-step procedure as follows. First, a whitening (or “sphering”) matrix W is computed and applied to the data. Since the new data are spatially white and one is also looking for a white vector Y, the latter can be obtained only by an orthonormal transformation V of the whitened data because only orthonormal transforms can preserve the whiteness. Thus, in such a scheme, the separating matrix B is found as a product B = VW. This approach leads to interesting implementations because the whitening matrix can be obtained straightforwardly as any matrix square root of the inverse covariance matrix of X and the optimization of a contrast function with respect to an orthonormal matrix can also be implemented efficiently by the Jacobi technique described in section 4. The orthonormal approach to ICA need not be implemented as a twostage Jacobi-based procedure; it also exists as a one-stage gradient algorithm (see also Cardoso & Laheld, 1996). Assume that the relative/natural gradient of some contrast function leads to a particular function H(·) for the update rule, equation 1.1, with stationary points given by equation 1.2. Then the stationary points for the optimization of the same contrast function with respect to orthonormal transformations are characterized by EH(Y) − H(Y)† = 0 where the superscript † denotes transposition. On the other hand, for zeromean variables, the whiteness constraint is EYY† = I, which we can also
160
Jean-Fran¸cois Cardoso
write as EYY† − I = 0. Because EYY† − I is a symmetric matrix matrix while EH(Y) − H(Y)† is a skew-symmetric matrix, the whiteness condition and the stationarity condition can be combined in a single one by just adding them. The resulting condition is E{YY† − I + H(Y) − H(Y)† } = 0. When it holds true, both the symmetric part and the skew-symmetric part cancel; the former expresses that Y is white, the latter that the contrast function is stationary with respect to all orthonormal transformations. Thus, if the algorithm in equation 1.1 optimizes a given contrast function with H given by equation 1.4, then the same algorithm optimizes the same contrast function under the whiteness constraint with H given by
H(y) = yy† − I + ψ(y)y† − yψ(y)† .
(1.5)
It is thus simple to implement orthogonal versions of gradient algorithms once a regular version is available. 1.3 Data-Based Versus Statistic-Based Techniques. Comon (1994) compares the data-based option and the statistic-based option for computing off-line an ICA of a batch x(1), . . . , x(T) of T samples; this article will also introduce a mixed strategy (see section 4.3). In the data-based option, successive linear transformations are applied to the data set until some criterion of independence is maximized. This is the iterative technique outlined above. Note that it is not necessary to update explicitly a separating matrix B in this scheme (although one may decide to do so in a particular implementation); P the data themselves are updated until the average field T1 Tt=1 H(y(t)) is small enough; the transform B is implicitly contained in the set of transformed data. Another option is to summarize the data set into a smaller set of statistics computed once and for all from the data set; the algorithm then estimates a separating matrix as a function of these statistics without accessing the data. This option may be followed in cumulant-based algebraic techniques where the statistics are cumulants of X. 1.4 Outline of the Article. In section 2, the ICA problem is recast in the framework of (blind) identification, showing how entropic contrasts readily stem from the maximum likelihood (ML) principle. In section 3, high-order approximations to the entropic contrasts are given, and their algebraic structure is emphasized. Section 4 describes different flavors of Jacobi algorithms optimizing fourth-order contrast functions. A comparison between Jacobi techniques and a gradient-based algorithm is given in section 5 based on a real data set of electroencephalogram (EEG) recordings.
High-Order Contrasts for Independent Component Analysis
161
2 Contrast Functions and Maximum Likelihood Identification Implicitly or explicitly, ICA tries to fit a model for the distribution of X that is a model of independent components: X = AS, where A is an invertible n×n matrix and S is an n×1 vector with independent entries. Estimating the parameter A from samples of X yields a separating matrix B = A−1 . Even if the model X = AS is not expected to hold exactly for many real data sets, one can still use it to derive contrast functions. This section exhibits the contrast functions associated with the estimation of A by the ML principle (a more detailed exposition can be found in Cardoso, 1998). Blind separation based on ML was first considered by Gaeta and Lacoume (1990) (but the authors used cumulant approximations as those described in section 3), Pham and Garat (1997), and Amari et al. (1996). 2.1 Likelihood. Assume that the probability distribution of each entry PS of the random vector Si of S has a density ri (·).1 Then, the distribution Q S has a density r(·) in the form r(s) = ni=1 ri (si ), and the density of X for a given mixture A and a given probability density r(·) is: p(x; A, r) = | det A|−1 r(A−1 x),
(2.1)
so that the (normalized) log-likelihood LT (A, r) of T independent samples x(1), . . . , x(T) of X is def
LT (A, r) = =
T 1X log p(x(t); A, r) T t=1 T 1X log r(A−1 x(t)) − log | det A|. T t=1
(2.2)
Depending on the assumptions made about the densities r1 , . . . , rn , several contrast functions can be derived from this log-likelihood. 2.2 Likelihood Contrast. Under mild assumptions, the normalized loglikelihood LT (A, r), which is a sample average, converges for large T to its ensemble average by law of large numbers: LT (A, r) =
T 1X log r(A−1 x(t)) − log | det A| T t=1
−→T→∞ E log r(A−1 x) − log | det A|,
(2.3)
1 All densities considered in this article are with respect to the Lebesgue measure on R or Rn .
162
Jean-Fran¸cois Cardoso
which simple manipulations (Cardoso, 1997) show to be equal to −H(PX ) − K(PY |PS ). Here and in the following, H(·) and K(·|·), respectively, denote the differential entropy and the Kullback-Leibler divergence. Since H(PX ) does not depend on the model parameters, the limit for large T of −LT (A, r) is, up to a constant, equal to def
φ ML (Y) = K(PY |PS ).
(2.4)
Therefore, the principle of ML coincides with the minimization of a specific contrast function, which is nothing but the (Kullback) divergence K(PY |PS ) between the distribution PY of the output and a model distribution PS . The classic entropic contrasts follow from this observation, depending on two options: (1) trying or not to estimate PS from the data and (2) forcing or not the components to be uncorrelated. 2.3 Infomax. The technically simplest statistical assumption about PS is to select fixed densities r1 , . . . , rn for each component, possibly on the basis of prior knowledge. Then PS is a fixed distributional assumption, and the minimization of φ ML (Y) is performed only over PY via Y = BX. This can be rephrased: Choose B such that Y = BX is as close as possible in distribution to the hypothesized model distribution PS , the closeness in distribution being measured in the Kullback divergence. This is also the contrast function derived from the infomax principle by Bell and Sejnowski (1995). The connection between infomax and ML was noted in Cardoso (1997), MacKay (1996), and Pearlmutter and Parra (1996). 2.4 Mutual Information. The theoretically simplest statistical assumption about PS is to assume no model at all. In this case, the Kullback mismatch K(PY |PS ) should be minimized not only by optimizing over B to change the distribution of Y = BX but also with respect to PS . For each fixed B, that is, for each fixed distribution PY , the result of this minimization is theoretically very simple: the minimum is reached when PS = P¯ Y , which denotes the distribution of independent components with each marginal distribution equal to the corresponding marginal distribution of Y. This stems from the property that K(PY |PS ) = K(PY |P¯ Y ) + K(P¯ Y |PS )
(2.5)
for any distribution PS with independent components (Cover & Thomas, 1991). Therefore, the minimum in PS of K(PY |PS ) is reached by taking PS = P¯ Y since this choice ensures K(P¯ Y |PS ) = 0. The value of φ ML at this point then is def φ MI (Y) = min K(PY |PS ) = K(PY |P¯ Y ). PS
(2.6)
High-Order Contrasts for Independent Component Analysis
163
We use the index MI since this quantity is well known as the mutual information between the entries of Y. It was first proposed by Comon (1994), and it can be seen from the above as deriving from the ML principle when optimization is with respect to both the unknown system A and the distribution of S. This connection was also noted in Obradovic and Deco (1997), and the relation between infomax and mutual information is also discussed in Nadal and Parga (1994). 2.5 Minimum Marginal Entropy. An orthogonal contrast φ(Y) is, by definition, to be optimized under the constraint that Y is spatially white: orthogonal contrasts enforce decorrelation, that is, an exact “second-order” independence. Any regular contrast can be used under the whiteness constraint, but by taking the whiteness constraint into account, the contrast may be given a simpler expression. This is the case of some cumulant-based MI (Y) because the mucontrasts described in section 3. It is also the case of φP MI tual information can also be expressed as φ (Y) = ni=1 H(PYi ) − H(PY ); since the entropy H(PY ) is constant under orthonormal transforms, it is equivalent to consider φ ME (Y) =
n X i=1
H(PYi )
(2.7)
to be optimized under the whiteness constraint EYY† = I. This contrast could be called orthogonal mutual information, or the marginal entropy contrast. The minimum entropy idea holds more generally under any volumepreserving transform (Obradovic & Deco, 1997). 2.6 Empirical Contrast Functions. Among all the above contrasts, only φ ML or its orthogonal version are easily optimized by a gradient technique because the relative gradient of φ ML simply is the matrix EH(Y) with H(·) defined in equation 1.4. Therefore, the relative gradient algorithm, equation 1.1, can be employed using either this function H(·) or its symmetrized form, equation 1.5, if one chooses to enforce decorrelation. However, this contrast is based on a prior guess PS about the distribution of the components. If the guess is too far off, the algorithm will fail to discover independent components that might be present in the data. Unfortunately, evaluating the gradient of contrasts based on mutual information or minimum marginal entropy is more difficult because it does not reduce to the expectation of a simple function of Y; for instance, Pham (1996) minimizes explicitly the mutual information, but the algorithm involves a kernel estimation of the marginal distributions of Y. An intermediate approach is to consider a parametric estimation of these distributions as in Moulines, Cardoso, and Gassiat (1997) or Pearlmutter and Parra (1996), for instance. Therefore, all these contrasts require that the distributions of components be known, or approx-
164
Jean-Fran¸cois Cardoso
imated or estimated. As we shall see next, this is also what the cumulant approximations to contrast functions are implicitly doing. 3 Cumulants This section presents higher-order approximations to entropic contrasts, some known and some novel. To keep the exposition simple, it is restricted to symmetric distributions (for which odd-order cumulants are identically zero) and to cumulants of orders 2 and 4. Recall that for random variables def def X1 , . . . , X4 , second-order cumulants are Cum(X1 , X2 ) = EX¯ 1 X¯ 2 where X¯ i = Xi − EXi and the fourth-order cumulants are Cum(X1 , X2 , X3 , X4 ) = EX¯ 1 X¯ 2 X¯ 3 X¯ 4 − EX¯ 1 X¯ 2 EX¯ 3 X¯ 4 − EX¯ 1 X¯ 3 EX¯ 2 X¯ 4 − EX¯ 1 X¯ 4 EX¯ 2 X¯ 3 .
(3.1)
The variance and the kurtosis of a real random variable X are defined as σ 2 (X) = Cum(X, X) = EX¯ 2 , def
def k(X) = Cum(X, X, X, X) = EX¯ 4 − 3E2 X¯ 2 ,
(3.2)
that is, they are the second- and fourth-order autocumulants. A cumulant involving at least two different variables is called a cross-cumulant. 3.1 Cumulant-Based Approximations to Entropic Contrasts. Cumulants are useful in many ways. In this section, they show up because the probability density of a scalar random variable U close to the standard normal n(u) = (2π)−1/2 exp −u2 /2 can be approximated as ¶ k(U) σ 2 (U) − 1 h2 (u) + h4 (u) , p(u) ≈ n(u) 1 + 2 4! µ
(3.3)
where h2 (u) = u2 − 1 and h4 (u) = u4 − 6u2 + 3, respectively, are the secondand fourth-order Hermite polynomials. This expression is obtained by retaining the leading terms in an Edgeworth expansion (McCullagh, 1987). If U and V are two real random variables with distributions close to the standard normal, one can, at least formally, use expansion 3.3 to derive an approximation to K(PU |PV ). This is K(PU |PV ) ≈
1 2 1 (σ (U) − σ 2 (V))2 + (k(U) − k(V))2 , 4 48
(3.4)
which shows how the pair (σ 2 , k) of cumulants of order 2 and 4 play in some sense the role of a local coordinate system around n(u) with the quadratic
High-Order Contrasts for Independent Component Analysis
165
form 3.4 playing the role of a local metric. This result generalizes to multivariates, in which case we denote for conciseness RU ij = Cum(Ui , Uj ) and QU ijkl = Cum(Ui , Uj , Uk , Ul ) and similarly for another random n-vector V with entries V1 , . . . , Vn . We give without proof the following approximation: def
K(PU |PV ) ≈ K24 (PU |PV ) =
´2 1 X³ U Rij − RV ij 4 ij ´2 1 X³ U + . Qijkl − QV ijkl 48 ijkl
(3.5)
Expression 3.5 turns out to be the simplest possible multivariate generalization of equation 3.4 (the two terms in equation 3.5 are a double sum over all the n2 pairs of indices and a quadruples over all the n4 quadruples of indices). Since the entropic contrasts listed above have all been derived from the Kullback divergence, cumulant approximations to all these contrasts can be obtained by replacing the Kullback mismatch K(PU |PV ) by a cruder measure: its approximation is a cumulant mismatch by equation 3.5. 3.1.1 Approximation to the Likelihood Contrast. The infomax-ML contrast φ ML (Y) = K(PY |PS ) for ICA (see equation 2.4) is readily approximated by using expression 3.5. The assumption PS on the distribution of S is now replaced by an assumption about the cumulants of S. This amounts to very little: all the cross-cumulants of S being 0 thanks to the assumption of independent sources, it is needed only to specify the autocumulants σ 2 (Si ) and k(Si ). The cumulant approximation (see equation 3.5) to the infomax-ML contrast becomes: φ ML (Y) ≈ K24 (PY |PS ) =
´2 1 X³ Y Rij − σ 2 (Si )δij 4 ij ´2 1 X³ Y + Qijkl − k(Si )δijkl , 48 ijkl
(3.6)
where the Kronecker symbol δ equals 1 with identical indices and 0 otherwise. 3.1.2 Approximation to the Mutual Information Contrast. The mutual information contrast φ MI (Y) was obtained by minimizing K(PY |PS ) over all the distributions PS with independent components. In the cumulant approximation, this is trivially done: the free parameters for PS are σ 2 (Si ) and k(Si ). Each of these scalars enters in only one term of the sums in equation 3.6 so that the minimization is achieved for σ 2 (Si ) = RYii and k(Si ) = QYiiii . In other words, the construction of the best approximating distribution with
166
Jean-Fran¸cois Cardoso
independent marginals P¯ Y , which appears in equation 2.5, boils down, in the cumulant approximation, to the estimation of the variance and kurtosis of each entry of Y. Fitting both σ 2 (Si ) and k(Si ) to RYii and QYiii , respectively, has the effect of exactly cancelling the diagonal terms in equation 3.6, leaving only def
MI (Y) = φ MI (Y) ≈ φ24
1 X ³ Y ´2 1 X ³ Y ´2 , Rij + Q 4 ij6=ii 48 ijkl6=iiii ijkl
(3.7)
which is our cumulant approximation to the mutual information contrast in equation 2.6. The first term is understood as the sum over all the pairs of distinct indices; the second term is a sum over all quadruples of indices that are not all identical. It contains only off-diagonal terms, that is, cross-cumulants. Since cross-cumulants of independent variables identically vanish, it is not surprising to see the mutual information approximated by a sum of squared cross-cumulants. 3.1.3 Approximation to the Orthogonal Likelihood Contrast. The cumulant approximation to the orthogonal likelihood is fairly simple. The orthogonal approach consists of first enforcing the whiteness of Y that is RYij = δij or RY = I. In other words, it consists of normalizing the components by assuming that σ 2 (Si ) = 1 and making sure the second-order mismatch is zero. This is equivalent to replacing the weight 14 in equation 3.6 by an infinite weight, hence reducing the problem to the minimization (under the whiteness constraint) of the fourth-order mismatch, or the second (quadruple) sum in equation 3.6. Thus, the orthogonal likelihood contrast is approximated by def
OML (Y) = φ24
´2 1 X³ Y Qijkl − k(Si )δijkl . 48 ijkl
(3.8)
This contrast has an interesting alternate expression. Developing the squares gives 1 X Y 2 1 X 2 2 X OML 2 (Y) = (Qijkl ) + k (Si )δijkl − k(Si )δijkl QYijkl . φ24 48 ijkl 48 ijkl 48 ijkl The first sum above is constant under the whiteness constraint (this is readily checked using equation 3.13 for an orthonormal transform), and the second sum does not depend on Y; finally the last sum contains only diagonal nonzero terms. It follows that: 1 X k(Si )QYiiii 24 i 1 X 1 X c =− k(Si )k(Yi ) = − k(Si )EY¯ i4 , 24 i 24 i c
OML (Y) = − φ24
(3.9)
High-Order Contrasts for Independent Component Analysis
167
c
where · = · denotes an equality up to a constant. An interpretation of the second equality is that the contrast is minimized by maximizing the scalar product between the vector [k(Y1 ), . . . , k(Yn )] of the kurtosis of the components and the corresponding vector of hypothesized kurtosis [k(S1 ), . . . , k(Sn )]. The last equality stems from the definition in equation 3.2 of the kurtosis and the constancy of EY¯ i2 under the whiteness constraint. This last form is c OML (Y) = remarkable because it shows that for zero-mean observations, φ24 P 1 4 El(Y), where l(Y) = − 24 i k(Si )Yi , so the contrast is just the expectation of a simple function of Y. We can expect simple techniques for its maximization. 3.1.4 Approximation to the Minimum Marginal Entropy Contrast. Under the whiteness constraint, the first sum in the approximation, equation 3.7, is zero (this is the whiteness constraint) so that the approximation to mutual information φ MI (Y) reduces to the last term: def
ME (Y) = φ ME (Y) ≈ φ24
1 X ³ Y ´2 c 1 X ³ Y ´2 Qijkl = − Qiiii . 48 ijkl6=iiii 48 i
(3.10)
P Again, the last equality up to constant follows from the constancy of ijkl (QYijkl )2 under the whiteness constraint. These approximations had already been obtained by Comon (1994) from an Edgeworth expansion. They say something simple: Edgeworth expansions suggest testing the independence between the entries of Y by summing up all the squared cross-cumulants. In the course of this article, we will find two similar contrast functions. The JADE contrast, def
φ JADE (Y) =
´2 X ³ QYijkl ,
(3.11)
ijkl6=iikl
also is a sum of squared cross-cumulants (the notation indicates a sum is over all the quadruples (ijkl) of indices with i 6= j). Its interest is to be also a criterion of joint diagonality of cumulants matrices. The SHIBBS criterion, def
φ SH (Y) =
´2 X ³ QYijkl ,
(3.12)
ijkl6=iikk
is also introduced in section 4.3 as governing a similar but less memorydemanding algorithm. It also involves only cross-cumulants: those with indices (ijkl) such that i 6= j or k 6= l. 3.2 Cumulants and Algebraic Structures. Previous sections reviewed the use of cumulants in designing contrast functions. Another thread of ideas using cumulants stems from the method of moments. Such an approach is called for by the multilinearity of the cumulants. Under a linear
168
Jean-Fran¸cois Cardoso
transform Y = BX, which also reads Yi = (for instance) transform as: Cum(Yi , Yj , Yk , Yl ) =
X
P
p bip Xp , the cumulants of order 4
bip bjq bkr bls Cum(Xp , Xq , Xr , Xs ),
(3.13)
pqrs
which can easily be exploited for our purposes since the ICA model is linear. Using this fact and the assumption of independence by which Cum(Sp , Sq , Sr , Ss ) = k(Sp )δ(p, q, r, s), we readily obtain the simple algebraic structure of the cumulants of X = AS when S has independent entries, Cum(Xi , Xj , Xk , Xl ) =
n X
k(Su )aiu aju aku alu ,
(3.14)
u=1
[ um(Xi , Xj , where aij denotes the (ij)th entry of matrix A. When estimates C Xk , Xl ) are available, one may try to solve equation 3.4 in the coefficients aij of A. This is tantamount to cumulant matching on the empirical cumulants of X. Because of the strong algebraic structure of equation 3.14, one may try to devise fourth-order factorizations akin to the familiar second-order singular value decomposition (SVD) or eigenvalue decomposition (EVD) (see Cardoso, 1992; Comon, 1997; De Lathauwer, De Moor, & Vandewalle, 1996). However, these approaches are generally not equivalent to the optimization of a contrast function, resulting in estimates that are generally not equivariant (Cardoso, 1995). This point is illustrated below; we introduce cumulant matrices whose simple structure offers straightforward identification techniques, but we stress, as one of their important drawbacks, their lack of equivariance. However, we conclude by showing how the algebraic point of view and the statistical (equivariant) point of view can be reconciled. 3.2.1 Cumulant Matrices. The algebraic nature of cumulants is tensorial (McCullagh, 1987), but since we will concern ourselves mainly with secondand fourth-order statistics, a matrix-based notation suffices for the purpose of our exposition; we only introduce the notion of cumulant matrix defined as follows. Given a random n × 1 vector X and any n × n matrix M, we define the associated cumulant matrix QX (M) as the n × n matrix defined component-wise by def
[QX (M)]ij =
n X
Cum(Xi , Xj , Xk , Xl )Mkl .
(3.15)
k,l=1
If X is centered, the definition in equation 3.1 shows that QX (M) = E{(X† MX) XX† } − RX tr(MRX ) − RX MRX − RX M† RX ,(3.16)
High-Order Contrasts for Independent Component Analysis
169
where tr(·) denotes the trace and RX denotes the covariance matrix of X, that is, [RX ]ij = Cum(Xi , Xj ). Equation 3.16 could have been chosen as an index-free definition of cumulant matrices. It shows that a given cumulant matrix can be computed or estimated at a cost similar to the estimation cost of a covariance matrix; there is no need to compute the whole set of fourthorder cumulants to obtain the value of QX (M) for a particular value of M. Actually, estimating a particular cumulant matrix is one way of collecting part of the fourth-order information in X; collecting the whole fourth-order information requires the estimation of O(n4 ) fourth-order cumulants. The structure of a cumulant matrix QX (M) in the ICA model is easily deduced from equation 3.14: QX (M) = A1(M)A† ³ ´ 1(M) = Diag k(S1 ) a†1 Ma1 , . . . , k(Sn ) a†n Man ,
(3.17)
where ai denotes the ith column of A, that is, A = [a1 , . . . , an ]. In this factorization, the (generally unknown) kurtosis enter only in the diagonal matrix 1(M), a fact implicitly exploited by the algebraic techniques described below. 3.3 Blind Identification Using Algebraic Structures. In section 3.1, contrast functions were derived from the ML principle assuming the model X = AS. In this section, we proceed similarly: we consider cumulant-based blind identification of A assuming X = AS from which the structures 3.14 and 3.17 result. Recall that the orthogonal approach can be implemented by first sphering def
explicitly vector X. Let W be a whitening, and denote Z = WX the sphered vector. Without loss of generality, the model can be normalized by assuming that the entries of S have unit variance so that S is spatially white. Since def
Z = WX = WAS is also white by construction, the matrix U = WA must be orthonormal: UU† = I. Therefore sphering yields the model Z = US with U orthonormal. Of course, this is still a model of independent components so that, similar to equation 3.17, we have for any matrix M the structure of the corresponding cumulant matrix of Z, † ˜ QZ (M) = U1(M)U ³ ´ ˜ 1(M) = Diag k(S1 ) u†1 Mu1 , . . . , k(Sn ) u†n Mun ,
(3.18)
where ui denotes the ith column of U. In a practical orthogonal statisticb estimate based technique, one would first estimate a whitening matrix W, b compute an orthonormal estimate U b of U some cumulants of Z = WX, b −1 U b of A as A b= W b using these cumulants, and finally obtain an estimate A −1 † b b b b or obtain a separating matrix as B = U W = U W.
170
Jean-Fran¸cois Cardoso
3.3.1 Nonequivariant Blind Identification Procedures. We first present two blind identification procedures that exploit in a straightforward manner the structure 3.17; we explain why, in spite of attractive computational simplicity, they are not well behaved (not equivariant) and how they can be fixed for equivariance. The first idea is not based on an orthogonal approach. Let M1 and M2 def
def
be two arbitrary n × n matrices, and define Q1 = QX (M1 ) and Q2 = QX (M2 ). According to equation 3.17, if X = AS we have Q1 = A11 A† and Q2 = A12 A† with 11 and 12 two diagonal matrices. Thus, G = Q1 Q−1 2 = (A11 A† )(A12 A† )−1 = A1A−1 , where 1 is the diagonal matrix 11 1−1 2 . It follows that GA = A1, meaning that the columns of A are the eigenvectors of G (possibly up to scale factors). An extremely simple algorithm for blind identification of A follows: Select two arbitrary matrices M1 and M2 ; compute sample estimates Qˆ 1 and Qˆ 2 using equation 3.16; find the columns of A as the eigenvectors of Qˆ 1 Qˆ −1 2 . There is at least one problem with this idea: we have assumed invertible matrices throughout the derivation, and this may lead to instability. However, this specific problem may be fixed by sphering, as examined next. Consider now the orthogonal approach as outlined above. Let M be some arbitrary matrix M, and note that equation 3.18 is an eigendecomposition: the columns of U are the eigenvectors of QZ (M), which are orthonormal indeed because QZ (M) is symmetric. Thus, in the orthogonal approach, another immediate algorithm for blind identification is to estimate U as an (orthonormal) diagonalizer of an estimate of QZ (M). Thanks to sphering, problems associated with matrix inversion disappear, but a deeper problem associated with these simple algebraic ideas remains and must be addressed. Recall that the eigenvectors are uniquely determined2 if and only if the eigenvalues are all distinct. Therefore, we need to make sure that the eigenvalues of QZ (M) are all distinct in order to preserve blind identifiability based on QZ (M). According to equation 3.18, these eigenvalues depend on the (sphered) system, which is unknown. Thus, it is not possible to determine a priori if a given matrix M corresponds to distinct eigenvalues of QZ (M). Of course, if M is randomly chosen, then the eigenvalues are distinct with probability 1, but we need more than this in practice because the algorithms use only sample estimates of the cumulant matrices. A small error in the sample estimate of QZ (M) can induce a large deviation of the eigenvectors if the eigenvalues are not well enough separated. Again, this is impossible to guarantee a priori because an appropriate selection of M requires prior knowledge about the unknown mixture. In summary, the diagonalization of a single cumulant matrix is computadef
2 In fact, determined only up to permutations and signs that do not matter in an ICA context.
High-Order Contrasts for Independent Component Analysis
171
tionally attractive and can be proved to be almost surely consistent, but it is not satisfactory because the nondegeneracy of the spectrum cannot be controlled. As a result, the estimation accuracy from a finite number of samples depends on the unknown system and is therefore unpredictable in practice; this lack of equivariance is hardly acceptable. One may also criticize these approaches on the ground that they rely on only a small part of the fourth-order information (summarized in an n × n cumulant matrix) rather than trying to exploit more cumulants (there are O(n4 ) fourth-order independent cumulant statistics). We examine next how these two problems can be alleviated by jointly processing several cumulant matrices. 3.3.2 Recovering Equivariance. Let M = {M1 , . . . , MP } be a set of P madef
trices of size n × n and denote Qi = QZ (Mi ) for 1 ≤ i ≤ P the associated cumulant matrices for the sphered data Z = US. Again, as above, for all i we have Qi = U1i U† with 1i a diagonal matrix given by equation 3.18. As a measure of nondiagonality of a matrix F, define Off(F) as the sum of the squares of the nondiagonal elements: def
Off(F) =
X ¡ ¢2 fij .
(3.19)
i6=j
We have in particular Off(U† Qi U) = Off(1i ) = 0 since Qi = U1i U† and U† U = I. For any matrix set M and any orthonormal matrix V, we define the following nonnegative joint diagonality criterion, def
DM (V) =
X
Off(V† QZ (Mi )V),
(3.20)
Mi ∈M
which measures how close to diagonality an orthonormal matrix V can simultaneously bring the cumulants matrices generated by M. To each matrix set M is associated a blind identification algorithm as follows: (1) find a sphering matrix W to whiten in the data X into Z = WX; (2) estimate the cumulant matrices QZ (M) for all M ∈ M by a sample version of equation 3.16; (3) minimize the joint diagonality criterion, equation 3.20, that is, make the cumulant matrices as diagonal as possible by an orthonormal transform V; (4) estimate A as A = VW −1 or its inverse as B = V † W or the component vector as Y = V † Z = V † WX. Such an approach seems to be able to alleviate the drawbacks mentioned above. Finding the orthonormal transform as the minimizer of a set of cumulant matrices goes in the right direction because it involves a larger number of fourth-order statistics and because it decreases the likelihood of degenerate spectra. This argument can be made rigorous by considering a maximal set of cumulant matrices. By definition, this is a set obtained whenever M is an orthonormal basis for the linear space of n × n matrices. Such a basis contains n2 matrices so that the corresponding cumulant matrices total
172
Jean-Fran¸cois Cardoso
n2 × n2 = n4 entries, that is, as many as the whole fourth-order cumulant set. For any such maximal set (Cardoso & Souloumiac, 1993):
DM (V) = φ JADE (Y) with Y = V † Z,
(3.21)
where φ JADE (Y) is the contrast function defined at equation 3.11. The joint diagonalization of a maximal set guarantees blind identifiability of A if k(Si ) = 0 for at most one entry Si of S (Cardoso & Souloumiac, 1993). This is a necessary condition for any algorithm using only second- and fourth-order statistics (Comon, 1994). A key point is made by relationship 3.21. We managed to turn an algebraic property (diagonality) of the cumulants of the (sphered) observations into a contrast function—a functional of the distribution of the output Y = V † Z. This fact guarantees that the resulting estimates are equivariant (Cardoso, 1995). The price to pay with this technique for reconciling the algebraic approach with the naturally equivariant contrast-based approach is twofold: it entails the computation of a large (actually, maximal) set of cumulant matrices and the joint diagonalization of P = n2 matrices, which is at least as costly as P times the diagonalization of a single matrix. However, the overall computational burden may be similar (see examples in section 5) to the cost of adaptive algorithms. This is because the cumulant matrices need to be estimated once for a given data set and because it exists as a reasonably efficient joint diagonalization algorithm (see section 4) that is not based on gradient-style optimization; it thus preserves the possibility of exploiting the underlying algebraic nature of the contrast function, equation 3.11. Several tricks for increasing efficiency are also discussed in section 4. 4 Jacobi Algorithms This section describes algorithms for ICA sharing a common feature: a Jacobi optimization of an orthogonal contrast function as opposed to optimization by gradient-like algorithms. The principle of Jacobi optimization is applied to a data-based algorithm, a statistic-based algorithm, and a mixed approach. The Jacobi method is an iterative technique of optimization over the set of orthonormal matrices. The orthonormal transform is obtained as a sequence of plane rotations. Each plane rotation is a rotation applied to a pair of coordinates (hence the name: the rotation operates in a two-dimensional plane). If Y is an n×1 vector, the (i, j)th plane rotation by an angle θij changes the coordinates i and j of Y according to ¸· ¸ · ¸ · Yi cos(θij ) sin(θij ) Yi ← , (4.1) Yj Yj − sin(θij ) cos(θij ) while leaving the other coordinates unchanged. A sweep is one pass through
High-Order Contrasts for Independent Component Analysis
173
all the n(n − 1)/2 possible pairs of distinct indices. This idea is classic in numerical analysis (Golub & Van Loan, 1989); it can be considered in a wider context for the optimization of any function of an orthonormal matrix. Comon introduced the Jacobi technique for ICA (see Comon, 1994 for a data-based algorithm and an earlier reference in it for the Jacobi update of high-order cumulant tensors). Such a data-based Jacobi algorithm for ICA works through a sequence of Jacobi sweeps on the sphered data until a given orthogonal contrast φ(Y) is optimized. This can be summarized as: 1. Initialization. Compute a whitening matrix W and set Y = WX. 2. One sweep. For all n(n − 1)/2 pairs, that is for 1 ≤ i < j ≤ n, do: a. Compute the Givens angle θij , optimizing φ(Y) when the pair (Yi , Yj ) is rotated. b. If θij < θmin , do rotate the pair (Yi , Yj ) according to equation 4.1. 3. If no pair has been rotated in previous sweep, end. Otherwise go to 2 for another sweep. Thus, the Jacobi approach considers a sequence of two-dimensional ICA problems. Of course, the updating step 2b on a pair (i, j) partially undoes the effect of previous optimizations on pairs containing either i or j. For this reason, it is necessary to go through several sweeps before optimization is completed. However, Jacobi algorithms are often very efficient and converge in a small number of sweeps (see the examples in section 5), and a key point is that each plane rotation depends on a single parameter, the Givens angle θij , reducing the optimization subproblem at each step to a one-dimensional optimization problem. An important benefit of basing ICA on fourth-order contrasts becomes apparent: because fourth-order contrasts are polynomial in the parameters, the Givens angles can often be found in close form. In the above scheme, θmin is a small angle, which controls the accuracy of the optimization. In numerical analysis, it is determined according to machine precision. For a statistical problem as ICA, θmin should be selected in such a way that rotations by a smaller angle are not statistically significant. √ −2 √ . In our experiments, we take θmin to scale as 1/ T, typically: θmin = 10 T This scaling can be related to the existence of a performance bound in the orthogonal approach to ICA(Cardoso, 1994). This value does not seem to be critical, however, because we have found Jacobi algorithms to be very fast at finishing. In the remainder of this section, we describe three possible implementations of these ideas. Each one corresponds to a different type of contrast function and to different options about updating. Section 4.1 describes a data-based algorithm optimizing φ OML (Y); section 4.2 describes a statisticbased algorithm optimizing φ JADE (Y); section 4.3 presents a mixed approach
174
Jean-Fran¸cois Cardoso
optimizing φ SH (Y); finally, section 4.4 discusses the relationships between these contrast functions. 4.1 A Data-Based Jacobi Algorithm: MaxKurt. We start by a Jacobi technique for optimizing the approximation, equation 3.9, to the orthogonal likelihood. For the sake of exposition, we consider a simplified version of φ OML (Y) obtained by setting k(S1 ) = k(S2 ) = . . . = k(Sn ) = k, in which case the minimization of contrast function, equation 3.9, is equivalent to the minimization of def
φ MK (Y) = −k
X
QYiiii .
(4.2)
i
This criterion is also studied by Moreau and Macchi (1996), who propose a two-stage adaptive procedure for its optimization; it also serves as a starting point for introducing the one-stage adaptive algorithm of Cardoso and Laheld (1996). Denote Gij (θ) the plane rotation matrix that rotates the pair (i, j) by an angle θ as in step 2b above. Then simple trigonometry yields: φ MK (Gij (θ)Y) = µij − kλij cos(4(θ − Äij )),
(4.3)
where µij does not depend on θ and λij is nonnegative. The principal determination of angle Äij is characterized by Äij =
³ ´ 1 Y arctan 4QYiiij − 4QYijjj , QYiiii + Qjjjj − 6QYiijj , 4
(4.4)
where arctan(y, x) denotes the angle α ∈ (−π, π ] such that cos(α) = √ and sin(α) = √
y
x2 +y2
x x2 +y2
. If Y is a zero-mean sphered vector, expression 4.3
further simplifies to Äij =
³ ³ ´ ³ ´´ 1 arctan 4E Yi3 Yj − Yi Yj3 , E (Yi2 − Yj2 )2 − 4Yi2 Yj2 . (4.5) 4
The computations are given in the appendix. It is now immediate to minimize φ MK (Y) for each pair of components and for either choice of the sign of k. If one looks for components with positive kurtosis (often called supergaussian), the minimization of φ MK (Y) is identical to the maximization of the sum of the kurtosis of the components since we have k > 0 in this case. The Givens angle simply is θ = Äij since this choice makes the cosine in equation 4.2 equal to its maximum value. We refer to the Jacobi algorithm outlined above as MaxKurt. A Matlab implementation is listed in the appendix, whose simplicity is consistent with the data-based approach. Note, however, that it is also possible to use
High-Order Contrasts for Independent Component Analysis
175
the same computations in a statistic-based algorithm. Rather than rotating the data themselves at each step by equation 4.1, one instead updates the set of all fourth-order cumulants according to the transformation law, equation 3.13, with the Givens angle for each pair still given by equation 4.3. In this case, the memory requirement is O(n4 ) for storing all the cumulants as opposed to nT for storing the data set. The case k < 0 where, looking for light-tailed components, one should minimize the sum of the kurtosis is similar. This approach could be extended to kurtosis of mixed signs but the contrast function then has less symmetry. This is not included in this article. 4.1.1 Stability. What is the effect of the approximation of equal kurtosis made to derive the simple contrast φ MK (Y)? When X = AS with S of independent components, we can at least use the stability result of Cardoso and Laheld (1996), which applies directly to this contrast. Define the normalized kurtosis as κi = σi−4 k(Si ). Then B = A−1 is a stable point of the algorithm with k > 0 if κi + κj > 0 for all pairs 1 ≤ i < j ≤ n. The same condition also holds with all signs reversed for components with negative kurtosis. 4.2 A Statistic-Based Algorithm: JADE. This section outlines the JADE algorithm (Cardoso & Souloumiac, 1993), which is specifically a statisticbased technique. We do not need to go into much detail because the general technique follows directly from the considerations of section 3.3. The JADE algorithm can be summarized as: ˆ and set Z = WX. ˆ 1. Initialization. Estimate a whitening matrix W 2. Form statistics. Estimate a maximal set {Qˆ Zi } of cumulant matrices. 3. Optimize an orthogonal contrast. Find the rotation matrix Vˆ such that the cumulant matrices are as diagonal as possible, that is, solve Vˆ = P ˆ Z V). arg min i Off(V† Q i ˆ −1 and/or estimate the components 4. Separate. Estimate A as Aˆ = Vˆ W as Sˆ = Aˆ −1 X = Vˆ † Z. This is a Jacobi algorithm because the joint diagonalizer at step 3 is found by a Jacobi technique. However, the plane rotations are applied not to the data (which are summarized in the cumulant matrices) but to the cumulant matrices themselves; the algorithm updates not data but matrix-valued statistics of the data. As with MaxKurt, the Givens angle at each step can be computed in closed form even in the case of possibly complex matrices (Cardoso & Souloumiac, 1993). The explicit expression for the Givens angles is not particularly enlightening and is not reported here. (The interested reader is referred to Cardoso & Souloumiac, 1993, and may request a Matlab implementation from the author.)
176
Jean-Fran¸cois Cardoso
A key issue is the selection of the cumulant matrices to be involved in the estimation. As explained in section 3.2, the joint diagonalization criterion P †ˆZ i Off(V Qi V) is made identical to the contrast function, equation 3.11, by using a maximal set of cumulant matrices. This is a bit surprising but very fortunate. We do not know of any other way for a priori selecting cumulant matrices that would offer such a property (but see the next section). In any case, it guarantees equivariant estimates because the algorithm, although operating on statistics of the sphered data, also optimizes implicitly a function of Y = V † Z only. Before proceeding, we note that true cumulant matrices can be exactly jointly diagonalized when the model holds, but this is no longer the case when we process real data. First, only sample statistics are available; second, the model X = AS with independent entries in S cannot be expected to hold accurately in general. This is another reason that it is important to P ˆ Z V) is a contrast function. select cumulant matrices such that i Off(V† Q i In this case, the impossibility of an exact joint diagonalization corresponds to the impossibility of finding Y = BX with independent entries. Making a maximal set of cumulant matrices as diagonal as possible coincides with making the entries of Y as independent as possible as measured by (the sample version of) criterion 3.11. There are several options for estimating a maximal set of cumulant matrices. Recall that such a set is defined as {QZ (Mi )|i = 1, n2 } where {Mi |i = 1, n2 } is any basis for the n2 -dimensional linear space of n×n matrices. A canonical basis for this space is {ep e†q |1 ≤ p, q ≤ n}, where ep is a column vector with a 1 in pth position and 0’s elsewhere. It is readily checked that [QZ (ep e†q )]ij = Cum(Zi , Zj , Zp , Zq ).
(4.6)
In other words, the entries of the cumulant matrices for the canonical basis are just the cumulants of Z. A better choice is to consider a symmetric/skewsymmetric basis. Denote Mpq an n × n matrix defined as follows: Mpq = ep ep† if p = q, Mpq = 2−1/2 (ep e†q +eq ep† ) if p < q and Mpq = 2−1/2 (ep e†q −eq ep† ) if p > q. This is an orthonormal basis of Rn×n . We note that because of the symmetries of the cumulants QZ (ep e†q ) = QZ (eq ep† ) so that QZ (Mpq ) = 2−1/2 QZ (ep e†q ) if p < q and QZ (Mpq ) = 0 if p > q. It follows that the cumulant matrices QZ (Mpq ) for p > q do not even need to be computed. Being identically zero, they do not enter in the joint diagonalization criterion. It is therefore sufficient to estimate and to diagonalize n+n(n−1)/2 (symmetric) cumulant matrices. There is another idea to reduce the size of the statistics needed to represent exhaustively the fourth-order information. It is, however, applicable only when the model X = AS holds. In this case, the cumulant matrices do have the structure shown at equation 3.18, and their sample estimates are close to it for large enough T. Then the linear mapping M → QZ (M) has rank n
High-Order Contrasts for Independent Component Analysis
177
(more precisely, its rank is equal to the number of components with nonzero kurtosis) because there are n linear degrees of freedom for matrices in the form U1U† , namely, the n diagonal entries of 1. From this fact and from the symmetries of the cumulants, it follows that it exists n eigenmatrices E1 , . . . , En , which are orthonormal, and satisfies QZ (Ei ) = µi Ei where the scalar µi is the corresponding eigenvector. These matrices E1 , . . . , En span the range of the mapping M → QZ (M), and any matrix M orthogonal to them is in the kernel, that is, QZ (M) = 0. This shows that all the information contained in QZ can be summarized by the n eigenmatrices associated with the n nonzero eigenvalues. By inserting M = ui u†i in the expressions 3.18 and using the orthonormality of the columns of U (that is, u†i uj = δij ), it is readily checked that a set of eigenmatrices is {Ei = ui u†i }. The JADE algorithm was originally introduced as performing ICA by a joint approximate diagonalization of eigenmatrices in Cardoso and Souloumiac (1993), where we advocated the joint diagonalization of only the n most significant eigenmatrices of QZ as a device to reduce the computational load (even though the eigenmatrices are obtained at the extra cost of the eigendecomposition of an n2 × n2 array containing all the fourth-order cumulants). The number of statistics is reduced from n4 cumulants or n(n + 1)/2 symmetric cumulant matrices of size n×n to a set of n eigenmatrices of size n×n. Such a reduction is achieved at no statistical loss (at least for large T) only when the model holds. Therefore, we do not recommend reduction to eigenmatrices when processing data sets for which it is not clear a priori whether the model X = AS actually holds to good accuracy. We still refer to JADE as the process of jointly diagonalizing a maximal set of cumulant matrices, even when it is not further reduced to the n most significant eigenmatrices. It should also be pointed out that the device of truncating the full cumulant set by reduction to the most significant matrices is expected to destroy the equivariance property when the model does not hold. The next section shows how these problems can be overcome in a technique borrowing from both the data-based approach and the statistic-based approach. 4.3 A Mixed Approach: SHIBBS. In the JADE algorithm, a maximal set of cumulant matrices is computed as a way to ensure equivariance from the joint diagonalization of a fixed set of cumulant matrices. As a benefit, cumulants are computed only once in a single pass through the data set, and the Jacobi updates are performed on these statistics rather than on the whole data set. This is a good thing for data sets with a large number T of samples. On the other hand, estimating a maximal set requires O(n4 T) operations, and its storage requires O(n4 ) memory positions. These figures can become prohibitive when looking for a large number of components. In contrast, gradient-based techniques have to store and update nT samples. This section describes a technique standing between the two extreme positions represented by the all-statistic approach and the all-data approach.
178
Jean-Fran¸cois Cardoso
Recall that an algorithm is equivariant as soon as its operation can be expressed only in terms of the extracted components Y (Cardoso, 1995). This suggests the following technique: 1. Initialization. Select a fixed set M = {M1 , . . . , MP } of n × n matrices. ˆ and set Y = WX. ˆ Estimate a whitening matrix W 2. Estimate a rotation. Estimate the set {Qˆ Y (Mp )|1 ≤ p ≤ P} of P cumulant matrices and find a joint diagonalizer V of it. 3. Update. If V is close enough to the identity transform, stop. Otherwise, rotate the data: Y ← V † Y and go to 2 . Such an algorithm is equivariant thanks to the reestimation of the cumulants of Y after updating. It is in some sense data based since the updating in step 3 is on the data themselves. However, the rotation matrix to be applied to the data is computed in step 2 as in a statistic-based procedure. What would be a good choice for the set M? The set of n matrices M = {e1 e†1 , . . . , en e†n } seems a natural choice: it is an order of magnitude smaller than the maximal set, which contains O(n2 ) matrices. The kth cumulant matrix in such a set is QY (ek e†k ), and its (i, j)th entry is Cum(Yi , Yj , Yk , Yk ), which is just an n × n square block of cumulants of Y. We call the set of n cumulant matrices obtained in this way when k is shifted from 1 to n the set of SHIfted Blocks for Blind Separation (SHIBBS), and we use the same name for the ICA algorithm that determines the rotation V by an iterative joint diagonalization of the SHIBBS set. Strikingly enough, the small SHIBBS set guarantees a performance identical to JADE when the model holds for the following reason. Consider the final step of the algorithm where Y is close to S if it holds that X = AS with S of independent components. Then the cumulant matrices QY (ep e†q ) are zero for p 6= q because all the cross-cumulants of Y are zero. Therefore, the only nonzero cumulant matrices used in the maximal set of JADE are those corresponding to ep = eq , i.e. precisely those included in SHIBBS. Thus the SHIBBS set actually tends to the set of “significant eigen-matrices” exhibited in the previous section. In this sense, SHIBBS implements the original program of JADE—the joint diagonalization of the significant eigen-matrices—but it does so without going through the estimation of the whole cumulant set and through the computation of its eigen-matrices. Does the SHIBBS algorithm correspond to the optimization of a contrast function? We cannot resort to the equivalence of JADE and SHIBBS because it is established only when the model holds and we are looking for a statement independent of this later fact. Examination of the joint diagonality criterion for the SHIBBS set suggests that the SHIBBS technique solves the problem of optimizing the contrast function φ SH (Y) defined in equation 3.12. As a matter of fact, the condition for a given Y to be a fixed
High-Order Contrasts for Independent Component Analysis
179
point of the SHIBBS algorithm is that for any pair 1 ≤ i < j ≤ n: X Cum(Yi , Yj , Yk , Yk ) (Cum(Yi , Yi , Yk , Yk ) k
¢ − Cum(Yj , Yj , Yk , Yk ) = 0,
(4.7)
and we can prove that this is also the stationarity condition of φ SH (Y). We do not include the proofs of these statements, which are purely technical. 4.4 Comparing Fourth-Order Orthogonal Contrasts. We have considered two approximations, φ JADE (Y) and φ SH (Y), to the minimum marginal entropy/mutual information contrast φ MI (I), which are based on fourthorder cumulants and can be optimized by Jacobi technique. The approxiME (Y) proposed by Comon also belongs to this category. One may mation φ24 wonder about the relative statistical merits of these three approximations. ME (Y) stems from an Edgeworth expansion for approximatThe contrast φ24 ME ing φ (Y), which in turn has been shown to derive from the ML principle (see section 2). Since ML estimation offers (asymptotic) optimality properME (Y). However, ties, one may be tempted to conclude to the superiority of φ24 this is not the case, as discussed now. First, when the ICA model holds, it can be shown that even though ME (Y) and φ JADE (Y) are different criteria, they have the same asymptotic φ24 performance when applied to sample statistics (Souloumiac & Cardoso, 1991). This is also true of φ SH (Y) since we have seen that it is equivalent to JADE in this case (a more rigorous proof is possible, based on equation 4.7, but is not included). Second, when the ICA model does not hold, the notion of identification accuracy does not make sense anymore, but one would certainly favor an orthogonal contrast reaching its minimum at a point as close as possible to the point where the “true” mutual information φ ME (Y) is minimized. However, it seems difficult to find a simple contrast (such as those considered here) that would be a good approximation to φ ME (Y) for any wide ME (Y) class of distributions of X. Note that the ML argument in favor of φ24 is based on an Edgeworth expansion that is valid for “almost gaussian” distributions—those distributions that make ICA very difficult and of dubious significance: In practice, ICA should be restricted to data sets where the components show a significant amount of nongaussianity, in which case the Edgeworth expansions cannot be expected to be accurate. ME (Y). There is another way than Edgeworth expansion for arriving at φ24 Consider cumulant matching: the matching of the cumulants of Y to the corresponding cumulants of a hypothetical vector S with independent comME (Y) ponents. The orthogonal contrast functions φ JADE (Y), φ SH (Y), and φ24 can be seen as matching criteria because they penalize the deviation of the cross-cumulants of Y from zero (which is the value of cross-cumulants of a vector S with independent components, indeed), and they do so under
180
Jean-Fran¸cois Cardoso
the constraint that Y is white—that is, by enforcing an exact match of the second-order cumulants of Y. It is possible to devise an asymptotically optimal matching criterion by taking into account the variability of the sample estimates of the cumulants. Such a computation is reported in Cardoso et al. (1996) for the matching of all second- and fourth-order cumulants of complex-valued signals, but a similar computation is possible for real-valued problems. It shows that the optimal weighting of the cross-cumulants depends on the distributions of the components so that the “flat weighting” of all the cross-cumulants, as in equation 3.10, is not the best one in general. However, in the limit of “almost gaussian” signals, the optimal weights tend to values corresponding precisely to the contrast K24 (Y|S) defined in equation 3.5. This is not unexpected and confirms that the crude cumulant expansion used in deriving equation 3.5 is sensible, though not optimal for significantly nongaussian components. ME (Y) that these It seems from the definitions of φ JADE (Y), φ SH (Y), and φ24 different contrasts involve different types of cumulants. This is, however, an illusion because the compact definitions given above do not take into account the symmetries of the cumulants: the same cross-cumulant may be counted several times in each of these contrasts. For instance, the definition of JADE excludes the cross-cumulant Cum(Y1 , Y1 , Y2 , Y3 ) but includes the cross-cumulant Cum(Y1 , Y2 , Y1 , Y3 ), which is identical. Thus, in order to determine if any bit of fourth-order information is ignored by any particular contrast, a nonredundant description should be given. All the possible crosscumulants come in four different patterns of indices: (ijkl), (iikl), (iijj), and (ijjj). Nonredundant expressions in terms of these patterns are in the form: φ[Y] = Ca
X
eijkl + Cb
i<j
+ Cc
X i<j
def
X
(eiikl + ekkil + ellik )
i
eiijj + Cd
X¡ ¢ eijjj + ejjji , i<j
where eijkl = Cum(Yi , Yj , Yk , Yl )2 and the Ci ’s are numerical constants. It remains to count how many times a unique cumulant appears in the redundant definitions of the three approximations to mutual information considered so far. We give only the result of this uninspiring task in Table 1, which shows that all the cross-cumulants are actually included in the three contrasts, which therefore differ only by the different scalar weights given to each particular type. It means that the three contrasts essentially do the same thing. In particular, when the number n of components is large enough, the number of cross-cumulants of type [ijkl] (all indices distinct) grows as O(n4 ), while the number of other types grows as O(n3 ) at most. Therefore, the [ijkl] type outnumbers all the other types for large n: one may conjecture the equivalence of the three contrasts in this limit. Unfortunately, it seems
High-Order Contrasts for Independent Component Analysis
181
Table 1: Number of Times a Cross-Cumulant of a Given Type Appears in a Given Contrast. Constants
Ca
Cb
Cc
Cd
Pattern
ijkl
iikl
iijj
ijjj
Comon ICA JADE SHIBBS
24 24 24
12 10 12
6 4 4
4 2 4
difficult to draw more conclusions. For instance, we have mentioned the asymptotic equivalence between Comon’s contrast and the JADE contrast for any n, but it does not reveal itself directly in the weight table. 5 A Comparison on Biomedical Data The performance of the algorithms presented above is illustrated using the averaged event-related potential (ERP) data recorded and processed by Makeig and coworkers. A detailed account of their analysis is in Makeig, Bell, Jung, and Sejnowski (1997). For our comparison, we use the data set and the “logistic ICA” algorithm provided with version 3.1 of Makeig’s ICA toolbox.3 The data set contains 624 data points of averaged ERP sampled from 14 EEG electrodes. The implementation of the logistic ICA provided in the toolbox is somewhat intermediate between equation 1.1 and its off-line counterpart: H(Y) is averaged through subblocks of the data set. The nony linear function is taken to be ψ(y) = 1+e2 −y − 1 = tanh 2 . This is minus the r0 (y)
1 (β is a normallog-derivative ψ(y) = − r(y) of the density r(y) = β cosh(y/2) ization constant). Therefore, this method maximizes over A the likelihood of model X = AS under the assumptions that S has independent components 1 . with densities equal to β cosh(y/2)
Figure 1 shows the components YJADE produced by JADE (first column) and the components YLICA produced by the logistic ICA included in Makeig’s toolbox, which was run with all the default options; the third column shows the difference between the components at the same scale. This direct comparison is made possible with the following postprocessing: the components YLICA were normalized to have unit variance and were sorted by increasing values of kurtosis. The components YJADE have unit variance by construction; they were sorted and their signs were changed to match YLICA . Figure 1 shows that YJADE and YLICA essentially agree on 9 of 14 components. 3
Available from http://www.cnl.salk.edu/∼scott/.
182
Jean-Fran¸cois Cardoso
JADE
Logistic ICA
Difference
Figure 1: The source signals estimated by JADE and the logistic ICA and their differences.
High-Order Contrasts for Independent Component Analysis
183
Another illustration of this fact is given by the first row of Figure 2. The left panel shows the magnitude |Cij | of the entries of the transfer matrix C such that C YLICA = YJADE . This matrix was computed after the postprocessing of the components described in the previous paragraph: it should be the identity matrix if the two methods agreed, even only up to scales, signs, and permutations. The figure shows a strong diagonal structure in the northeast block while the disagreement between the two methods is apparent in the gray zone of the southwest block. The right panel shows JADE ) plotted against the kurtosis k(YiLICA ). A key observathe kurtosis k(Yi tion is that the two methods do agree about the most kurtic components; these also are the components where the time structure is the most visible. In other words, the two methods essentially agree wherever an human eye finds the most visible structures. Figure 2 also shows the results of SHIBBS and MaxKurt. The transfer matrix C for MaxKurt is seen to be more diagonal than the transfer matrix for JADE, while the transfer for SHIBBS is less diagonal. Thus, the logistic ICA and MaxKurt agree more on this data set. Another figure (not included) shows that JADE and SHIBBS are in very close agreement over all components. These results are very encouraging because they show that various ICA algorithms agree wherever they find structure on this particular data set. This is very much in support of the ICA approach to the processing of signals for which it is not clear that the model holds. It leaves open the question of interpreting the disagreement between the various contrast functions in the swamp of the low kurtosis domain. It turns out that the disagreement between the methods on this data set is, in our view, an illusion. Consider the eigenvalues λ1 , . . . , λn of the covariance matrix RX of the observations. They are plotted on a dB scale (this is 10 log10 λi ) in Figure 3. The two least significant eigenvalues stand rather clearly below the strongest ones with a gap of 5.5 dB. We take this as an indication that one should look for 12 linear components in this data set rather than 14, as in the previous experiments. The result is rather striking: by running JADE and the logistic ICA on the first 12 principal components, an excellent agreement is found over all the 12 extracted components, as seen on Figure 4. This observation also holds for MaxKurt and SHIBBS as shown by Figure 5. Table 2 lists the number of floating-point operations (as returned by Matlab) and the CPU time required to run the four algorithms on a SPARC 2 workstation. The MaxKurt technique was clearly the fastest here; however, it was applicable only because we were looking for components with positive kurtosis. The same is true for the version of logistic ICA considered in this experiment. It is not true of JADE or SHIBBS, which are consistent as soon as at most one source has a vanishing kurtosis, regardless of the sign of the nonzero kurtosis (Cardoso & Souloumiac, 1993). The logistic ICA required only about 50% more time than JADE. The SHIBBS algorithm is
184
Jean-Fran¸cois Cardoso
Transfer matrix
Comparing kurtosis
1 10
0.6 0.4
Logistic ICA
Logistic ICA
0.8
0.2 JADE
Transfer matrix
0
0
0
0.4 0.2
Transfer matrix
0 0
0.2 0
10
10
Logistic ICA
Logistic ICA
0.4
5 SHIBBS Comparing kurtosis
1 0.8
maxkurt
5
0
0.6
10
10
Logistic ICA
Logistic ICA
0.6
5 JADE Comparing kurtosis
1 0.8
SHIBBS
5
5
0 0
5 maxkurt
10
Figure 2: (Left column) Absolute values of the coefficients |Cij | of a matrix relating the signals obtained by two different methods. A perfect agreement would be for C = I: deviation from diagonal indicates a disagreement. The signals are sorted by kurtosis, showing a good agreement for high kurtosis. (Right column) Comparing the kurtosis of the sources estimated by two different methods. From top to bottom: JADE versus logistic ICA, SHIBBS versus logistic ICA, and maxkurt versus logistic ICA.
High-Order Contrasts for Independent Component Analysis
185
20 10 0 −10 −20 1
14
Figure 3: Eigenvalues of the covariance matrix RX of the data in dB (i.e., 10 log10 (λi )). Table 2: Number of Floating-Point Operations and CPU Time. Flops
CPU Secs.
Method
14 Components
Logistic ICA JADE SHIBBS MaxKurt
5.05e+07 4.00e+07 5.61e+07 1.19e+07
Flops
CPU Secs.
12 Components 3.98 2.55 4.92 1.09
3.51e+07 2.19e+07 2.47e+07 5.91e+06
3.54 1.69 2.35 0.54
slower than JADE here because the data set is not large enough to give it an edge. These remarks are even more marked when comparing the figures obtained in the extraction of 12 components. It should be clear that these figures do not prove much because they are representative of only a particular data set and of particular implementations of the algorithms, as well as of the various parameters used for tuning the algorithms. However, they do disprove the claim that algebraic-cumulant methods are of no practical value. 6 Summary and Conclusions The definitions of classic entropic contrasts for ICA can all be understood from an ML perspective. An approximation of the Kullback-Leibler divergence yields cumulant-based approximations of these contrasts. In the or-
186
Jean-Fran¸cois Cardoso
JADE
Logistic ICA
Difference
Figure 4: The 12 source signals estimated by JADE and a logistic ICA out of the first 12 principal components of the original data.
High-Order Contrasts for Independent Component Analysis
Transfer matrix
Comparing kurtosis
1 10
0.4
Logistic ICA
Logistic ICA
0.8 0.6
0.2 JADE
Transfer matrix
0 0
0.2
Transfer matrix
0 0
0.2 0
10
10
Logistic ICA
Logistic ICA
0.4
5 SHIBBS Comparing kurtosis
1 0.8
maxkurt
5
0
0.6
10
10
Logistic ICA
Logistic ICA
0.4
5 JADE Comparing kurtosis
1 0.8
SHIBBS
5
0
0.6
187
5
0 0
5 maxkurt
10
Figure 5: Same setting as for Figure 2 but the processing is restricted to the first 12 principal components, showing a better agreement among all the methods.
188
Jean-Fran¸cois Cardoso
thogonal approach to ICA where decorrelation is enforced, the cumulantbased contrasts can be optimized with Jacobi techniques, operating on either the data or statistics of the data, namely, cumulant matrices. The structure of the cumulants in the ICA model can be easily exploited by algebraic identification techniques, but the simple versions of these techniques are not equivariant. One possibility for overcoming this problem is to exploit the joint algebraic structure of several cumulant matrices. In particular, the JADE algorithm bridges the gap between contrast-based approaches and algebraic techniques because the JADE objective is both a contrast function and the expression of the eigenstructure of the cumulants. More generally, the algebraic nature of the cumulants can be exploited to ease the optimization of cumulant-based contrasts functions by Jacobi techniques. This can be done in a data-based or a statistic-based mode. The latter has an increasing relative advantage as the number of available samples increases, but it becomes impractical for large numbers n of components since the number of fourth-order cumulants grows as O(n4 ). This can be overcome to a certain extent by resorting to SHIBBS, which iteratively recomputes a number O(n3 ) of cumulants. An important objective of this article was to combat the prejudice that cumulant-based algebraic methods are impractical. We have shown that they compare very well to state-of-the-art implementations of adaptive techniques on a real data set. More extensive comparisons remain to be done involving variants of the ideas presented here. A technique like JADE is likely to choke on a very large number of components, but the SHIBBS version is not as memory demanding. Similarly, the MaxKurt method can be extended to deal with components with mixed kurtosis signs. In this respect, it is worth underlining the analogy between the MaxKurt update and the relative gradient update, equation 1.1, when function H(·) is in the form of equation 1.5. A comment on tuning the algorithms: In order to code an all-purpose ICA algorithm based on gradient descent, it is necessary to devise a smart learning schedule. This is usually based on heuristics and requires the tuning of some parameters. In contrast, Jacobi algorithms do not need to be tuned in their basic versions. However, one may think of improving on the regular Jacobi sweep through all the pairs in prespecified order by devising more sophisticated updating schedules. Heuristics would be needed then, as in the case of gradient descent methods. We conclude with a negative point about the fourth-order techniques described in this article. By nature, they optimize contrasts corresponding somehow to using linear-cubic nonlinear functions in gradient-based algorithms. Therefore, they lack the flexibility of adapting the activation functions to the distributions of the underlying components as one would ideally do and as is possible in algorithms like equation 1.1. Even worse, this very type of nonlinear function (linear cubic) has one major drawback: potential sensitivity to outliers. This effect did not manifest itself in the ex-
High-Order Contrasts for Independent Component Analysis
189
amples presented in this article, but it could indeed show up in other data sets. Appendix: Derivation and Implementation of MaxKurt A.1 Givens Angles for MaxKurt. An explicit form of the MaxKurt contrast as a function of the Givens angles is derived. For conciseness, we denote [ijkl] = QYijkl and we define aij =
[iiii] + [jjjj] − 6[iijj] 4
bij = [iiij] − [jjji]
λij =
q a2ij + b2ij .
(A.1)
The sum of the kurtosis for the pair of variables Yi and Yj after they have been rotated by an angle θ depends on θ as follows (where we set c = cos(θ ) and s = sin(θ)): k(cos(θ)Yi + sin(θ)Yj ) + k(− sin(θ )Yi + cos(θ)Yj ) 4
3
2 2
3
4
= c [iiii] + 4c s[iiij] + 6c s [iijj] + 4cs [ijjj] + s [jjjj] 4
3
2 2
3
4
+ s [iiii] − 4s c[iiij] + 6s c [iijj] − 4sc [ijjj] + c [jjjj]
(A.2) (A.3) (A.4)
= (c4 + s4 )([iiii] + [jjjj]) + 12c2 s2 [iijj] + 4cs(c2 − s2 )([iiij] − [jjji]) (A.5) [iiii] + [jjjj] − 6[iijj] c + 4cs(c2 − s2 )([iiij] − [jjji]) = −8c2 s2 (A.6) 4 c = −2 sin2 (2θ)aij + 2 sin(2θ ) cos(2θ )bij = cos(4θ )aij + sin(4θ )bij (A.7) ¢ ¡ = λij cos(4θ) cos(4Äij ) + sin(4θ ) sin(4Äij ) = λij cos(4(θ − Äij )). (A.8) where the angle 4Äij is defined by aij cos(4Äij ) = q a2ij + b2ij
sin(4Äij ) = q
bij a2ij + b2ij
.
(A.9)
This is obtained by using the multilinearity and the symmetries of the cumulants at lines A.3 and A.4, followed by elementary trigonometrics. If Yi and Yj are zero-mean and sphered, EYi Yj = δij , we have [iiii] = QYiiii = EYi4 − 3E2 Yi2 = EYi4 − 3 and for i 6= j: [iiij] = QYiiij = EYi3 Yj as well as [iijj] = QYiijj = EYi2 Yj2 − 1. Hence an alternate expression for aij and bij is: aij =
´ 1 ³ 4 E Yi + Yj4 − 6Yi2 Yj2 4
³ ´ bij = E Yi3 Yj − Yi Yj3 .
(A.10)
It may be interesting to note that all the moments required to determine the Givens angle for a given pair (i, j) can be expressed in terms of the two
190
Jean-Fran¸cois Cardoso
variables ξij = Yi Yj and ηij = Yi2 − Yj2 . Indeed, it is easily checked that for a zero-mean sphered pair (Yi , Yj ), one has
aij =
´ 1 ³ 2 E ηij − 4ξij2 4
¡ ¢ bij = E ηij ξij .
(A.11)
A.2 A Simple Matlab Implementation of MaxKurt. A Matlab implementation could be as follows, where we have tried to maximize readability but not the numerical efficiency: function Y = maxkurt(X) % [n T] = size(X) ; Y = X - mean(X,2)*ones(1,T); % Remove the mean Y = inv(sqrtm(X*X’/T))*Y ; % Sphere the data encore = 1 ; % Go for first sweep while encore, encore=0; for p=1:n-1, % These two loops go for q=p+1:n, % through all pairs xi = Y(p,:).*Y(q,:); eta = Y(p,:).*Y(p,:) - Y(q,:).*Y(q,:); Omega = atan2( 4*(eta*xi’), eta*eta’ - 4*(xi*xi’) ); if abs(Omega) > 0.1/sqrt(T)
% A ‘statistically % angle encore = 1 ; % This will not be %last sweep c = cos(Omega/4); s = sin(Omega/4); Y([p q],:) = [ c s ; -s c ] * Y([p q],:) ; % % end
small’ the
Plane rotation
end end end return
Acknowledgments We express our thanks to Scott Makeig and to his coworkers for releasing to the public domain some of their codes and the ERP data set. We also thank the anonymous reviewers whose constructive comments greatly helped in revising the first version of this article.
High-Order Contrasts for Independent Component Analysis
191
References Amari, S.-I. (1996). Neural learning in structured parameter spaces—Natural Riemannian gradient. In Proc. NIPS. Amari, S.-I., Cichocki, A., & Yang, H. (1996). A new learning algorithm for blind signal separation. In Advances in neural information processing systems, 8 (pp. 757–763). Cambridge, MA: MIT Press. Bell, A. J., & Sejnowski, T. J. (1995). An information-maximisation approach to blind separation and blind deconvolution. Neural Computation, 7, 1004–1034. Cardoso, J.-F. (1992). Fourth-order cumulant structure forcing. Application to blind array processing. In Proc. 6th SSAP workshop on statistical signal and array processing (pp. 136–139). Cardoso, J.-F. (1994). On the performance of orthogonal source separation algorithms. In Proc. EUSIPCO (pp. 776–779). Edinburgh. Cardoso, J.-F. (1995). The equivariant approach to source separation. In Proc. NOLTA (pp. 55–60). Cardoso, J.-F. (1997). Infomax and maximum likelihood for source separation. IEEE Letters on Signal Processing, 4, 112–114. Cardoso, J.-F. (1998). Blind signal separation: statistical principles. Proc. of the IEEE. Special issue on blind identification and estimation. Cardoso, J.-F., Bose, S., & Friedlander, B. (1996). On optimal source separation based on second and fourth order cumulants. In Proc. IEEE Workshop on SSAP. Corfu, Greece. Cardoso, J.-F., & Laheld, B. (1996). Equivariant adaptive source separation. IEEE Trans. on Sig. Proc., 44, 3017–3030. Cardoso, J.-F., & Souloumiac, A. (1993). Blind beamforming for non Gaussian signals. IEEE Proceedings-F, 140, 362–370. Comon, P. (1994). Independent component analysis, a new concept? Signal Processing, 36, 287–314. Comon, P. (1997). Cumulant tensors. In Proc. IEEE SP Workshop on HOS. Banff, Canada. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley. De Lathauwer, L., De Moor, B., & Vandewalle, J. (1996). Independent component analysis based on higher-order statistics only. In Proc. IEEE SSAP Workshop (pp. 356–359). Gaeta, M., & Lacoume, J. L. (1990). Source separation without a priori knowledge: The maximum likelihood solution. In Proc. EUSIPCO (pp. 621–624). Golub, G. H., & Van Loan, C. F. (1989). Matrix computations. Baltimore: Johns Hopkins University Press. MacKay, D. J. C. (1996). Maximum likelihood and covariant algorithms for independent component analysis. In preparation. Unpublished manuscript. Makeig, S., Bell, A., Jung, T.-P., & Sejnowski, T. J. (1997). Blind separation of auditory event-related brain responses into independent components. Proc. Nat. Acad. Sci. USA, 94, 10979–10984. McCullagh, P. (1987). Tensor methods in statistics. London: Chapman and Hall.
192
Jean-Fran¸cois Cardoso
Moreau, E., & Macchi, O. (1996). High order contrasts for self-adaptive source separation. International Journal of Adaptive Control and Signal Processing, 10, 19–46. Moulines, E., Cardoso, J.-F., & Gassiat, E. (1997). Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models. In Proc. ICASSP’97 (pp. 3617–3620). Nadal, J.-P., & Parga, N. (1994). Nonlinear neurons in the low-noise limit: A factorial code maximizes information transfer. NETWORK, 5, 565–581. Obradovic, D., & Deco, G. (1997). Unsupervised learning for blind source separation: An information-theoretic approach. In Proc. ICASSP (pp. 127–130). Pearlmutter, B. A., & Parra, L. C. (1996). A context-sensitive generalization of ICA. In International Conference on Neural Information Processing (Hong Kong). Pham, D.-T. (1996). Blind separation of instantaneous mixture of sources via an independent component analysis. IEEE Trans. on Sig. Proc., 44, 2768–2779. Pham, D.-T., & Garat, P. (1997). Blind separation of mixture of independent sources through a quasi-maximum likelihood approach. IEEE Tr. SP, 45,1712– 1725. Souloumiac, A., & Cardoso, J.-F. (1991). Comparaison de m´ethodes de s´eparation de sources. In Proc. GRETSI. Juan les Pins, France (pp. 661–664). Received February 7, 1998; accepted June 25, 1998.
LETTER
Communicated by Michael Jordan
Variational Learning in Nonlinear Gaussian Belief Networks Brendan J. Frey Beckman Institute, University of Illinois at Urbana-Champaign, Urbana, IL 61801, U.S.A.
Geoffrey E. Hinton Gatsby Computational Neuroscience Unit, University College London, London, England WC1N 3AR, U.K.
We view perceptual tasks such as vision and speech recognition as inference problems where the goal is to estimate the posterior distribution over latent variables (e.g., depth in stereo vision) given the sensory input. The recent flurry of research in independent component analysis exemplifies the importance of inferring the continuous-valued latent variables of input data. The latent variables found by this method are linearly related to the input, but perception requires nonlinear inferences such as classification and depth estimation. In this article, we present a unifying framework for stochastic neural networks with nonlinear latent variables. Nonlinear units are obtained by passing the outputs of linear gaussian units through various nonlinearities. We present a general variational method that maximizes a lower bound on the likelihood of a training set and give results on two visual feature extraction problems. We also show how the variational method can be used for pattern classification and compare the performance of these nonlinear networks with other methods on the problem of handwritten digit recognition. 1 Introduction There have been many proposals for unsupervised, multilayer neural networks that contain a stochastic generative model and learn by adjusting their parameters to maximize the likelihood of generating the observed data. Two of the most tractable models of this kind are factor analysis (Everitt, 1984) and independent component analysis (Comon, Jutten, & Herault, 1991; Bell & Sejnowski, 1995; Amari, Cichocki, & Yang, 1996; MacKay, 1997). 1.1 Linear Generative Models. In factor analysis there is one hidden layer that contains fewer units than the visible layer. In the generative model, the hidden units are driven by zero-mean, unit-variance, independent gaussian noise. The hidden units provide top-down input to the linear visible units via the generative weights and each visible unit has its own level of Neural Computation 11, 193–213 (1999)
c 1999 Massachusetts Institute of Technology °
194
Brendan J. Frey and Geoffrey E. Hinton
added gaussian noise. Given the generative weights and the noise levels of the visible units, it is tractable to compute the posterior distribution of the hidden activities induced by an observed vector of visible activities. This posterior distribution is a full covariance gaussian whose mean depends on the visible activities. Once this distribution has been computed, it is straightforward to adjust the generative weights to maximize the likelihood of the observed data using either a gradient method or the expectationmaximization (EM) algorithm (Rubin & Thayer, 1982). Unfortunately, factor analysis ignores all the statistical structure in the data that is not contained in the covariance matrix and its hidden representations are linearly related to the data, so it is unable to extract many of the hidden causes of the data that are important in tasks such as vision and speech recognition. In independent component analysis the generative model is still linear, but the independent noise levels for the hidden units are nongaussian. This makes it difficult to compute the full posterior distribution across the hidden units given a visible vector. However, by using the same number of hidden and visible units and by setting the noise levels of the visible units to zero, it is possible to collapse the posterior distribution across the hidden units to a point that is found by multiplying the visible activities by the inverse of the matrix of hidden-to-visible generative weights. To maximize the likelihood of the data, the weights are adjusted to make the posterior points have high log probability under the noise models of the hidden units, while keeping the determinant of the generative weight matrix small so that probability density in the space of hidden activities gets concentrated when it is mapped into the visible space. Unfortunately, independent component analysis extracts components that are a linear function of the data and it assumes the data are noise-free, so it too is unable to extract hidden causes that are nonlinearly related to observed, noisy data. Recently, attempts have been made to enhance the representational capabilities of independent component analysis by adding noise to the visible units (Olshausen & Field, 1996; Lewicki & Sejnowski, 1998). 1.2 Very Nonlinear Generative Models. An appealing approach to understanding how the cortex constructs models of sensory data is to assume that it uses maximum likelihood to learn a hierarchical generative model. For tasks such as vision and speech recognition, the cortex probably requires distributed representations that are a nonlinear function of the data and allow noise at every level of the hierarchy. Attempts at developing learning algorithms capable of constructing such generative models have been less successful in practice than the simpler linear models. This is because it is hard to compute (or even to represent) the posterior probability distribution across the hidden representations when given a visible vector and a set of weights and noise variances. The unsupervised version of the Boltzmann machine (Hinton & Sejnowski, 1986) is a multilayer generative model that learns distributed rep-
Variational Learning
195
resentations that are a nonlinear function of the data. It uses symmetrically connected stochastic binary units and has a relatively simple learning rule that follows the gradient of the log-likelihood of the data under the generative model. Unfortunately, to get this gradient, it is necessary to perform Gibbs sampling in the hidden activities until they reach thermal equilibrium with a data vector clamped on the visible units. This is very time-consuming, and the problem is made even worse by the need to compute derivatives of the partition function, which requires the network to reach thermal equilibrium with the visible units unclamped. The sampling noise and the difficulty in reaching equilibrium in networks with large weights make the learning algorithm painfully slow. When binary stochastic units are connected in a directed acyclic graph, we get a “binary sigmoidal belief network” (Pearl, 1988; Neal, 1992). (Here, “acyclic” means that there are not any closed paths when following edge directions. There may be closed paths when the edge directions are ignored.) The net input to each unit is given by a weighted sum of the activities of the unit’s parents. Learning is easier in this network than in a Boltzmann machine because there is no need to compute the derivative of a partition function and the gradient of the log-likelihood does not involve a difference in sampled statistics. Most important, it is no longer necessary for the Gibbs sampling to converge to thermal equilibrium before the weights are adjusted. Using the analysis of EM provided by Neal and Hinton (1993), it can be shown that on average, the learning algorithm improves a bound on the log probability of the data even when the Gibbs sampling is too brief to get close to equilibrium (Hinton, Sallans, & Ghahramani, 1998). There have been several attempts to avoid Gibbs sampling altogether when fitting a sigmoidal belief network to data. (See Frey, 1998, for a review of these methods.) They all rely on the idea that learning can still improve a bound on the log-likelihood of the data even when the posterior distribution over hidden states is computed incorrectly. The stochastic Helmholtz machine (Hinton, Dayan, Frey, & Neal, 1995) uses a separate, stochastic recognition network to compute a quick and dirty approximation to a sample from the posterior distribution over the hidden units when given a visible vector. There is a very simple rule for learning both the generative weights and the recognition weights, but the approximation produced by the recognition network is often poor, and the method of learning the recognition weights is not guaranteed to improve it. The deterministic Helmholtz machine (Dayan, Hinton, Neal, & Zemel, 1995) makes even more restrictive assumptions than the stochastic version about the probability distribution that is used to approximate the full posterior distribution over the binary hidden states when given a data vector. It assumes that the approximating distribution can be written as a product of separate probabilities for each hidden unit. It also assumes that the approximating product distribution can be computed by a deterministic recognition network in a single bottom-up pass. This latter assumption is relaxed in variational approaches
196
Brendan J. Frey and Geoffrey E. Hinton
(Saul, Jaakkola, & Jordan, 1996; Jaakkola, Saul, & Jordan, 1996), which eliminate the separate recognition model and use the generative weights and numerical optimization to find the set of probabilities that minimizes the asymmetric divergence from the true posterior distribution. 1.3 Continuous Sigmoidal Belief Networks. For real-valued data that come from real physical processes, binary units are often an inappropriate model because they fail to capture the approximately linear structure of the data over small ranges. For example, very small changes in the position, orientation, or scale of an object lead to linear changes in the pixel intensities. One way to endow the linear gaussian networks described above with representations that are nonlinear functions of the data is to apply a smooth sigmoidal squashing function to the output of each gaussian before passing the activity down the network. The nonlinear squashing function allows each unit to take on a variety of behaviors, ranging from nearly gaussian to nearly binary. Frey (1997a, b) showed that a Markov chain Monte Carlo and a variational method could be used to train small networks of these units. However, the smoothness of the squashing function prevents units from placing probability mass on a single point, and so these units are unable to produce activities exactly equal to zero. The ability of a network to set activities exactly equal to zero is important for sparse representations where many units do not participate in explaining an input pattern. 1.4 Piecewise Linear Belief Networks. In an attempt to produce sparse distributed representations of real-valued data, Hinton and Ghahramani (1997) investigated generative models composed of multiple layers of rectified linear units. In the generative model, each unit receives top-down input that is a linear function of the rectified states in the layer above and it adds gaussian noise to get its own real-valued unrectified state, ½ X 0 if xj < 0, (1.1) wij f (xj ) + noise, where f (xj ) = xi = wi0 + xj if xj ≥ 0, j∈Ai
and Ai is the set of indices for the parents of unit i. The output that a unit sends to the layer below is equal to its unrectified state if it is positive but is equal to 0 if it is negative. Networks of these units can set the activities of some units exactly equal to zero so that they do not participate in explaining the current input pattern. Hinton and Ghahramani (1997) showed that Gibbs sampling was feasible in such networks and that multilayer networks of rectified linear units could learn to extract sparse hidden representations that were nonlinearly related to images. 1.5 Nonlinear Gaussian Belief Networks. Linear generative models, binary sigmoidal belief networks, continuous sigmoidal belief networks,
Variational Learning
197
and piecewise linear belief networks can all be viewed as networks of gaussian units that apply various nonlinearities to their gaussian states. The probability density function over the pre-nonlinearity variables x = (x1 , . . . , xN ) in such a nonlinear gaussian belief network (NLGBN) is à ! P N N Y Y xi − j∈Ai wij fj (xj ) 1 p(xi |{xj }j∈Ai ) = φ , (1.2) p(x) = ψ ψi i=1 i=1 i where Ai is the set of indices for the parents of unit i and φ(·) is the standard normal density function: 1 2 φ(y) = √ e−y /2 . 2π
(1.3)
ψi2 is the variance of the gaussian noise for unit i and fj (·) is the nonlinear function for unit j. For example, some units may use a step function (making them binary sigmoidal units with a cumulative gaussian activation function), whereas other units may use the rectification function (making them real-valued units that encourage sparse representations). We define f0 (x0 ) = 1 so that wi0 represents a constant bias for unit i in equation 1.2. In this article, we generalize the variational method developed by Jaakkola et al. (1996) for networks of binary units and show that it can be successfully applied to performing approximate inference and learning in nonlinear gaussian belief networks. The variational method can still be applied when different types of nonlinearity are used in the same network, such as networks of the kind described in Hinton, Sallans, & Ghahramani (1998), where binary and linear units come in pairs and the output of each linear unit is gated by its associated binary unit. 2 Variational Expectation Maximization A surprisingly simple variational technique can be used for inference and learning in NLGBNs. In this method, once some variables have been observed, we postulate a simple parametric variational distribution q(·) over the remaining unobserved variables. (The variational distribution q(·) is separate from the generative distribution p(·).) A numerical optimization method (e.g., conjugate gradients) is then used to adjust the variational parameters to bring q(·) as close to the true posterior as possible. We use a function that not only measures closeness in the Kullback-Leibler sense, but also bounds from below the log-likelihood of the input pattern. This choice of function leads to an efficient generalized EM learning algorithm. 2.1 A Function That Bounds the Log-Probability. Let V be the set of indices of the observed variables for the current input pattern and let H be the set of indices of the unobserved variables for the current input pattern,
198
Brendan J. Frey and Geoffrey E. Hinton
so that V ∪ H = {1, . . . , N}. The variational bound (Neal & Hinton, 1993) is F = hlog p(x)i − hlog q({xi }i∈H )i ≤ log p({xi }i∈V ),
(2.1)
where h·i indicates an expectation over the unobserved variables with respect to q(·). It is easily shown that for unconstrained q(·), F is maximized by setting q({xi }i∈H ) = p({xi }i∈H |{xi }i∈V ), in which case the bound in equation 2.1 is tight. This gives exact probabilistic inference, whereas using a constrained form for q(·) gives approximate probabilistic inference. The variational distribution we consider here is a product of gaussians: Y Y 1 µ xi − µi ¶ q(xi ) = φ , (2.2) q({xi }i∈H ) = σ σi i∈H i∈H i where µi and σi , i ∈ H are the variational parameters. By adjusting these parameters, we can obtain an axis-aligned gaussian approximation to the true posterior distribution over the prenonlinearity hidden variables. For this variational distribution, it turns out that F can be expressed in terms of the mean and variance of the postnonlinearity activities. Let Mi (µ, σ ) be the mean output of unit i when the input is gaussian noise with mean µ and variance σ 2 : µ ¶ Z x−µ 1 φ fi (x)dx. (2.3) Mi (µ, σ ) = σ x σ Let Vi (µ, σ ) be the variance at the output of unit i when the input is gaussian noise with mean µ and variance σ 2 : µ ¶ Z ª2 x−µ © 1 φ fi (x) − Mi (µ, σ ) dx. (2.4) Vi (µ, σ ) = σ σ x We assume that these can be easily computed, closely approximated, or in the case of Vi (·, ·), bounded from above (the latter will give a new lower bound on F). (See appendix C for these functions in the case of linear units, binary units, rectified units, and sigmoidal units.) The variational bound, equation 2.1, simplifies to1 ¾ ½h N i2 P X P 1 2 − w M (µ , σ ) + w V (µ , σ ) µ i ij j j j j j j j∈Ai j∈Ai ij 2ψi2 i=1 ¶ X N X1µ σ2 1 1 + log 2π σi2 − i2 − log 2π ψi2 . (2.5) + 2 ψ 2 i i∈H i=1
F=−
P
To see how hlog p(x)i simplifies, add and subtract both µi and w M (µ , σ ) j∈Ai ij j j j in P the numerator ofPthe argument of φ(·) in equation 1.2, so that hlog p(x)i = P P − i log(2πψi2 )/2 − h([xi − µi ] + [µi − j∈Ai wij Mj (µj , σj )] + j∈Ai wij [Mj (µj , σj ) − i 1
fj (xj )])2 i/2ψi2 . The cross-terms produced by the square vanish under the expectation.
Variational Learning
199
To make this formula concise, we have introduced dummy variational parameters for the observed variables: if xi is observed to have the value x∗i , we fix µi = x∗i and σi = 0. For unit i in equation 2.5, the term in curly braces measures the mean squared error under P q(·) between µi and the input to unit i as given by its parents: h[µi − j∈Ai wij fj (xj )]2 i. It is down-weighted by the model noise variance ψi2 , since for larger noise variances, a particular mean squared prediction error is less important. 2.2 Probabilistic Inference. Variational inference consists of first fixing µi = x∗i and σi = 0, i ∈ V in equation 2.5 and then maximizing F with respect to µi and log σi2 , i ∈ H. (The optimization for the variances is performed in the log-domain, since log σi2 is allowed to go negative.) We use the conjugate gradient method to perform this optimization, although other techniques can be used (e.g., steepest descent or possibly a covariant method; Amari, 1985). The derivatives of F with respect to µi and log σi2 , i ∈ H are given in appendix A. After optimization, the means and variances of the variational distribution represent the inference statistics. 2.3 Learning. We bound the log-probability of an entire training set by F , which is equal to the sum of the bounds for the individual training patterns. The variational EM algorithm based on F consists of iterating the following two steps: • E-step: Perform variational inference by maximizing F with respect to the sets of variational parameters corresponding to the different input patterns. • M-step: Maximize F with respect to the model parameters (w·· ’s and ψ· ’s). Notice that by maintaining sufficient statistics while scanning through the training set in the E-step, it is not necessary to store the sets of variational parameters. These sufficient statistics are described in appendix B. However, to speed up the current E-step, we initialize the set of variational parameters to the set found at the end of the last E-step for the same pattern. It turns out that the M-step can be performed very efficiently (see appendix B for details). Since the values of the model variances do not affect the values of the weights that maximize F in equation 2.5, we first maximize F with respect to the weights. As pointed out by Jaakkola et al. (1996) for their binary gaussian belief networks, F is quadratic in the weights, so we can use singular value decomposition to solve for the weights exactly. Next, the optimal model variances are computed directly. 2.4 Software. A set of UNIX programs that implement variational learning in NLGBNs is available at http://www.cs.utoronto.ca/∼frey. The soft-
200
Brendan J. Frey and Geoffrey E. Hinton
ware includes linear units, binary units, rectified units, and sigmoidal units. New types of unit can be added easily by providing the nonlinear function and its derivatives. 3 Visual Feature Extraction Approximate maximum likelihood estimation in latent variable models can be used to learn latent structure that is perceptually significant (Hinton et al., 1995). In this section, we consider two unsupervised feature extraction tasks and for each task we compare the representations learned by the variational method applied to two types of NLGBN and the representations learned by Gibbs sampling applied to a piecewise linear NLGBN. If the hidden units all use a piecewise linear activation function, then Gibbs sampling can be efficiently used for learning, as described in Hinton and Ghahramani (1997). One of the NLGBNs used for variational learning contains only binary hidden units of the type described in Jaakkola et al. (1996). In this section we see how the variational technique compares to another learning method for continuous hidden units, as well as how the generalization of the variational method from binary to continuous units compares to variational learning in binary networks. 3.1 The Continuous Bars Problem. An important problem in vision is modeling surface edges in a way that is consistent with physical constraints. The goal of the much simpler bars problem (Dayan & Zemel, 1995) is to learn without supervision to detect bars of two orthogonal orientations and to model the constraint that each image consists of bars of the same orientation. In Hinton and Ghahramani (1997), a continuous form of this problem was presented. Each training image is formed by first choosing between vertical and horizontal orientation with equal probability. Then each bar of that orientation is turned on with a probability of 0.3 with an intensity that is drawn uniformly from [0, 5]. Eight examples from a training set of 1000 6 × 6 images of this sort are shown in Figure 1a, where the area of each tiny white square indicates the pixel intensity. A noisy version of these data in which unit variance gaussian noise is added to each pixel is shown in Figure 1e (a black square indicates a negative pixel value). For each of the two data sets, we used 100 iterations of variational EM to train a three-layer NLGBN with 1 binary top-layer unit, 16 rectified middlelayer units, and 36 linear visible bottom-layer units. (Using more units in the hidden layers had little effect on the features extracted during learning.) The resulting weights projecting from each of the 16 middle-layer units to the 6×6 image are shown in Figures 1b and 1f. Surprisingly, clearer bar features were extracted from the noisy data. The weights (not shown) from the toplayer binary unit to the middle-layer units tend to make the top-layer unit active for one orientation and inactive for the other. The weights look similar if a rectified unit is used at the top, but a binary unit properly represents
Variational Learning
201
(a)
(e)
(b)
(f)
(c)
(g)
(d)
(h)
Figure 1: Learning in NLGBNs using the variational method and Gibbs sampling, for noise-free (a–d) and noisy (e–h) bar patterns. (a, e) Training examples. (b, f) Weights learned by the variational method with rectified units. (c, g) Weights learned by the variational method with binary units. (d, h) Weights learned by Gibbs sampling with rectified units.
202
Brendan J. Frey and Geoffrey E. Hinton
the discrete choice between horizontal and vertical. Figures 1c and 1g show the weights learned by variational EM in a network where all of the hidden units are binary. The individual bars are not properly extracted for either the noise-free or noisy training data. The log-probability bounds for the trained binary-rectified-linear NLGBNs are 27.4 nats and −60.3 nats, whereas the bounds for the trained binary-binary-linear NLGBNs are −48.3 nats and −65.6 nats, significantly lower. We also trained a three-layer NLGBN with rectified hidden units using Gibbs sampling. For this method, a learning rate must be chosen, and we used 0.1. Sixteen sweeps of Gibbs sampling were performed for each pattern presentation before the parameters were adjusted online. The weights obtained after 10 passes through each data set are shown in Figures 1d and 1h. The features for the noise-free data are not as clear as the ones extracted using the variational method. The features for the noisy data can be cleaned up if some weight decay is used (Hinton & Ghahramani, 1997). We did not estimate the log-probability of the data in this case, since Gibbs sampling does not readily provide a straightforward way to obtain such estimates. In our experiments, variational EM and the Gibbs sampling method took roughly the same time. However, an online version of variational EM may be faster. 3.2 The Continuous Stereo Disparity Problem. Another vision problem where the latent variables are nonlinearly related to the input is the estimation of depth from a stereo pair of sensory images. In the simplified version of this problem presented in Becker and Hinton (1992), the goal is to learn that the visual input consists of randomly positioned dots on a one-dimensional surface placed at one of two depths. In our experiments, four blurred dots (gaussian functions) were randomly positioned uniformly on the continuous interval [0, 12] and the brightness of each dot (magnitude of the gaussian) was drawn uniformly from [0, 5]. Next, a left or a right shift was applied with equal probability to obtain a second activity pattern. Finally, two sensory images containing 12 real values each were obtained by dividing each interval into 12 pixels and assigning to each pixel the net activity within the pixel. Twelve examples from a training set of 1000 pairs of images obtained in this manner are shown in Figure 2a, where the images are positioned so that the relative shift is evident. A noisy version of these data in which unit variance gaussian noise is added to each sensor is shown in Figure 2e. The stereo disparity problem is much more difficult than the bars problem, since there is more overlap between the underlying features. To see this, imagine a “multieyed” disparity problem in which there are as many sensory images as there are one-dimensional sensors. We expect the depth inference to be easier in this case, since there is more evidence for each of the two possible directions of shift. Imagine stacking the sensory images on top of each other, so that each resulting square image will contain blurred
Variational Learning
203
(a)
(e)
(b)
(f)
(c)
(g)
(d)
(h)
Figure 2: Learning in NLGBNs using the variational method and Gibbs sampling, for noise-free (a–d) and noisy (e–h) stereo disparity patterns. (a, e) Training examples. (b, f) Weights learned by the variational method with rectified units. (c, g) Weights learned by the variational method with binary units. (d, h) Weights learned by Gibbs sampling with rectified units.
204
Brendan J. Frey and Geoffrey E. Hinton
diagonal bars that are oriented either up and to the right or up and to the left. Extracting disparity from these data is roughly equivalent to extracting bar orientation in the data from the previous section. For each of the noisy and noise-free data sets, we used 100 iterations of variational EM to train a three-layer NLGBN with 1 binary top-layer unit, 20 rectified middle-layer units, and 24 linear visible bottom-layer units. The resulting weights projecting from each of the 20 middle-layer units to the two sets of 12 pixels are shown in Figures 2b and 2f. In both cases, the algorithm has extracted features that are spatially local and represent each of the two possible depths. Figures 2c and 2g show the weights learned by variational EM in a network where all of the hidden units are binary. The log-probability bounds for the trained binary-rectified-linear NLGBNs are 1.0 nats and −43.7 nats, whereas the bounds for the trained binarybinary-linear NLGBNs are −37.8 nats and −46.1 nats. The Gibbs sampling method with rectified hidden units and using the same learning parameters as described in the previous section produced the weight patterns shown in Figures 2d and 2h. For the noisy data, the features extracted by Gibbs sampling appear to be slightly cleaner than those extracted by variational EM. 4 Handwriting Recognition Variational inference and learning in NLGBNs can be used to do real-valued pattern classification by training one NLGBN on each class of data. If Ci is the event that a pattern comes from class i ∈ {0, 1, . . .}, then the posterior class probabilities are given by Bayes’ rule p({xk }k∈V |Ci )P(Ci ) , P(Ci |{xk }k∈V ) = P j p({xk }k∈V |Cj )P(Cj )
(4.1)
where P(Cj ) is the prior probability that a pattern comes from class j. The likelihood p({xk }k∈V |Cj ) for class j is approximated by the value of the variational bound obtained from a generalized E-step. In this section, we report the performances of several completely automated learning procedures on the problem of recognizing gray-level 8 × 8 images of handwritten digits from the CEDAR CDROM 1 database of zip codes (Hull, 1994). The DELVE evaluation platform (Rasmussen et al., 1996) was used to obtain fair comparisons, including levels of statistical significance. For each of three different sizes of training set, we empirically estimated the performances of k-nearest neighbors, with the neighborhood for each class of data determined using leave-one-out cross validation; mixture of diagonal (axis-aligned) gaussians, with the number of gaussians for each class determined using a validation set; factor analysis, with the number of factors for each class determined using a validation set; and one- and
Variational Learning
205
two-hidden layer NLGBNs using rectified hidden units and linear visible units, with the number of hidden units determined using a validation set. For each of the latter four methods, the training set was first split according to class, and then one-third of the data was set aside for validation. Models with different complexities (number of gaussians or number of hidden units) were trained on the remaining two-thirds of the data using EM or generalized EM until convergence. The model that gave the highest validation set log-probability (or log-probability bound) was further trained to convergence on all of the data for the corresponding class. To prevent degenerate overfitting of a pixel that happens to have the same value in all of the training cases, the variances for the visible units were not allowed to fall below 0.01 in the latter four methods. For each method, equation 4.1 was used to classify each test pattern. To obtain robust estimates of the relative performances of these methods on the problem of handwritten digit recognition, we trained and tested each method multiple times using disjoint training set–test set pairs. The original data set of 11,000 8 × 8 images was partitioned into a set of 8000 images used for training and a set of 3000 images used for testing. For training set sizes of 1000 and 2000 patterns, four disjoint training set–test set pairs were extracted from the two partitions (each test set had 750 images). For the training set size of 4000 patterns, two disjoint training set–test set pairs were extracted (each test set had 1500 images). The results are shown in Figure 3, where each horizontal bar gives an estimate of the expected error rate for a particular method using a particular training set size. The methods are ordered from left to right for each training set size as follows: k-nearest neighbors, mixture of gaussians, factor analysis, one-hidden layer NLGBN, and two-hidden layer NLGBN. Each vertical bar gives an estimate of the error (one standard deviation) in the corresponding estimate of the expected error rate. Integers in the boxes lying beneath the x-axis are p-values (in percent) for a paired t-test that compares the performances of the corresponding methods. Select a method from the list in the lower left-hand corner of the figure, and scan from left to right. Whenever you see a number, that means another method has performed better than the method you selected, with the given statistical significance. A low p-value indicates the difference in the misclassification rates is significant. More precisely, the p-value is an estimate of the probability of obtaining a difference in performance that is equal to or greater than the observed difference, given that the true difference is zero (the null hypothesis). On the largest training set size, the two-hidden layer NLGBN performs better than k-nearest neighbors (p = 5%), mixture of gaussians (p = 9%), factor analysis (p = 5%), and the one-hidden layer NLGBN (p = 16%). The performance of the NLGBN with one hidden layer is fairly indistinguishable from the performance of factor analysis (p = 25%). However, on average the NLGBN used only half as many hidden units as were used by factor analysis, indicating that the NLGBN provides a more compact
206
Brendan J. Frey and Geoffrey E. Hinton
Figure 3: Estimated error rates on gray-level handwritten digit recognition using different sizes of training set (1000, 2000, and 4000 images) for the following methods: k-nearest neighbor, mixture of diagonal (axis-aligned) gaussians, factor analysis, and rectified gaussian belief networks with one and two hidden layers. For each size of training set, the error rates for the different methods are given in the above order. The numbers in the boxes are p-values (in percent) for a paired t-test on the null hypothesis that the corresponding two methods have identical performance. A dot indicates the p-value was above 9%.
representation of the input. It is also interesting that this more compact representation emerged despite the fact that the posterior distribution over the hidden units in the NLGBN was approximated by an axis-aligned gaussian distribution, whereas in factor analysis, the exact full-covariance posterior distribution over the hidden units is used. 5 Conclusions Results on visual feature extraction show that the variational technique can extract perceptually significant continuous nonlinear latent structure. In contrast with networks with continuous hidden variables, networks with binary hidden variables do not extract spatially local features from the data. Similarly, linear methods like factor analysis and independent component
Variational Learning
207
analysis fail to extract spatially local features (Zoubin Ghahramani, personal communication). Results also show that the variational method presented in this article is a viable alternative to Gibbs sampling in stochastic neural networks with rectified hidden variables. Advantages of the variational method over Gibbs sampling include the absence of a learning rate and the ability to compute the log-probability bound very efficiently. The latter is particularly useful for pattern classifiers that train one network on each class of data and then classify a novel pattern by picking the network that gives the highest estimate of the log-probability (Frey, 1998). Results show that for handwritten digit recognition, there is a regime of training set size in which NLGBNs perform better than k-nearest neighbors, mixture of gaussians, and the linear factor analysis method. The variational method may be made more powerful by making each distribution q(xi ) in equation 2.1 a mixture of gaussians, by making the entire distribution q(·) a mixture of product form distributions (Jaakkola & Jordan, 1998), or by grouping together small numbers of hidden variables over which full-covariance gaussians are fit during variational inference. We considered three types of continuous nonlinear unit in this article: binary, rectified, and sigmoidal. The variational method can be easily extended to other types of units (such as “twinned” units; Hinton et al., 1998) as long as the output mean function M(µ, σ ) and the output variance function V(µ, σ ) can be computed. To perform a gradient-based E-step, the gradients of these functions with respect to their arguments are also needed. Although we have not focused our attention on implementations of the variational algorithm that are suited to biology, we believe that with some modifications, they can be made so. The inference algorithm uses the logprobability bound derivatives given in equation A.3, and these are computed from simple differences passed locally in the network. A partial Mstep can be used for online learning, in which case the derivative of the bound for just the current input pattern is followed. This derivative can be followed by applying a delta-type rule based on locally computed differences. Appendix A: The E-Step Here we show how to compute F and its derivatives with respect to the variational parameters. For the current set of variational parameters (including the ones fixed by the current input pattern), we first compute for each unit the current values of the mean output mi ← Mi (µi , σi ), the output variance vi ← Vi (µi , σi ), and the mean net input:
ni ←
X j∈Ai
wij mj .
(A.1)
208
Brendan J. Frey and Geoffrey E. Hinton
Then the bound on the log-probability of the input pattern is computed from N i X P 1 h 2 2 − n ) + w v (µ i i j j∈A ij i 2ψi2 i=1 ¶ X N X1µ σ2 1 1 + log 2πσi2 − i2 − log 2π ψi2 . + 2 ψ 2 i i∈H i=1
F←−
(A.2)
To perform a gradient-based optimization in the E-step, the derivatives of the bound with respect to µj and log σj2 for j ∈ H are computed as follows: nj − µj ∂Mj (µj , σj ) X wij (ni − µi ) ∂F ← − 2 ∂µj ∂µj ψi2 ψj i∈Cj −
2 ∂Vj (µj , σj ) X wij , ∂µj 2ψi2 i∈C j
2 ∂Vj (µj , σj ) X wij ∂Mj (µj , σj ) X wij (ni − µi ) ∂F ← − − 2 2 2 ψi2 ∂ log σj ∂ log σj i∈Cj 2ψi ∂ log σj2 i∈Cj
−
σj2 2ψj2
1 + , 2
(A.3)
where Cj is the set of indices for the children of unit j. Appendix C gives expressions for the derivatives of the nonlinear functions for binary units, rectified units, and sigmoidal units. (t) An E-step produces one set of variational parameters µ(t) i , σi , i = 1, . . . , (t) (t) N and the corresponding m(t) i , vi , and ni , i = 1, . . . , N for each training pattern, t = 1, . . . , T. These are used to initialize the next E-step. Appendix B: The M-Step In the log-probability bound in equation 2.5, the model variances do not influence the optimal weights. So in the M-step, we first maximize the total bound with respect to the weights. Since the bound is quadratic in the weights, we use singular value decomposition to solve for them exactly. In fact, the weights associated with the input to each variable are decoupled from the other weights in the network. That is, the value of wij does not affect the optimal value of wkl if i 6= k. Consequently, solving for the optimal weights is a matter of solving N linear systems, where system i, i = 1, . . . , N, has dimensionality equal to the number of parents for unit i and once solved gives the weights on the incoming connections to unit i.
Variational Learning
209
(t) (t) Consider the input means µ(t) i , output means mi , input variances σi , (t) (t) output variances vi , and mean net inputs ni , i = 1, . . . , N, which are computed for training pattern t in the E-step. It is not necessary to store all of these sets for all T training patterns. However, they are used to compute the following sufficient statistics:
ajk ←
1 X (t) (t) 1 X (t) 1 X (t) (t) mj mk , bj ← vj , cij ← µ m , T t T t T t i j 1 X (t) 1 X (t) 2 (nj − µj(t) )2 , ej ← σ , dj ← T t T t j
(B.1)
for j = 0, . . . , N, k = 0, . . . , N, and i ∈ cj . These can be accumulated while scanning through the training set during the E-step. Once the sufficient statistics have been computed, we first solve for the weights. The system of equations for the weights associated with the input to unit i is X
ajk wik + bj wij = cij ,
j ∈ Ai ,
(B.2)
k∈Ai
where i is fixed in this set of equations. We use singular value decomposition to solve for each set of weights. In fact, the system of equations for unit i has dimensionality equal to the number of parents for unit i (including its bias). Finally, the model variances are computed from ψj2 ← dj + ej +
X
2 wjk bk .
(B.3)
k∈Aj
Appendix C: M(µ, σ ), V(µ, σ ) and Their Derivatives for Interesting Nonlinear Functions In this appendix, we give the output means and variances for some useful types of units: linear units, binary units, rectified units, and sigmoidal units. C.1 Linear Units. Although this article is about how to deal with nonlinear units, it is often useful to include some units (e.g., visible units) that are linear: f (x) = x.
(C.1)
For this unit, the output mean and variance are M(µ, σ ) = µ,
V(µ, σ ) = σ 2 .
(C.2)
210
Brendan J. Frey and Geoffrey E. Hinton
The derivatives are ∂M(µ, σ ) = 0, ∂ log σ 2 ∂V(µ, σ ) = σ 2. ∂ log σ 2
∂M(µ, σ ) = 1, ∂µ ∂V(µ, σ ) = 0, ∂µ
(C.3)
C.2 Binary Units. To obtain a stochastic binary unit, take ½
if x < 0, if x ≥ 0.
0 1
f (x) =
(C.4)
For this unit, the output mean and variance are M(µ, σ ) = 8
³µ´ σ
,
V(µ, σ ) = 8
³µ´ h ³ µ ´i 1−8 , σ σ
(C.5)
where 8(·) is the cumulative gaussian function: Z 8(y) =
y α=−∞
φ(α)dα.
(C.6)
The derivatives are 1 ³µ´ ∂M(µ, σ ) µ ³µ´ ∂M(µ, σ ) = φ , φ , = − ∂µ σ σ ∂ log σ 2 2σ σ ³ µ ´i 1 ³µ´ h ∂V(µ, σ ) = φ 1 − 28 , ∂µ σ σ σ ³ µ ´i µ ³µ´ h ∂V(µ, σ ) φ 1 − 28 . = − ∂ log σ 2 2σ σ σ
(C.7)
C.3 Rectified Units. A rectified unit is linear if its input exceeds 0 and outputs 0 otherwise: ½ f (x) =
0 x
if x < 0, if x ≥ 0.
(C.8)
For this unit, the output mean and variance are ³µ´ + σφ , σ ³µ´ ³ µ ´σ + µσ φ − M(µ, σ )2 . V(µ, σ ) = (µ2 + σ 2 )8 σ σ M(µ, σ ) = µ8
³µ´
(C.9)
Variational Learning
211
The derivatives are ³µ´ ∂M(µ, σ ) σ ³µ´ ∂M(µ, σ ) =8 , φ , = ∂µ σ ∂ log σ 2 2 σ ³µ´ ³µ´ ³µ´ ∂V(µ, σ ) = 2µ8 + 2σ φ − 2M(µ, σ )8 , ∂µ σ σ σ ´ ³ ´ ³ µ µ ∂V(µ, σ ) − σ M(µ, σ )8 . = σ 28 2 ∂ log σ σ σ
(C.10)
C.4 Sigmoidal Units. The cumulative gaussian squashing function, f (x) = 8(x),
(C.11)
leads to closed-form expressions for the output mean and its derivatives. We have not found a closed-form expression for the output variance, but it can be approximately bounded by a new function V 0 (µ, σ ), giving a new lower bound on F. The output mean and variance bound are ¶ µ µ , M(µ, σ ) = 8 √ 1 + σ2 µ ¶· µ ¶¸ µ µ 1−8 √ V(µ, σ ) ≤ V 0 (µ, σ ) = 8 √ 1 + σ2 1 + σ2 2 σ . (C.12) × 2 σ + π/2 The derivatives are
µ ¶ 1 µ ∂M(µ, σ ) = √ φ √ , ∂µ 1 + σ2 1 + σ2 ¶ µ µσ 2 µ ∂M(µ, σ ) , = − φ √ ∂ log σ 2 2(1 + σ 2 )3/2 1 + σ2 ¶· µ ¶¸ µ σ2 µ µ ∂V 0 (µ, σ ) = 1 − 28 √ , φ √ √ ∂µ (σ 2 + π/2) 1 + σ 2 1 + σ2 1 + σ2 µ ¶· µ ¶¸ π/2 µ µ 8 √ 1−8 √ σ 2 + π/2 1 + σ2 1 + σ2 ¶· µ ¶¸¾ µ 2 µ µ µσ 1 − 28 √ . (C.13) φ √ − 2 2(1 + σ 2 )3/2 1+σ 1 + σ2
σ2 ∂V 0 (µ, σ ) = ∂ log σ 2 σ 2 + π/2
½
Acknowledgments We thank Peter Dayan, Zoubin Ghahramani, and Tommi Jaakkola for helpful discussions. We also appreciate the useful feedback provided by Michael
212
Brendan J. Frey and Geoffrey E. Hinton
Jordan, an anonymous reviewer, and Karla Miller. The Gibbs sampling experiments were performed using software developed by Zoubin Ghahramani (see http://www.cs.utoronto.ca/∼zoubin). This research was funded by grants from the Arnold and Mabel Beckman Foundation, the Natural Science and Engineering Research Council of Canada, and the Information Technology Research Center of Ontario. References Amari, S.-I. (1985). Differential-geometrical methods in statistics. New York: Springer-Verlag. Amari, S.-I., Cichocki, A., & Yang, H. (1996). A new learning algorithm for blind signal separation. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8. Cambridge, MA: MIT Press. Becker, S., & Hinton, G. E. (1992). A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355, 161–163. Bell, A. J., & Sejnowski, T. J. (1995). An information maximization approach to blind separation and blind deconvolution. Neural Computation, 7, 1129–1159. Comon, P., Jutten, C., & Herault, J. (1991). Blind separation of sources. Signal Processing, 24, 11–20. Dayan, P., Hinton, G. E., Neal, R. M., & Zemel, R. S. (1995). The Helmholtz machine. Neural Computation, 7, 889–904. Dayan, P., & Zemel, R. S. (1995). Competition and multiple cause models. Neural Computation, 7, 565–579. Everitt, B. S. (1984). An introduction to latent variable models. New York: Chapman and Hall. Frey, B. J. (1997a). Continuous sigmoidal belief networks trained using slice sampling. In M. C. Mozer, M. I. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems, 9. Cambridge, MA: MIT Press. Available online at: http://www.cs.utoronto.ca/∼frey. Frey, B. J. (1997b). Variational inference for continuous sigmoidal Bayesian networks. In Sixth International Workshop on Artificial Intelligence and Statistics. Frey, B. J. (1998). Graphical models for machine learning and digital communication. Cambridge, MA: MIT Press. Available online at: http://mitpress.mit.edu/book-home.tcl?isbn=026206202X. Hinton, G. E., Dayan, P., Frey, B. J., & Neal, R. M. (1995). The wake-sleep algorithm for unsupervised neural networks. Science, 268, 1158–1161. Hinton, G. E., & Ghahramani, Z. (1997). Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society of London B, 352, 1177–1190. Hinton, G. E., Sallans, B., & Ghahramani, Z. (1998). A hierarchical community of experts. In M. I. Jordan (Ed.), Learning and inference in graphical models. Norwell, MA: Kluwer. Hinton, G. E., & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed
Variational Learning
213
processing: Explorations in the microstructure of cognition (vol. 1, pp. 282–317). Cambridge, MA: MIT Press. Hull, J. J. (1994). A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 550–554. Jaakkola, T. S., & Jordan, M. I. (1998). Approximating posteriors via mixture models. In M. I. Jordan (Ed.), Learning and inference in graphical models. Norwell MA: Kluwer. Jaakkola, T., Saul, L. K., & Jordan, M. I. (1996). Fast learning by bounding likelihoods in sigmoid type belief networks. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8. Cambridge, MA: MIT Press. Lewicki, M. S., & Sejnowski, T. J. (1998). Learning nonlinear overcomplete representations for efficient coding. In M. I. Jordan, M. I. Kearns, & S. A. Solla (Eds.), Advances in neural information processing systems, 10. Cambridge, MA: MIT Press. MacKay, D. J. C. (1997). Maximum likelihood and covariant algorithms for independent component analysis. Unpublished manuscript. Available online at: http://wol.ra.phy.cam.ac.uk/mackay. Neal, R. M. (1992). Connectionist learning of belief networks. Artificial Intelligence, 56, 71–113. Neal, R. M., & Hinton, G. E. (1993). A new view of the EM algorithm that justifies incremental and other variants. Unpublished manuscript. Available via FTP at: ftp://ftp.cs.utoronto.ca/pub/radford/em.ps.Z. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive-field properties by learning a sparse code for natural images. Nature, 381, 607–609. Pearl, J. (1988). Probabilistic reasoning in intelligent systems. San Mateo, CA: Morgan Kaufmann. Rasmussen, C. E., Neal, R. M., Hinton, G. E., van Camp, D., Revow, M., Ghahramani, Z., Kustra, R., & Tibshirani, R. (1996). The DELVE manual. Toronto: University of Toronto. Available online at: http://www.cs.utoronto.ca/∼delve. Rubin, D., & Thayer, D. (1982). EM algorithms for ML factor analysis. Psychometrika, 47, 69–76. Saul, L. K., Jaakkola, T., & Jordan, M. I. (1996). Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research, 4, 61–76. Received November 7, 1997; accepted May 19, 1998.
LETTER
Communicated by Michael Jordan
Propagating Distributions Up Directed Acyclic Graphs Eric B. Baum Warren D. Smith NEC Research Institute, Princeton, NJ 08540, U.S.A.
In a previous article, we considered game trees as graphical models. Adopting an evaluation function that returned a probability distribution over values likely to be taken at a given position, we described how to build a model of uncertainty and use it for utility-directed growth of the search tree and for deciding on a move after search was completed. In some games, such as chess and Othello, the same position can occur more than once, collapsing the game tree to a directed acyclic graph (DAG). This induces correlations among the distributions at sibling nodes. This article discusses some issues that arise in extending our algorithms to a DAG. We give a simply described algorithm for correctly propagating distributions up a game DAG, taking account of dependencies induced by the DAG structure. This algorithm is exponential time in the worst case. We prove that it is #P complete to propagate distributions up a game DAG correctly. We suggest how our exact propagation algorithm can yield a fast but inexact heuristic.
1 Introduction Recently there has been considerable interest in using of directed graphical models for inference and modeling in problems involving uncertainty (Jensen, 1996). In playing a game, one typically searches a subtree of the game tree in order to reduce one’s uncertainty about which move to make. We have recently explored the use of a probabilistic model in this procedure (Baum & Smith, 1997). Instead of using an evaluation function that returns a scalar value as in standard game programs, we used an evaluation function that returns a probability distribution over the possible values of a position. Assuming independence of the distributions at the leaves of the search subtree, we built a model of our uncertainty. We described how to use this model for utility-directed growth of the search tree and for the choice of move after the tree is grown. Our algorithm is an example of the use of a directed graphical model, but is simpler in at least two respects than the general case. First, the graphs we explored had no loops, and second, in a general graphical model, the nodes take values from a distribution that could depend in an arbitrary way on Neural Computation 11, 215–227 (1999)
c 1999 Massachusetts Institute of Technology °
216
Eric B. Baum and Warren D. Smith
the distribution at connected nodes. In game trees, there is a natural notion of causality: the leaves have values (or probability distributions of values), and the distributions of values taken by child nodes determine the distributions of their parents through “negamax” (or equivalently “min-max”). Because of these simplifications, we were able to describe near-linear-time algorithms. In this article, we discuss relaxing the first of these simplifications, and thus the extension of our methods to more general directed acyclic graphs (DAGs). In games such as chess and Othello, the same position can occur more than once in a game tree, which thus collapses to a DAG. Competitive programs for such games generally use a hash table to spot recurrences of previously evaluated positions efficiently, and then one need neither valuate nor store that node twice. This idea is equally valid in our formalism.1 The new feature for our methods when we allow DAGs comes from the correlation between distributions at different nodes. In a DAG there may be nodes with common ancestors. In our previous work, we assumed that the distributions at leaves were independent, and this implied that the distributions at all sibling nodes were independent. This article assumes that the distributions at the sources of the DAG are independent and then gives an algorithm that propagates distributions up a game DAG taking correct account of all the dependencies then induced by the DAG structure. Although conceptually simple, this algorithm is, unfortunately, exponential time in the worst case. We also show that it is #P complete to propagate distributions correctly up a game DAG, so that there is no algorithm for efficiently propagating distributions up a game DAG if P 6= NP. The intractability of propagation of distributions on general Bayes’ nets was previously known (Cooper, 1990). Our result is stronger in showing that propagation is intractable even when the dependence of the value of a node on that of its neighbors is restricted to negamax. We suggest an approach by which our exact (but slow in the worst case) propagation algorithm can yield fast but inexact heuristics. Section 2 reviews the handling of distributions on search trees. Section 3 gives our new results about DAGS. Section 4 suggests a plausible approach to acceptably fast but inexact propagation of distributions on game DAGs and discusses how standard distribution propagation algorithms would fare in the game application.
1 Some subtleties in the use of hash tables are mentioned in Baum & Smith (1997). Note in particular that our algorithm iteratively expands the most utilitarian leaves. The utility of expanding a leaf depends on how knowledge gained from appending successor positions to the search tree may affect move choice and later expansion decisions. In a DAG, the “influence function” at a node is the sum of the influence functions at the tree nodes it represents, so that one accounts for the utilities arising from different paths to the root (Baum & Smith, 1997).
Propagating Distributions Up Directed Acyclic Graphs
217
Figure 1: A search tree rooted at position R. From R, one can move to positions A and B. From A, one can move to positions C and D. Leaves C, D, and E are, respectively, assigned values −1, 3, and −2. From these, the values associated with positions A, B, and R are computed by the negamax algorithm described in the text.
2 The Model In this section we review search trees, the introduction of distribution valued evaluation functions, and the propagation of distributions up trees. In playing a game, one typically grows a search tree (see Figure 1) by looking ahead. The present position is R, or root, and one has expanded a portion of the game tree looking ahead (down). If the exact value of each leaf position were known, and assuming that both players knew those values, the value at each of the other nodes would then be determined by the negamax algorithm.2 This determines the value of a node to be the maximum of the negative of the values of its successor positions. This negamax algorithm is isomorphic to the alternative max-min algorithm. Both simply assume that what is good for a player is bad for his opponent and recursively value the nodes on the assumption that the player makes his optimal choice according to the valuation. Usually one does not have the computational resources to search to terminal positions of the game, and this introduces uncertainty into the values of the leaves. Computer game programs adopt some form of evaluation function that estimates the expected values of leaf positions. For example, the 2 Incidentally, we speak of the values of the leaf positions as “causing” the values of the positions at the internal nodes because the values of the internal nodes are in fact defined from the values of the source positions by the negamax algorithm. For example, a game-theoretic won position is, by definition, one from which one has a winning move—a move that takes one to a position from which one’s opponent’s moves all lose for him.
218
Eric B. Baum and Warren D. Smith
evaluation function might be a neural net trained to predict game outcome. Standard game-playing algorithms do not handle the uncertainty in a principled way, but simply use these estimates as if they were exact values for the leaves and propagate them using negamax. We have discussed (Baum & Smith, 1997) how this leads to errors. Instead, we proposed adopting an evaluation function that associates with each leaf a probability distribution that estimates the probability a position takes will acquire various values (see Figure 2).3 The distribution associated with a given source typically depends on features of the position; in chess, for example, it might depend on the pawn structure and the material balance. This evaluation function is typically prepared by training from game data. Our evaluation function returns a distribution written as a weighted sum over point masses4 ρ (η) (x) =
X
(η)
(η)
pi δ(x − xi ).
(2.1)
i
Here ρ (η) (x) is the probability distribution giving the probability node η has (η) value x. pi is thus the probability that node η has value xi . δ is the Dirac delta function. We assume that the distributions at the leaves (also called sources) are probabilistically independent. This does not imply that the means of the distributions are similar or dissimilar, anymore than the means of sources for any other Bayes’ net. In the stereotypical causal net (cf. Jensen, 1996, p. 10), a source for “earthquake occurred” and “burglary occurred” are deemed independent (absent evidence regarding the value of their descendants), yet the mean value of each source is low: earthquakes and burglaries are rare events. Similarly, when we look ahead from position R to estimate the values at positions C and D, our estimate may well be that these similar positions have similar expected values. We are assuming that our uncertainties in the values of these positions are uncorrelated. We are assuming (as is common in game playing algorithms) that we have no additional information about the value of nodes internal to the search tree. As shown in Figure 2b, the search tree often collapses to a search DAG, because of transpositions: different nodes of the tree may correspond to the same board positions. We assume that the value of each source is drawn from its associated probability distribution. We further assume (effectively) that a player later reaching any position internal to the search DAG (below the root) would then be able to search below the leaves, with the search giving more informa3 These values are not game-theoretic win or loss, but rather our expected payoff if we play from the position. For a discussion, see Baum and Smith (1997). 4 One could imagine a more general form for such distributions, but assuming this form is essential to our algorithms.
Propagating Distributions Up Directed Acyclic Graphs
219
Figure 2: (a) A search tree rooted at position R. From R one can move to positions A and B. From A one can move to positions C and D. Looking ahead from position R, we have assigned probability distributions to the leaves C, D, and E using an evaluation function. The distributions associated with the nodes are given as tuples of values, with subscripts encoding their respective probabilities. For example, node C takes value −1 with probability 1/2 and value 2 with probability 1/2. We have then calculated distributions at the nodes A, B, and R using the negamax rule, described in the text, assuming that the distributions at the leaves are independent. For example, the joint probability that C has value −1 and D has value 3 is 3/8. In this event, A will have value equal to the maximum of the negatives of the values of its children, or max(1, −3) = 1. Now it turns out that the board position E is identical to the board position D, although these positions were reached by different sequences of moves from R. This is represented with the DAG of (b) where nodes D and E have been merged. The distributions at A and B are no longer independent, since they have a common parent, which leads to a different distribution for R. It is customary when talking about a game tree to refer to “children” as being positions reached by moving from “parents.” However, in the DAG, the arrows flow the other way, following the direction of causality: the value of the position at node A, for instance, is determined by the values of the positions C and D. In the text we have used the language natural to the DAG representation, wherein the successor positions in the game tree are called the “parents,” so we speak of D as the parent of A, but alternatively speak of D as a successor of A.
tion about the value of the leaves. Our model is that the player would then know the exact value of the leaves (chosen from the associated distribution) and would move accordingly. This induces a probability distribution at all the internal nodes: the probability that internal node X has value x is the joint probability that the sources are in some configuration y yielding x by negamax propagation (see Figure 2). For further motivation and discussion of this approach see Baum and Smith (1997). For a discussion of techniques
220
Eric B. Baum and Warren D. Smith
for training such distribution-based evaluation functions and experimental results, see Smith, Baum, Garrett, and Tudor (1997). It is equivalent to look at this in the ensemble picture (Baum & Smith, 1997) (see Figure 3). Instead of one DAG with a probability distribution at each node, there is equivalently an ensemble of DAGs, each with a scalar value at each leaf. Each DAG represents one configuration, that is, assignment of values to the leaves. Each DAG has values assigned to all internal nodes as the negamax of their successors (parents in the DAG). The DAGs each have a probabilistic weight, given by the product of the weights of the leaves. The distribution induced on an internal node as in the preceding paragraph is the probability that if you choose a DAG from the ensemble, it assigns a given value to the node. Unfortunately, the number of DAGs in the ensemble grows exponentially with the number of leaves, so the ensemble picture is not very useful for computation. The distribution at R depends importantly on recognizing the DAG structure. If we simply unwrapped the DAG of Figure 2b into the tree of Figure 2a (so that node D was treated as two different leaves with identical distributions), we would treat the distributions at A and B as independent. This leads to a distribution at node R different from the correct value. Since the Bayesian algorithm proposed in Baum and Smith (1997) uses such distributions in decisions about where to search and which move to make, these differences can importantly affect play. In this case, for example, treating the graph as a tree leads to a dependence of the distribution at R on the distribution at C, so expanding C would have some utility, whereas we will see below that R’s distribution depends only on D when correctly calculated in the DAG. Recall the efficient algorithm for propagating (independent) distributions on a tree (Palay, 1985; Baum & Smith, 1997). One first defines two different types of cumulative distribution functions (CDFs): Falling CDF : c(η) (x) =
X
(η)
pi
(η)
i, xi ≥x Z ∞
ρ (η) (u)du = Prob (value(η) ≥ x) X (η) Rising CDF : c(η) (x) = pi =
(2.2)
x
(η)
=
i, xi ≤x Z x −∞
ρ (η) (u)du = Prob (value(η) ≤ x) .
(2.3)
Because our ρ is a sum of point masses, these CDFs are “staircase” functions. Evidently there is a bijection between staircase CDFs and distributions of the form of equation 2.1. Now one may efficiently calculate the distribution at a position from the distributions at its successors using the relation (Palay, 1985):
Propagating Distributions Up Directed Acyclic Graphs
221
Figure 3: The DAG of Figure 1b is equivalent to an ensemble of four DAGs, with scalar values associated with each node and a weight associated with each DAG. The values at the leaves C and D are drawn from the distribution associated with nodes A and B. The weight of the DAG is the product of the probabilities of the values of C and D. The values assigned to the nodes A, B, and R in each DAG in the ensemble are given by the max of the negatives of the values of their children. Choose a DAG from the ensemble of Figure 2 according to its weight. The distributions assigned to nodes A, B, and R in Figure 2b give the probability this DAG assigns a given value to these nodes.
c(position) (−x) =
b Y
c(successori ) (x).
(2.4)
i=1
This relationship holds because, under the definition of negamax, a position’s value is less than −x if and only if all its successors’ values are above x, and because on a tree, the distributions at the successors are indepen-
222
Eric B. Baum and Warren D. Smith
dent. This equation leads5 to an O(N log b) time algorithm for computing a position’s distribution, given its successors’, where N is the total number of point masses in all the successors’ distributions together, and b is the number of successors. From this, one gets an O(NT dL log b) time algorithm for computing the distributions at all nodes in a search tree, where NT is the total number of spikes in all leaf distributions, dL is the average depth of the spikes in the leaves, and log b is the log of the geometric mean branching factor. 3 Results on DAGS Section 2 described how to valuate the nodes of a tree. We now calculate the distributions at each node in the DAG, where the distributions at siblings can no longer be assumed independent. First note Lemma 1. The CDF at any node in a DAG is a multilinear function of the CDFs at the leaves. Proof. This is evident in the ensemble viewpoint of Figure 3. The probability distribution at any node is given by a sum over the different configurations of its leaf descendants, with configurations weighted by their probabilities. But each term in this sum is a multilinear function of the leaf probabilities, since we have assumed the leaf distributions are independent. Now we have: Theorem 1. To propagate CDFs up a DAG, first use equation 2.4 to naively yield the CDF at each node as a polynomial function of the CDFs at the leaves. Then expand that polynomial into a sum of monomials, and wherever in this formula any leaf CDF ci (x) appears raised to a power greater than one, simply replace the power by 1. Example.
See Figure 4.
5 There are two ways to achieve O(N log b) run time. First, we could do log b passes, 2 at each pass combining pairs of child distributions and thus reducing the number of such distributions by a factor of 2 so that after the final pass only one would remain. Each such pair-combining step (which involves computer code similar to “ordered list merge”) would run in time linear in the number of point masses in the two pairs. The second method would combine all b child distributions at once, using a so-called heap data structure to facilitate finding the next point mass to process in O(log b) time rather than the naive O(b).
Propagating Distributions Up Directed Acyclic Graphs
223
Figure 4: From the negamax DAG of Figure 3a, assigning falling CDFs C(x) to leaf C and D(x) to leaf D, we naively calculate from equation 2.4 the falling CDFs assigned to leaves A, B, and R, as shown in Figure 3b. (Note that 1 − c(x) ≡ c(x).) Linearizing at the root (1−(1−C)(1−D))D = CD+D2 −CD2 → CD+D−CD = D) takes care of the dependency and yields Figure 3c. That this is correct is evident from the alpha-beta cutoff structure in the ensemble viewpoint. Node C cannot influence the root in minimax propagation, as either the value of node D is greater than that of node C, in which case C disappears at max node A, or D is smaller, in which case C disappears at min node R. L Proof. By the lemma above, the CDF ci (x) at node Pi is a sum ofQ2 monomials (where there are L leaves); it can be written as S⊆{1,...,L} aS j∈S cj (−1d x) where d is the height of node i above leaf j. (Note that the players alternate turns, so we never have two paths from a leaf to the same position of differing parities.6 ) We must determine only the coefficients aS . There are 2L such 6 A position is defined by both board position and the player to move: two positions with identical board position but different players to move are different. In games where players may pass, such as Go, typically positions may be importantly distinct depending on whether the last move was a pass. The Go game ends when the players pass consecutively. There may be other information distinguishing otherwise identical positions in principle. In chess, when a position is repeated three times, it is a draw. Thus, a position that has recurred twice before is different from one that has not previously recurred in the very real sense that moves from it have different consequences. This is typically neglected
224
Eric B. Baum and Warren D. Smith
coefficients, which can thus be determined by interpolating the function at 2L points. At the 2L assignments where we set the cj (x) to be 1 or 0, we have cjn = cj and thus the formula given in the theorem is correct by inspection. Unfortunately the prescription offered is not efficiently computable for minimax DAGs with a large number L of sources because the number of terms in the formula is 2L . The following theorem shows that one cannot generally compute the distributions in time polynomial in the number of leaves.7 Theorem 2. It is #P-hard to evaluate the mean value of the root of a pure negamaxing game DAG whose leaves are probability distributions. The result holds even if the leaf probability distributions are all independent 50-50 coin flips (i.e., assume value 0 with probability 1/2, 1 with probability 1/2), the DAG has only negamaxing nodes, and has depth 2, and no node’s branching number (except for the root node’s) exceeds 2. Proof. Regard the leaves as boolean (0–1) variables arising from independently and identically distributed coin flips. The minimax value of a two-level DAG, whose root is min and whose depth 1 nodes are max nodes, is then an AND of the ORs of the children of the level 1 nodes. Let the depth 1 nodes have branching factor ≤ 2. Then determining the probability that the root assumes value 1 is precisely (except for a normalization by a factor of 2L , where the DAG has L leaves) the problem of counting satisfying assignments in “monotone 2-SAT,” which was shown to be #P-complete in Valiant (1979). We can approximate the exact CDFs at all nodes, taking full account of the DAG dependencies, by Monte Carlo evaluation. (See also Karp, Luby, & Madras, 1989.) This may be useful in evaluating our move choice, but we believe it is likely to be too slow in convergence to be useful in deciding how to expand the tree. The reason is the following. Game players are interested in changes to the utility coming from expanding N leaves, where N is a large number. Hence we are interested in changes in utility of order 1/N caused by any given leaf. Hence our Monte Carlo calculation must sample from regions in the ensemble space with volume of order 1/N. But of course it takes time N to write down any given configuration, so the fastest one might in representing the search tree as a DAG, but this neglect gives rise to a problem called the graph history interaction (Palay, 1985, p. 131), which in practice causes errors in computer play. 7 Unless P = NP = #P. #P-hard problems are at least as hard as NP-complete problems. An easy corollary is that it is also #P-hard even to determine the sign of the mean value of a game DAG.
Propagating Distributions Up Directed Acyclic Graphs
225
imagine a naive8 Monte Carlo calculation being computed9 is Ä(N2 ), which would be unacceptable for game applications, where alpha-beta programs often grow trees involving billions of leaves. 4 Discussion Alternatively, for small DAGs our exact prescription is feasible. This suggests the use of a local heuristic node-valuation method in which the dependencies arising from the local structure of a game DAG are treated exactly, but where we continue to pretend that the CDFs at “far-away” DAG nodes are independent. One could, for example, use the propagation algorithm of Theorem 1 to calculate the CDF at each node exactly in terms of the CDFs of its descendants d levels deeper in the DAG. In the limiting case of d = 1, we neglect all DAG structure and obtain our usual linear time propagation algorithm for trees. For finite d > 1 we compromise, using an algorithm that accounts for local DAG structure. The size d that is acceptably fast in practice will depend on the branching factor of the nodes and the DAG structure. There are, of course, numerous algorithms in the literature for propagating probability distributions up DAGs, notably the junction tree algorithm (Lauritzen & Spiegelhalter, 1988; Jensen, 1996), the arc reversal algorithm (Shachter, 1986), and conditioning (Pearl, 1986). These algorithms are more general than ours, since they do not assume negamax propagation, but therefore they do not attempt to profit from the special nature of game tree propagation. They are exponential time in the worst case, but may be practical in some circumstances, so it is interesting to ask how they might fare in practice on game DAGs. In many games, there is relatively little DAG structure, so taking account of it would be relatively easy but equally relatively uninteresting. In Othello, for example, use of a hash table typically speeds up alpha-beta algorithms by a factor of only about 2 (Mr. Buro, personal communication, 1997), indicating a relatively uninteresting DAG structure. Even here, however, it seems unlikely that standard techniques (or our new algorithm in its exact form) would be tractable. The junction tree algorithm, for example, attaches tables to cliques in the junction graph (cf. Jensen, 1996) of size the product, 8 One might imagine a more sophisticated Monte Carlo procedure that evaluated the utility of leaf L by sampling uniformly directly from the region in which the value of leaf L is relevant and then computed the integral by some procedure analogous to that of Karp et al. (1989), but we have been unable to construct a procedure along these lines that is efficient and rapidly mixing. 9 Actually the situation is worse than this, for two reasons. First, we must sample from such regions many times because of the inherent noise in the Monte Carlo procedure. Second, in a game like chess, we are interested in move accuracies at least of order 1/50 (if we would like to make 50 moves without expecting an error) and so are interested in changes in utility of order 1/(50N) from expanding a given leaf.
226
Eric B. Baum and Warren D. Smith
over the nodes in the clique, of the number of the states of the variables at each node. The use of such a large matrix is an example where working for general dependencies of one variable on its parents, rather than exploiting the special nature of negamax propagation, may create problems. The number of states of each variable in the game DAG is large: proportional to the number of sources that are ancestors of the node, with a proportionality factor that can be of order 10 (Smith et al., 1997). The size of the largest cliques will be large. A node in the DAG has parents for every following position in the game tree. For some nodes (e.g., the root, or in losing positions) where one must consider every legal move, this can be 20 or larger, which will result in cliques of at least size 20. Moreover, the DAG is of depth about 22 and contains loops of circumference up to 44, which will involve adding additional chords to the junction graph. For chess, the DAG structure is much richer yet than for Othello, and thus there is much more interest in exploiting it. Use of a hash table typically lowers the effective branching ratio of alpha-beta chess programs from about 6 to about 4 (Buro, personal communication, 1997). In other words, the alpha-beta program grows a DAG of perhaps 106 sources, which would have perhaps 100 times as many leaves if unwrapped into a tree. This DAG is depth, say, 11 (and sometimes could be considerably deeper for our Bayesian algorithm, searching the most utilitarian lines), and has loops of circumference up to twice its depth. It will in practice have nodes with 50 or more parents, and thus cliques in the junction graph at least of size 50. To compete with alpha-beta algorithms (not using probability distributions), the propagation step must be reasonably fast—taking time growing at most linearly in the size of the DAG, and with an acceptable proportionality constant so that propagation of probability distributions occupies only a reasonable constant fraction of total thinking time. It seems unlikely that any algorithm will suffice in practice for tractable, exact propagation of probability distributions on such graphs, but some version of the heuristic we suggested above (e.g., for small enough d) could achieve acceptable speed. Only extensive experimentation, however, can determine whether a useful level of accuracy could be obtained if the heuristic were tuned for sufficient speed. Acknowledgments We thank the referees for comments on the draft of this article. References Baum, E. B., & Smith, W. D. (1997). A Bayesian approach to relevance in game playing. AI Journal, 97, 195–242. Cooper, F. G. (1990). The computational complexity of probabilistic inference using Bayesian belief networks. AIJ, 42, 393–405.
Propagating Distributions Up Directed Acyclic Graphs
227
Jensen, F. (1996). An introduction to Bayesian networks. London: UCL Press. Karp, R. M., Luby, M., & Madras, N. (1989). Monte-Carlo approximation algorithms for enumeration problems. Journal of Algorithms, 10, 429–448. Lauritzen, S. L., & Spiegelhalter, D. J. (1988). Local computations with probabilities on graphical structures and their application to expert systems (with discussion). J. Royal Statistical Soc. Ser B, 50, 157–224. Palay, A. J. (1985). Searching with probabilities. New York: Pitman. Pearl, J. (1986). A constraint-propagation approach to probabilistic reasoning. In L. M. Kanal & J. Lemmer (Eds.), Uncertainty in artificial intelligence (pp. 357– 370). Amsterdam: North-Holland. Shachter, R. D. (1986). Evaluating influence diagrams. Operations Research, 34, 871–882. Smith, W. D., Baum, E. B., Garrett, C., & Tudor, R. (1997). Experiments with a Bayesian game player. Unpublished manuscript. Available online at: http://www.neci.nj.nec.com:80/homepages/eric/eric.html. Valiant, L. G. (1979). The complexity of enumeration and reliability problems. SIAM J. Computing, 8, 410–421.
Received January 8, 1998; accepted June 19, 1998.
LETTER
Communicated by Dean Pomerleau
Modeling and Prediction of Human Behavior Alex Pentland Massachusetts Institute of Technology, Cambridge, MA 02139, U.S.A.
Andrew Liu Nissan Cambridge Basic Research, Cambridge, MA 02142, U.S.A.
We propose that many human behaviors can be accurately described as a set of dynamic models (e.g., Kalman filters) sequenced together by a Markov chain. We then use these dynamic Markov models to recognize human behaviors from sensory data and to predict human behaviors over a few seconds time. To test the power of this modeling approach, we report an experiment in which we were able to achieve 95% accuracy at predicting automobile drivers’ subsequent actions from their initial preparatory movements. 1 Introduction Our approach to modeling human behavior is to consider the human as a device with a large number of internal mental states, each with its own particular control behavior and interstate transition probabilities. Perhaps the canonical example of this type of model would be a bank of standard linear controllers (e.g., Kalman filters plus a simple control law), each using different dynamics and measurements, sequenced together with a Markov network of probabilistic transitions. The states of the model can be hierarchically organized to describe both short-term and longer-term behaviors; for instance, in the case of driving an automobile, the longer-term behaviors might be passing, following, and turning, while shorter-term behaviors would be maintaining lane position and releasing the brake. Such a model of human behavior could be used to produce improved human-machine systems. If the machine could recognize the human’s behavior or, even better, if it could anticipate the human’s behavior, it could adjust itself to serve the human’s needs better. To accomplish this, the machine would need to be able to determine which of the human’s control states was currently active and to predict transitions between control states. It could then configure itself to achieve its best overall performance. Because the internal states of the human are not directly observable, this scenario requires that the human’s internal state be determined through an indirect estimation process. To accomplish this, we have adapted the expectation-maximization methods developed for use with hidden Markov Neural Computation 11, 229–242 (1999)
c 1999 Massachusetts Institute of Technology °
230
Alex Pentland and Andrew Liu
models (HMM). By using these methods to identify a user’s current pattern of control and predict the most likely pattern of subsequent control states, we have been able to recognize human driving behaviors accurately and anticipate the human’s behavior for several seconds into the future. Our research builds on the observation that although human behaviors such as speech (Rabinee & Juang, 1986), handwriting (Starner, Makhoul, Schwartz, & Chou, 1994), hand gestures (Yang, Xu, & Chen, 1997; Pentland, 1996), and even American Sign Language (Pentland, 1996; Starner & Pentland, 1995) can be accurately recognized by use of HMMs, they do not produce a model of the observations that is accurate enough for simulation or prediction. In these cases, the human behavior displays additional properties, such as smoothness and continuity, that are not captured within the HMM statistical framework. We believe that these missing additional constraints are typically due to the physical properties of human movement and consequently best described by dynamic models such as the well-known Kalman filter (Kalman & Bucy, 1961). Our proposal is to describe the small-scale structure of human behavior by a set of dynamic models (thus incorporating constraints such as smoothness and continuity) and the large-scale structure by coupling together these control states into a Markov chain. It has been proposed that the basic element of cortical processing can be modeled as a Kalman filter (e.g., Pentland, 1992; Rao & Ballard, 1997); in this article, we are proposing that these basic elements are chained together to form larger behaviors. The resulting framework, first proposed by Pentland and Liu (1995), is related to research in robot control (Meila & Jordan 1995), and machine vision (Isard & Blake, 1996; Bregler, 1997), in which elements from dynamic modeling or control theory are combined with stochastic transitions. These efforts have shown utility in tracking human motion and recognizing atomic actions such as grasping or running. Our approach goes beyond this to describe and classify more extended and elaborate behaviors, such as passing a vehicle while driving, which consist of several atomic actions chained together in a particular sequence. Our framework has consequently allowed us to predict sequences of human behaviors from initial, preparatory motions. 2 Simple Dynamic Models Among the simplest nontrivial models that have been considered for modeling human behavior are single dynamic processes, ˙ k = f(Xk , t) + ξ(t), X
(2.1)
where the function f models the dynamic evolution of state vector Xk at time k. Let us define an observation process, Yk = h(Xk , t) + η(t),
(2.2)
Modeling and Prediction of Human Behavior
231
where the sensor observations Y are a function h of the state vector and time. Both ξ and η are white noise processes having known spectral density matrices. ˆk Using Kalman’s result, we can then obtain the optimal linear estimate X of the state vector Xk by use of the following Kalman filter, Xˆ k = X∗k + Kk (Yk − h(X∗k , t)),
(2.3)
provided that the Kalman gain matrix Kk is chosen correctly (Kalman & Bucy, 1961). At each time step k, the filter algorithm uses a state prediction X∗k , an error covariance matrix prediction P∗k , and a sensor measurement Yk to determine an optimal linear state estimate Xˆ k , error covariance matrix estimate Pˆ k , and predictions X∗k+1 , P∗k+1 for the next time step. The prediction of the state vector X∗k+1 at the next time step is obtained by combining the optimal state estimate Xˆ k and equation 2.1: ˆ k , t)1t. X∗k+1 = Xˆ k + f(X
(2.4)
In some applications this prediction equation is also used with larger time steps, to predict the human’s future state. For instance, in a car, such a prediction capability can allow us to maintain synchrony with the driver by giving us the lead time needed to alter suspension components. In our experience, this type of prediction is useful only for short time periods, for instance, in the case of quick hand motions for up to one-tenth of a second (Friedmann, Starner, & Pentland, 1992a). Classically f, h are linear functions and ξ , η assumed gaussian. It is common practice to extend this formulation to “well-behaved” nonlinear problems by locally approximating the nonlinear system by linear functions using a local Taylor expansion; this is known as an extended Kalman filter. However, for strongly nonlinear problems such as are addressed in this article, one must either employ nonlinear functions and/or multimodal noises, or adopt the multiple-model and sequence-of-models approach described in the following sections. 3 Multiple Dynamic Models Human behavior is normally not as simple as a single dynamic model. The next most complex model of human behavior is to have several alternative models of the person’s dynamics, one for each class of response (Willsky, 1986). Then at each instant we can make observations of the person’s state, decide which model applies, and make our response based on that model. This multiple model approach produces a generalized maximum likelihood estimate of the current and future values of the state variables. Moreover, the cost of the Kalman filter calculations is sufficiently small to make the approach quite practical, even for real-time applications.
232
Alex Pentland and Andrew Liu
Intuitively, this approach breaks the person’s overall behavior down into several prototypical behaviors. For instance, in the driving situation, we might have dynamic models corresponding to a relaxed driver, a very tight driver, and so forth. We then classify the driver’s behavior by determining which model best fits the driver’s observed behavior. Mathematically, this is accomplished by setting up a set of states S, each associated with a Kalman filter and a particular dynamic model, ˆ (i) = X∗(i) + K(i) (Yk − h(i) (X∗(i) , t)), X k k k k
(3.1)
where the superscript (i) denotes the ith Kalman filter. The measurement innovations process for the ith model (and associated Kalman filter) is then 0k(i) = Yk − h(i) (X∗(i) k , t).
(3.2)
The measurement innovations process is zero-mean with covariance R. The ith measurement innovations process is, intuitively, the part of the observation data that is unexplained by the ith model. The model that explains the largest portion of the observations is, of course, the model most likely to be correct. Thus, at each time step, we calculate the probability Pr(i) of the m-dimensional observations Yk given the ith model’s dynamics,
Pr(i) (Yk |X∗k ) =
´ ³ exp − 12 0k(i)T R−1 0k(i) (2π)m/2 Det(R)1/2
,
(3.3)
and choose the model with the largest probability. This model is then used to estimate the current value of the state variables, predict their future values, and choose among alternative responses. After the first time step, where R and Pk are assumed known a priori, they may be estimated from the incoming data (Kalman & Bucy, 1961). Note that when optimizing predictions of measurements 1t in the future, equation 3.2 must be modified slightly to test the predictive accuracy of state estimates from 1t in the past: (i) ˆ (i) 0k(i) = Yk − h(i) (X∗(i) k−1t + f (Xk−1t , 1t)1t, t)).
(3.4)
We have used this method accurately to remove lag in a high-speed telemanipulation task by continuously reestimating the user’s arm dynamics (e.g., tense and stiff, versus relaxed and inertia dominated) (Friedmann, Starner, & Pentland, 1992b). We found that using this multiple-model approach, we were able to obtain significantly better predictions of the user’s hand position than was possible using a single dynamic or static model.
Modeling and Prediction of Human Behavior
Doing nothing
Prepare
Execute
233
Conclude
Figure 1: A Markov dynamic model of driver action. Only the substates in the Prepare state will be used for action recognition.
4 Markov Dynamic Models In the multiple dynamic model, all the processes have a fixed likelihood at each time step. However, this is uncharacteristic of most situations, where there is often a fixed sequence of internal states, each with its own dynamics. Consider driving through a curve. The driver may be modeled as having transitioned through a series of states λ = (s1 , s2 , . . . , sk ), si ²S, for instance, entering a curve, in the curve, and exiting a curve. Transitions between these states happened only in the order indicated. Thus, in considering state transitions among a set of dynamic models, we should make use of our current estimate of the driver’s internal state. We can accomplish this fairly generally by considering the Markov probability structure of the transitions between the different states. The input to decide the person’s current internal state (e.g., which dynamic model currently applies) will be the measurement innovations process as above, but instead of using this directly in equation 3.3, we will also consider the Markov interstate transition probabilities. We will call this type of multiple dynamic model a Markov dynamic model (MDM). Conceptually, MDMs are exactly like HMMs except that the observations are the innovations (roughly, prediction errors) of a Kalman filter or other dynamic, predictive process. In the case of the dynamic processes used here, these innovations correspond to accelerations that were not anticipated by the model. Thus, our MDMs describe how a set of dynamic processes must be controlled in order to generate the observed signal rather than attempting to describe the signal directly. The initial topology for an MDM can be determined by estimating how many different states are involved in the observed phenomenon. Finetuning this topology can be performed empirically. Figure 1, for instance, shows a four-state MDM to describe long-time-scale driver behavior. Each state has substates, again described by an MDM, to describe the fine-grain structure of the various behaviors.
234
Alex Pentland and Andrew Liu
As with HMMs, there are three key problems in MDM use (Huang, Ariki, & Jack, 1990): the evaluation, estimation, and decoding problems. The evaluation problem is that given an observation sequence and a model, what is the probability that the observed sequence was generated by the model (Pr(Y|λ))? If this can be evaluated for all competing models for an observation sequence, then the model with the highest probability can be chosen for recognition. As with HMMs, the Viterbi algorithm provides a quick means of evaluating a set of MDMs as well as providing a solution for the decoding problem (Huang et al., 1990; Rabiner & Juang, 1986). In decoding, the goal is to recover the state sequence given an observation sequence. The Viterbi algorithm can be viewed as a special form of the forward-backward algorithm where only the maximum path at each time step is taken instead of all paths. This optimization reduces computational load and allows the recovery of the most likely state sequence. Since Viterbi guarantees only the maximum of Pr(Y, S|λ) over all state sequences S (as a result of the first-order Markov assumption) instead of the sum over all possible state sequences, the resultant scores are only an approximation. However, Rabiner and Juang (1986) show that this is typically sufficient. Because the innovations processes that drive the MDM interstate transitions are continuous, we must employ the actual probability densities for the innovations processes. Fortunately, Baum-Welch parameter estimation, the Viterbi algorithm, and the forward-backward algorithms can be modified to handle a variety of characteristic densities (Huang et al., 1990; Juang, 1985). However, in this article, the densities will be assumed to be as in equation 3.3. 5 An Experiment Using Markov Dynamic Models Driving is an important, natural-feeling, and familiar type of human behavior that exhibits complex patterns that last for several seconds. From an experimental point of view, it is important that the number of distinct driving behaviors is limited by the heavily engineered nature of the road system, and it is easy to instrument a car to record human hand and foot motions. These characteristics make driving a nearly ideal experimental testbed for modeling human behaviors. We have therefore applied MDMs to try to identify automobile drivers’ current internal (intentional) state and to predict the most likely subsequent sequence of internal states. In the case of driving, the macroscopic actions are events like turning left, stopping, or changing lanes. The internal states are the individual steps that make up the action, and the observed variables will be changes in heading and acceleration of the car. The intuition is that even apparently simple driving actions can be broken down into a long chain of simpler subactions. A lane change, for instance,
Modeling and Prediction of Human Behavior
235
may consist of the following steps: (1) a preparatory centering the car in the current lane, (2) looking around to make sure the adjacent lane is clear, (3) steering to initiate the lane change, (4) the change itself, (5) steering to terminate the lane change, and (6) a final recentering of the car in the new lane. In this article we are statistically characterizing the sequence of steps within each action and then using the first few preparatory steps to identify which action is being initiated. To continue the example, the substates of “prepare” shown in Figure 1 might correspond to centering the car, checking the adjacent lane, and steering to initiate the change. To recognize which action is occurring, one compares the observed pattern of driver behavior to Markov dynamic models of each action, in order to determine which action is most likely given the observed pattern of steering and acceleration and braking. This matching can be done in real time on current microprocessors, thus potentially allowing us to recognize a driver’s intended action from his or her preparatory movements. If the pattern of steering and acceleration is monitored internally by the automobile, then the ability to recognize which action the driver is beginning to initiate can allow intelligent cooperation by the vehicle. If heading and acceleration are monitored externally by video cameras (as in Boer, Fernandez, Pentland, & Liu, 1996), then we can more intelligently control the traffic flow. 5.1 Experimental Design. The goal is to test the ability of our framework to characterize the driver’s steering and acceleration and braking patterns in order to classify the driver’s intended action. The experiment was conducted within the Nissan Cambridge Basic Research driving simulator, shown in Figure 2a. The simulator consists of the front half of a Nissan 240SX convertible and a 60 degree (horizontal) by 40 degree (vertical) image projected onto the wall facing the driver. The 240SX is instrumented to record driver control input such as steering wheel angle, brake position, and accelerator position. Subjects were instructed to use this simulator to drive through an extensive computer graphics world, illustrated in Figure 2b. This world contains a large number of buildings, many roads with standard markings, and other moving cars. Each subject drove through the simulated world for approximately 20 minutes; during that time, the driver’s control of steering angle and steering velocity, car velocity, and car acceleration were recorded at 1/10 second intervals. Drivers were instructed to maintain a normal driving speed of 30 to 35 miles per hour (13–15 meters per second). From time to time during this drive, text commands were presented onscreen for 1 second, whereupon the subjects had to assess the surrounding situation, formulate a plan to carry out the command, and then act to execute the command. The commands were: (1) stop at the next intersection, (2) turn left at the next intersection, (3) turn right at the next intersection, (4) change
236
(a)
Alex Pentland and Andrew Liu
(b)
Figure 2: (a) Nissan Cambridge Basic Research simulator. (b) Part of the simulated world seen by the subjects.
lanes, (5) pass the car in front of you, and (6) do nothing (e.g., drive normally, with no turns or lane changes). A total of 72 stop, 262 turn, 47 lane change, 24 passing, and 208 drive-normal episodes were recorded from eight adult male subjects. The time needed to complete each command varied from approximately 5 to 10 or more seconds, depending on the complexity of both the action and the surrounding situation. Command presentation was timed relative to the surroundings in order to allow the driver to execute the command in a normal manner. For turns, commands were presented 40 meters before an intersection (≈ 3 seconds, a headway used in some commercially available navigation aids), for passing the headway was 30 meters (≈ 2 seconds, which is the mean headway observed on real highways), and for stopping the headway was 70 meters (≈ 5 seconds). The variables of command location, road type, surrounding buildings, and traffic conditions were varied randomly thoughout the experiment. The dynamic models used were specific to a Nissan 240SX. Using the steering and acceleration data recorded while subjects carried out these commands, we built three-state models of each type of driver action (stopping, turn left, turn right, lane change, car passing, and do nothing) using expectation-maximization (EM) for the parameters of both the Markov chain and the state variables (heading, acceleration) of the dynamic models (Baum, 1972; Juang, 1985; Rabiner & Juang, 1986). Three states were used because preliminary investigation on informally collected data showed that the three state models performed slightly better than four or five state models. The form of the dynamic models employed is described in the appendix. To assess the classification accuracy of these models, we combined them with the Viterbi recognition algorithm and examined the stream of drivers’ steering and acceleration innovations in order to detect and classify each driver’s actions. All of the data were labeled, with the “do nothing” label serving as a “garbage class” for any movement pattern other than the
Modeling and Prediction of Human Behavior
237
five actions of interest. We then examined the computer’s classifications of the data immediately following each command and recorded whether the computer had correctly labeled the action. To obtain unbiased estimates of recognition performance, we employed the “leaving one out” method, and so can report both the mean and variance of the recognition rate. Recognition results were tabulated at 2 seconds after the beginning of the presentation of a command to the subject, thus allowing the driver up to 2 seconds to read the command and begin responding. As will be seen in the next section, the 2 second time point is before there is any large, easily recognizable change in car position, heading, or velocity. Because the driving situation allows visual preview of possible locations for turning, passing, and so forth, we may presume that the driver was primed to react swiftly. As the minimum response time to a command is approximately 0.5 second, the 2-second point is at most 1.5 seconds after the beginning of the driver’s action, which is on average 20% of the way through the action. 5.2 Results. Because the MDM framework is fairly complex, we must first try simpler methods to classify the data. For comparison, therefore, we used Bayesian classification, where the data were modeled using multivariate gaussian, nearest-neighbor, or three-state HMMs. The modeled data were the measured accelerator, brake, and steering wheel positions over the previous 1 second. 5.2.1 Results Using Classical Methods. At 2 seconds after the onset of the command text (approximately 1.5 seconds after the beginning of action, or roughly 20% of the way through the action) mean recognition accuracy (as evaluated using the leaving-one-out method) was not statistically different from chance performance for any of these three methods. Many different parameter settings and similar variations on these techniques were also tried, with no success. Examination of the data makes the reasons for these failures fairly clear. First, each action can occur over a substantial range of time scales, so a successful recognition method must incorporate some form of time warping. Second, even for similar actions, the pattern of brake taps, steering corrections, and accelerator motions varies almost randomly, as the exact pattern depends on microevents in the environment, variations in the driver’s attention, and so forth. It is only when we integrate these inputs via the Kalman filter’s physical model, to obtain the car’s motion state variables (e.g., velocity, acceleration, heading), that we see similar patterns for similar actions. This is similar to many human behaviors, where the exact sequence of joint angles and muscle activations is unimportant; it is the trajectory of the end effectors or center of mass that matters. These data characteristics cause methods such as multivariate gaussian and nearest-neighbor to fail because of time scale variations; HMM and time
238
Alex Pentland and Andrew Liu
75
REACTION TIME
Recognition Accuracy (%)
100
50
25
• Averaged over all maneuvers
0 0
1
2
3
4
5
6
Time after command onset (sec)
7 Manuever Completion (~7.5 sec avg.)
Figure 3: Recognition accuracy versus time. Very good recognition accuracy is obtained well before the main or functional part of the behavior.
warping methods fail because they operate on the control inputs rather than the intrinsic motion variables. By inserting an integrating dynamic model between the control inputs and the HMM, we bring out the underlying control intended by the human. 5.2.2 Results Using MDMs. At 2 seconds after the onset of the command text (approximately 1.5 seconds after the beginning of action, or roughly 20% of the way through the action), mean recognition accuracy was 95.24% ± 3.1%. These results, plus accuracies at longer lag times, are illustrated in Figure 3. As can be seen, the system is able to classify these behaviors accurately very early in the sequence of driver motions. It could be argued that the drivers are already beginning the main or functional portion of the various behaviors at the 2 second point, and that it is these large motions that are being recognized. However, the failure of standard statistical techniques to classify the behaviors demonstrates that there are no changes in mean position, heading, or velocity at the 2 second point that are statistically different from normal driving. Thus, the behaviors are being recognized from observation of the driver’s preparatory movements.
Modeling and Prediction of Human Behavior
239
To illustrate this point, we observe that 2 seconds after the onset of a lane change command, the MDM system was able to recognize the driver’s action with 93.3% accuracy (N = 47), even though the vehicle’s lateral offset averaged only 0.8 ± 1.38 meters (lateral offset while going straight has a standard deviation of σ = 0.51 meters). Since this lateral displacement is subtantially before the vehicle exits the lane (lane width is 4 meters), it is clear that the MDM system is detecting the maneuver well before the main or functional part of the behavior. For comparison, to achieve a 93% accuracy by thresholding lateral position, one would need to tolerate a 99% false alarm rate while going straight. To test whether our sample is sufficiently large to encompass the range of between-driver variation adequately, we compared these results to the case in which we train on all subjects and then test on the training data. In the test-on-training case the recognition accuracy was 98.8%, indicating that we have a sufficiently large sample of driving behavior in this experiment.
5.2.3 Discussion. We believe that these results support the view that human actions are best described as a sequence of control steps rather than as a sequence of raw positions and velocities. In the case of driving, this means that it is the pattern of acceleration and heading that defines the action. There are lots of ways to manipulate the car’s controls to achieve a particular acceleration or heading, and consequently no simple pattern of hand-foot movement that defines a driving action. Although these results are promising, caution must be taken in transferring them to other human actions or even to real-world driving. It is possible, for instance, that there are driving styles not seen in any of our subjects. Similarly, the driving conditions found in our simulator do not span the entire range of real driving situations. We believe, however, that our simulator is sufficiently realistic that comparable accuracies can be obtained in real driving. Moreover, there is no strong need for models that suit all drivers; most cars are driven by a relatively small number of drivers, and this fact can be used to increase classification accuracy. We are exploring these questions, and the initial results support our optimism.
6 Conclusion We have demonstrated that we can accurately categorize human driving actions very soon after the beginning of the action using our behavior modeling methodology. Because of the generic nature of the driving task, there is reason to believe that this approach to modeling human behavior will generalize to other dynamic human-machine systems. This would allow us to recognize automatically people’s intended action, and thus build control systems that dynamically adapt to suit the human’s purpose better.
240
Alex Pentland and Andrew Liu
Appendix: Dynamic Car Models The kinematic model is a three-wheeled model with two front wheels and one rear wheel. Given access to a system clock, we can figure out the amount of time elapsed since the last frame, 1t. To calculate the distance traveled since the last frame, we calculate dlin = v1t, where the function dlin is the linear distance traveled since the last frame and v is the speed of the car. To calculate the new position of the car, we must consider whether the car is turning or moving straight ahead. If moving ahead, then E PEnew = PE + (dlin H). If the car is making a turn, then E PEnew = PE + (rcircle sin(θd )H), where the angular travel of the car θd is given by θd = dlin /rcircle , and the turning circle of the car, rcircle , is given by rcircle =
dWB , tan θFW
where dWB is the wheelbase of the car. The new heading of the car is figured from the following equation, E LFW + V E RFW ) − PEnew , HE new = 0.5(V E LFW is the vector position of the front left wheel and V E RFW is the vector where V position of the front right wheel. Finally, we normalize the new heading vector, HE new . HE new = kHE new k The dynamical equations are equally simple for the vehicle. The new velocity of the car is computed based on the energy produced by the engine balanced by the drag from air and rolling resistance and braking. To calculate the new velocity, we first compute the kinetic energy based on the present velocity using 1 Mv2 , 2 where M is the mass of the car. Then we calculate the work done by the engine to propel the car forward, which is simply given by KEold =
Wengine = aPmaxeng 1t,
Modeling and Prediction of Human Behavior
241
where Pmaxeng is the maximum power output of the engine and a is the position of the accelerator (0 → 1). The limitation of this model is that the work is independent of the rpm. The work done by the brakes to slow the car is given by Wbrake = Fmaxbrake dlin (b + 0.6 bpark ), where Fmaxbrake is the maximum braking force of the brakes, b is the postion of the brake pedal (0 → 1), and bpark is the parking brake, which is either 0 or 1. The contribution of potential energy, if the world is not flat, can be expressed as: y
1PE = (Pnew − Py )Mg, y
where Py is the current y-coordinate and Pnew is the newly computed ycoordinate. The acceleration of gravity, g, is given in m/s2 . To calculate the effect of drag, we first find the total force resulting from the air drag and road friction, Fdrag = µair v2 + µroad Mg, where µroad is the coefficient of friction of the road and µair is the drag coefficient of the car. The total deceleration energy is the sum of the drag energy and the brake energy, Edrag = dlin Fdrag + Wbrake . Our final energy balance equation is: KEnew = KEold + Wengine − 1PE − Edrag r 2KEnew . vnew = M The skid state of the car depends on the speed and steering wheel angle and road surface coefficient of friction. In our experiments, we used constants that apply to a Nissan 240SX and a specified road-tire configuration. In this model, if the speed is below 11.2 mph, then the car always regains traction. Basically, if the following condition is satisfied, θSW < 24.51, e−0.0699v then the tires are within their adhesion limits and the car will not skid. References Baum, L. (1972). An inequality and associated maximization technique in statistical estimation of probabilistic functions of Markov processes. Inequalities, 3, 1–8.
242
Alex Pentland and Andrew Liu
Boer, E., Fernandez, M., Pentland, A., & Liu, A. (1996). Method for evaluating human and simulated drivers in real traffic situations. In IEEE Vehicular Tech. Conf. (pp. 1810–1814). Atlanta, GA. Bregler, C. (1997). Learning and recognizing human dynamics in video sequences. In IEEE Conf. on Computer Vision and Pattern Recognition (pp. 568– 574). San Jos´e, P.R. Friedmann, M., Starner, S., & Pentland, A. (1992a). Synchronization in virtual realities. Presence, 1, 139–144. Friedmann, M., Starner, T., & Pentland, A. (1992b). Device synchronization using an optimal linear filter. In Proc. ACM 1992 Symposium on Interactive 3D Graphics (pp. 128–134). Boston. Huang, X., Ariki, Y., & Jack, M. (1990). Hidden Markov models for speech recognition. Edinburgh: Edinburgh University Press. Isard, M., & Blake, A. (1996). Contour tracking by stochastic propagation of conditional density. In 1996 European Conf. on Computer Vision (pp. 343–356). Cambridge, U.K. Juang, B. (1985). Maximum likelihood estimation for mixture multivariate observations of Markov chains. AT&T Technical Journal, 64, 1235–1249. Kalman, R., & Bucy, R. (1961). New results in linear filtering and prediction theory. Transaction ASME, 83D, 95–108. Meila, M., & Jordan, M. (1995). Learning fine motion by Markov mixtures of experts (Tech. Memo No. 1567). Cambridge, MA: MIT, AI Laboratory. Pentland, A. (1992). Dynamic vision. In G. A. Carpenter & S. Grossburg (Ed.), Neural networks for vision and image processing (pp. 133–159). Cambridge, MA: MIT Press. Pentland, A. (1996). Smart rooms. Scientific American, 274, 68–76. Pentland, A., & Liu, A. (1995). Toward augmented control systems. In Proc. Intelligent Vehicles ’95 (pp. 350–355). Detroit. Rabiner, L., & Juang, B. (1986, January). An introduction to hidden Markov models. IEEE ASSP Magazine, pp. 4–16. Starner, T., Makhoul, J., Schwartz, R., & Chou, G. (1994). On-line cursive handwriting recognition using speech recognition methods. In ICASSP (vol. 5, pp. 1234–1244). Adelaide, Australia. Starner, T., & Pentland, A. (1995). Visual recognition of American Sign Language using hidden Markov models. In Proc. Int’l Workshop on Automatic Face- and Gesture-Recognition (pp. 38–44). Zurich, Switzerland. Rao, P., & Ballard, D. (1997). Dynamic model of visual recognition predicts neural response properties in the visual cortex. Neural Computation, 9, 721–763. Willsky, A. (1986). Detection of abrupt changes in dynamic systems. In M. Basseville & A. Benveniste (Eds.), Detection of abrupt changes in signals and dynamical systems (pp. 27–49). Berlin: Springer-Verlag. Yang, J., Xu, Y., & Chen, C. S. (1997). Human action learning via hidden Markov model. IEEE Trans. Systems, Man, and Cybernetics, 27, 34–44. Received December 5, 1997; accepted June 24, 1998.
LETTER
Communicated by Klaus Hepp
Analog VLSI-Based Modeling of the Primate Oculomotor System Timothy K. Horiuchi Christof Koch Computation and Neural Systems Program, California Institute of Technology, Pasadena, CA 91125, U.S.A.
One way to understand a neurobiological system is by building a simulacrum that replicates its behavior in real time using similar constraints. Analog very large-scale integrated (VLSI) electronic circuit technology provides such an enabling technology. We here describe a neuromorphic system that is part of a long-term effort to understand the primate oculomotor system. It requires both fast sensory processing and fast motor control to interact with the world. A one-dimensional hardware model of the primate eye has been built that simulates the physical dynamics of the biological system. It is driven by two different analog VLSI chips, one mimicking cortical visual processing for target selection and tracking and another modeling brain stem circuits that drive the eye muscles. Our oculomotor plant demonstrates both smooth pursuit movements, driven by a retinal velocity error signal, and saccadic eye movements, controlled by retinal position error, and can reproduce several behavioral, stimulation, lesion, and adaptation experiments performed on primates. 1 Introduction Using traditional software methods to model complex sensorimotor interactions is often difficult because most neural systems are composed of very large numbers of interconnected elements with nonlinear characteristics and time constants that range over many orders of magnitude. Their mathematical behavior can rarely be solved analytically, and simulations slow dramatically as the number of elements and their interconnections increase, especially when capturing the details of fast dynamics is important. In addition, the interaction of neural systems with the physical world often requires simulating both the motor system in question and its environment, which can be more difficult or time-consuming than simulating the model itself. Mead (1989b, 1990) and others (Koch, 1989; Douglas, Mahowald, & Mead, 1995) have argued persuasively that an alternative to numerical simulations on digital processors is the fabrication of electronic analogs of neurobiological systems. While parallel, analog computers have been used before to simulate retinal processing and other neural circuits (e.g., Fukushima, Neural Computation 11, 243–265 (1999)
c 1999 Massachusetts Institute of Technology °
244
Timothy K. Horiuchi and Christof Koch
Yamaguchi, Yasuda, & Nagata, 1970), the rapid growth of the field of synthetic neurobiology—the attempt to understand neurobiology by building functional models—has been made possible by using commercial chip fabrication processes, which allow the integration of many hundred thousand transistors on a square centimeter of silicon. Designing massively parallel sensory processing arrays on single chips is now practical. Much has happened in this field in the past eight or so years, producing a considerable number of new analog complementary metal oxide semiconductor (CMOS) building blocks for implementing neural models. Local memory modification and storage (Hasler, Diorio, Minch, & Mead, 1995; Diorio, Hasler, Minch, & Mead, 1997), redundant signal representations, and, most recently, long-distance spike-based signaling (Mahowald, 1992; Boahen, 1997; Mortara, 1997; Kalayjian & Andreou, 1997; Elias, 1993) now form part of the designer’s repertoire. In this article, we review work in our laboratory using neuromorphic analog VLSI techniques to build an interactive, one-dimensional model of the primate oculomotor system. The system consists of two chips: a visual attention-based, tracking chip and an oculomotor control chip. With the continued growth of this system, we hope to explore the system-level consequences of design using neurobiological principles. 1.1 Analog VLSI Approaches for Neural Modeling. The two main arguments for modeling biological systems in analog VLSI are its high speed of information processing and the potential benefits of working within similar design constraints. The human brain contains on the order of 1011 neurons. Although no digital simulation that we know of attempts to simulate the brain of an entire organism (including any species of Nematode),1 eventually such simulations will be desirable. Nearly all neural models are composed of fine-grained parallel-processing arrays, and implementing such models on serial machines usually results in low simulation speeds orders of magnitude away from real time. The investigation of sensorimotor systems is one example situation where the interaction with the real world must either be adequately simulated inside the computer or the simulation must run quickly enough to interface with a physical model. Typically, a sufficiently realistic simulation of the real world is impractical. Spike-based circuit modeling is another example where simulations can be particularly slow since large, fast swings in voltage are frequent. Additionally, mixing widely disparate time scales within the same simulation leads to stiff differential equations, which are notoriously slow to solve (e.g., learning in spiking networks). Provided the analog VLSI circuitry can produce the proper level of detail and is configurable for the types of models under investigation, neuromorphic 1 The most detailed simulation of C. elegans, which has only 302 neurons, involves but 10% of the neurons (Niebur & Erdos, ¨ 1993).
Modeling of the Primate Oculomotor System
245
analog VLSI models deliver the speed desirable for large-scale, real-time simulations. Furthermore, augmenting the system to accommodate more neurobiological detail or expanding the size of the sensory array does not affect the speed of operation. A second intriguing, yet more controversial, argument for the use of VLSI analogs to understand the brain revolves around the claim that certain constraints faced by the analog circuit designer are similar to the constraints faced by nervous systems during evolution and development. When designing analog circuits to operate in the natural world, the circuit designer must operate within a set of powerful constraints: (1) power consumption, when considering mobile systems, is important; (2) the sensory input array must deal with physical signals whose amplitude can vary up to 10 orders of magnitude; (3) component mismatch and noise limit the precision with which individual circuit components process and represent information; (4) since conventional VLSI circuits are essentially restricted to the surface of the silicon chip, there is a large cost associated with dense interconnection of processing units on a single chip. All of these constraints also operate in nervous systems. For instance, the human brain, using 12 to 15 W of power,2 must have evolved under similar pressure to keep the total power consumed to a minimum (Laughlin, van Steveninck, & Anderson, 1998; Sarpeshkar, 1997). Neurons must also solve the problems of mismatch, noise, and dynamic range. The wiring problem for the brain is severe and constrains wiring to relatively sparse interconnection. Although the general computing paradigm at the heart of the digital computer implies that in principle all of these constraints could be implemented in software simulations, in practice they rarely are. The reasons for this are convenience and simulation speed. There are, of course, limitations of analog VLSI design that are not found in the biological substrate (such as the two-dimensional substrate, or the lack of a technology that would allow wires to grow and to extend connections similar to filopodia) and some constraints in the biological substrate that are not found in analog VLSI (such as low-resistance wiring, viability throughout development, and evolutionary back-compatibility). By understanding the similarities and differences between biology and silicon technology and by using them carefully, it is possible to maintain the relevance of these circuits for biological modeling and gain insight into the solutions found by evolution. 1.2 Biological Eye Movements. Primate eye movements represent an excellent set of sensorimotor behaviors to study with analog VLSI for several reasons. From the motor control perspective, the oculomotor system has a relatively simple musculature, and there is extensive knowledge of
2
The same power budget as the Mars Sojourner!
246
Timothy K. Horiuchi and Christof Koch
the neural substrate driving it. Behaviorally, the primate eye shows a diversity of movements involving saccades (quick, reorienting movements), the vestibulo-ocular reflex (an image-stabilizing movement driven by the head velocity–sensitive semicircular canals), the optokinetic reflex (an imagestabilizing movement driven by wide-field image motion), smooth pursuit (a smooth movement for stabilizing subregions of an image), and vergence (binocular movements to foveate a target with both eyes). The required complexity of visual processing ranges from coarse temporal-change detection (for triggering reflexive saccades) to accurate motion extraction from subregions of the visual field (for driving smooth pursuit) to much more sophisticated processes involving memory, attention, and perception. Perhaps the most attractive aspect of eye movements is that the input and output representations have been well explored, and the purpose of eye movements is fairly clear. Although the human eye has a field of view of about 170 degrees, we see best in the central 1 degree, or fovea, where the density of photoreceptors is the greatest. Our eyes are constantly moving to scrutinize objects in the world with the fovea. Since visual acuity rapidly declines if retinal slip exceeds 2 or 3 degrees (Westheimer & McKee, 1975), our smooth eye movements are concerned with stabilizing these images. While the optokinetic reflex (OKR) uses whole-field visual motion to drive image-stabilizing eye movements, smooth pursuit eye movements are characterized by their use of subregions of the field of view. Smooth pursuit allows primates to track small objects accurately even across patterned backgrounds. Interestingly, smooth pursuit eye movements are found only in primates. While the eye movements described above are concerned with image stabilization, much of our visual behavior involves scanning a scene, moving from one part of an image to another. Saccadic eye movements are employed for this purpose, moving the eyes very rapidly to place visual objects onto the fovea. Saccades are rapid, reaching peak velocities of 600 degrees per second in humans, and last between 25 and 200 msec dependent on the saccade amplitude. Although saccades are fast, they have a relatively long latency, requiring between 150 msec and 250 msec from the onset of a visual target to the beginning of the observed movement (Becker, 1989). These are the only conjugate eye movements that humans can generate as voluntary acts (Becker, 1989). We are able to trigger saccades to visual, auditory, memorized, or even imagined targets. 2 A One-Dimensional Oculomotor Plant In this section we describe the construction and performance of our onedimensional oculomotor plant and the architecture of the analog VLSI chip, which controls the motors for both saccadic and smooth pursuit eye movements.
Modeling of the Primate Oculomotor System
247
2.1 The Physical Plant. The primate eye is driven by three sets of muscles: the horizontal, vertical, and oblique muscles. These muscles and other suspensory tissues hold the eye in its socket, producing mechanical dynamics with which the control circuits in the brain stem must contend. Both the muscles and the suspensory tissues are elastic and, in the absence of motorneuron activation, will return the orb to a forward-facing position. Both also provide significant viscosity, producing an over-damped, spring–mass system. While many different models of the oculomotor plant have been proposed to describe the physical dynamics (e.g., Westheimer, 1954; Robinson, 1964), a linear, second-order model of the form: 1 θ(s) = Text (s) (1 + sα1 )(1 + sα2 )
(2.1)
1p 2 m ± m − 4kI , 2I 2I
(2.2)
α1 , α2 =
has been the most widely used. In equation 2.2, k is the spring-constant, m is the damping coefficient, I is the rotational inertia, θ (s) represents the gaze angle, and Text (s) represents the externally applied torque. The measured dominant time constant in the eye has been found to be approximately 250 msec (Robinson, 1964; Keller, 1973). The force-length relationship of the eye muscles was first measured by Collins, O’Meara, and Scott (1975) by recording the agonist muscle tension required to hold the eye at different eye positions. While this relationship was fit well with a parabolic function, the combination of forces from both muscles tends to cancel the nonlinearity, producing a more-or-less linear force position relationship. The oculomotor plant model we have constructed is a 1 degree-of-freedom turntable (see Figure 1) driven by a pair of antagonistically pulling DC (direct current) motors. The DC motors are used to generate torque on the eye by creating tension on the ends of a thread attached at its center to the front of the turntable. The viscoelastic properties of the oculomotor plant are simulated electronically by measuring the angle and angular velocity of the eye and driving the motors to generate the appropriate torques on the eye. This allows the demonstration of the viscoelastic mechanical properties by directly manipulating the mechanical system. Because the biological dynamics are not too far from linear (Collins et al., 1975), the system’s dynamics have been modeled as linear to simplify analysis and construction. In the biological system, the fixation position is determined by the balance point of the agonist muscle tension and the passive elastic forces of all the muscle and suspensory tissues. In the hardware model, however, the two motors are not driven directly against each other (with one motor simulating the active muscle and the other motor simulating the combined elastic forces); rather, the calculated difference in forces is applied to only
248
Timothy K. Horiuchi and Christof Koch
Figure 1: The one-dimensional oculomotor plant. Two DC motors pull on both ends of a drive thread wrapped around the circumference of a small turntable. An analog silicon retina (described in section 3) is mounted vertically on the turntable for reduced rotational inertia. A lens is mounted directly onto the face of the chip to focus an image onto the silicon die. An electronic circuit (mounted inside the box) simulates the mechanical dynamics of the primate oculomotor plant, implementing an overdamped spring-mass system.
one motor. This type of differential drive avoids the increased motor-bearing friction resulting from driving the two motors directly against each other. In addition to the primary motor signals, a small, tonic drive on both motors prevents slack in the thread from building up. 2.2 The Saccadic Burst Generator and Neural Integrator. To drive the oculomotor plant described in the previous section, the brain stem control circuitry must provide the proper signals to overcome both tissue elasticity and viscosity. In order to maintain fixation away from the center position, a sustained pulling force must be generated. Also, to complete an eye movement faster than the eye’s natural time constant, a large, transient, acceleration force must be generated (Robinson, 1973). Both the transient and sustained component signals can be found in brain stem areas that drive the motor-neuron pools (Strassman, Highstein, & McCrea, 1986; Godaux & Cheron, 1996). Accurate balancing of these two component signals is necessary and is observed in the motor-neurons driving the eye muscles. If the transient component is too large, the eye will overshoot the target position, and the eye will drift backward to equilibrium; if the transient is too small, the eye undershoots the target, and the eye drifts onward to equilibrium after the saccade. To generate these oculomotor control signals, we have designed an analog VLSI circuit (see Figure 2), based on models by Jurgens, ¨ Becker, & Ko-
Modeling of the Primate Oculomotor System
249
rnhuber (1981), McKenzie and Lisberger (1986), and Nichols and Sparks (1995), consisting of three main parts: the saccadic burst generator (which converts desired eye displacement to a velocity signal), the neural integrator (which integrates velocity signals to position signals), and the smooth pursuit integrator (which integrates acceleration signals to velocity signals). For saccadic eye movements, only the first two are used. The saccadic burst integrator is used to control the burst duration, and the neural integrator holds a dynamic memory of the current eye position. This model uses initial motor error (the difference in position from the current gaze angle and the desired gaze angle) as the input to the system. Motor error represents the desired saccade vector that is easily derived from the retinal position of target, relative to the fovea. Figure 2 shows the block diagram of the oculomotor control circuitry. The input signal to the burst generator is a voltage, Vin , specifying the amplitude and direction of the saccade. This signal is held constant for the duration of the saccade. The model generates two signals: a transient pulse of spiking activity (see Figure 3A, signal A) and a step (signal B) of spiking activity, are combined as input to the motor units (signal C). A pair of these transientstep signals drives the two motors of the eye. Saccades are coordinated in this model by a pause system (not shown), which inhibits the burst generator until a trigger stimulus is provided. The trigger stimulus also activates a sample-and-hold circuit, which holds Vin constant throughout the saccade. During a saccade, the input, Vin , is continuously compared to the output of the burst integrator, which integrates the burst unit’s spike train. The burst neuron keeps emitting spikes until the difference is zero. This arrangement has the effect of firing a number of spikes proportional to the initial value of motor error, consistent with the behavior of short-lead burst neurons found in the saccade-related areas of the brain stem (Hepp, Henn, Vilis, & Cohen, 1989). In the circuit, the burst integrator is implemented by electrical charge accumulating on a 1.9 pF capacitor. After the burst is over, the burst (or eye displacement) integrator is reset to zero by the “pause” circuitry. This burst of spikes serves to drive the eye rapidly against the viscosity. The burst of activity is also integrated by the neural integrator (converting velocity commands to changes in eye position), which holds the local estimate of the current eye position from which the tonic signal is generated. The neural integrator provides two output spike trains that drive the left and right sustained components of the motor command. The motor units receive inputs from both the saccadic burst units and the neural integrator and compute the sum of these two signals. Figure 3A, signal C shows output data from the burst generator chip, which is qualitatively similar to spike trains seen in the motor neurons of the abducens nucleus of the rhesus monkey (see Figure 3B; King, Lisberger, & Fuchs, 1986). In addition to the saccadic burst generation circuitry, external inputs have been included to allow the smooth pursuit system to drive the eye
250
Timothy K. Horiuchi and Christof Koch
Figure 2: Block diagram of the oculomotor control circuitry. Saccadic eye movements are driven by the burst units to a given displacement determined by the input, Vin , provided by the retinal location of the visual target. Smooth pursuit eye movements are driven by the pursuit integrators, which output eye velocity commands. The pursuit integrators receive a target’s retinal slip as their input and provide an eye velocity signal as their output. Eye velocity signals are then integrated to eye position (neural integrator), thereby maintaining a memory of the current eye position. The motor unit outputs are an appropriately weighted combination of the velocity and position signals, given the dynamics of the oculomotor plant. Signals A, B, and C for a saccadic eye movement are shown in Figure 3A. See the text for an explanation of the burst units.
motors through the common motor output pathway. This input and its use in smooth pursuit will be discussed in section 3.3. 2.3 Performance. By connecting the motor outputs to the oculomotor plant, saccadic eye movements can be generated. For testing purposes, all of the saccadic eye movements in this section were specified by an external signal. In section 3, saccades were guided by stimuli presented to a onedimension visual tracking chip mounted on the eye. Figure 3A shows an example of a saccade with its underlying control signals, and Figure 4A shows an overlay of 20 saccadic trajectories. Figure 4B shows the peak velocity of these 20 saccades as a function of the commanded saccade amplitude. Similar to the peak velocity versus amplitude relation-
Modeling of the Primate Oculomotor System
251
Figure 3: (A) The eye position, burst unit, neural integrator output, and motor unit spike trains during a small saccade. The initial horizontal eye position is off-center, with the neural integrator providing the tonic holding activity. All three spike trains are digital outputs (0 to 5 volts). Only the motor unit output (C) is used externally. The small oscillation seen in the eye position trace is due to the pulse-frequency modulation technique used to drive the eye. Eye position is measured directly using the potentiometer, which doubles as the mechanical bearing for the system. (B) Motor neuron spike train with horizontal (H) and vertical (V) eye position shown during an oblique saccade in a rhesus monkey. (From King et al., 1986)
ship in primate saccades, the peak velocity increases for increasing saccade amplitude and then saturates. As the peak velocity of the saccades saturates, the duration of the saccades begins to increase linearly with amplitude. This saturation is due to the saturating transfer function in the burst unit. These characteristics are qualitatively consistent with primate saccades (Becker, 1989). 2.4 Adaptation of Postsaccadic Drift. Through repeated experience in the world, nearly all animals modify their behavior on the basis of some type of memory. The ability to adapt and learn is not simply an added feature of neural systems; it is a fundamental mechanism that drives the development of the brain and may explain much about the structures and representations that it uses to compute. For this reason, we are beginning to explore the use of adaptation in our oculomotor system. Although there are many different forms of adaptation and learning in the saccadic system, we focus on a particular form that has been implemented in our system. In the generation of saccades, the sustained (or tonic) component of the command determines the final eye position, while the ideal transient (or
252
Timothy K. Horiuchi and Christof Koch
Figure 4: (A) Angular eye position versus time for 20 traces of different saccades triggered from the center position. The input was swept uniformly for different saccade amplitudes from leftward to rightward. The small oscillations in the eye position are due to the discrete pulses used to drive the eye motors; at rest, the pulses are at their lowest frequency and thus most visible. (B) Plot of the peak velocity during each of the saccades on the left. These velocities were computed by performing a least-squares fit to the slope of the center region of the saccade trace. Peak velocities of up to 870 degrees per second have been recorded on this system with different parameter settings than used here.
burst) component should bring the eye to exactly that position by the end of the burst. Mismatch of the burst and tonic components leads to either forward or backward drift following an undershoot or overshoot of the final eye position, known as postsaccadic drift. Studies in both humans and monkeys show that in the case of muscle weakening or nerve damage, which produces systematic undershooting of saccades, postsaccadic drift can be compensated for by adaptive processes that have time constants on the order of 1.5 days (Optican & Robinson, 1980). Ablation studies have shown that control of the burst and tonic gains is independent and that their control depends on different areas of the cerebellum. In addition, retinal slip has been shown to be a necessary and sufficient stimulus to elicit these adaptive changes (Optican & Miles, 1985). To implement the memory structure for the burst gain in our burst generator, we have used a relatively new floating-gate structure that combines nonvolatile memory and computation into a single transistor. Floating-gate structures in VLSI (a metal oxide semiconductor transistor gate completely insulated from the circuitry by silicon dioxide) offer extremely effective analog parameter storage with retention measured in years. Until recently, however, the use of floating gates required the use of either ultraviolet (UV) radiation (Glasser, 1985; Mead, 1989a; Kerns, 1993) or bidirectional tunnel-
Modeling of the Primate Oculomotor System
253
ing processes (Carley, 1989; Lande, Ranjbar, Ismail, & Berg, 1996) to modify the charge on the floating node, and both have significant drawbacks. The recent development of a complementary strategy of tunneling and hot electron injection (Hasler et al., 1995; Diorio, Mahajan, Hasler, Minch, & Mead, 1995) in a commercially available bipolar complementary metal oxide semiconductor process has alleviated some of these difficulties. Adding and removing electrons from the floating gate can be performed at extremely low rates, making it possible to create long training time constants. To demonstrate the ability to reduce postsaccadic drift in our VLSI system, a sensitive direction-selective motion detector chip (Horiuchi & Koch, 1996) was mounted on the one-dimensional eye, and motion information was read from the chip 100 msec after the end of the saccadic burst activity. The burst activity period (lower trace in Figures 5A and 5B) is detected by reading a signal on the burst generator chip representing the suppression of the pause circuitry. A standard leftward saccade amplitude of about 23 degrees was programmed into the burst generator input, and a saccade was repeatedly triggered. The motion sensor was facing a stationary stripe stimulus, which would elicit a motion signal during and after the saccade burst. The direction-of-motion information was summed across the motion detector array, and a simple, fixed-learning-rate algorithm was used to determine whether to increase or decrease the gain. One hundred msec after each trial saccade, the motion detector output current was compared against two threshold values. If the output value was greater than the rightward motion threshold, indicating overshoot, a unit hot-electron injection pulse was issued, which would reduce the floating-gate voltage and thus reduce the burst gain. If the integrated value was less than the leftward motion threshold, indicating undershoot, a unit tunneling pulse was issued, which would increase the floating-gate voltage and thus increase the pulse gain. Figure 5A shows an experiment where the pulse gain was initialized to zero. With the particular learning rates used, eight trials were required before the gain was raised sufficiently to eliminate the postsaccadic drift. Figure 5B shows a similar experiment where the pulse gain was initialized to a large value. In this case, within 41 trials, the pulse gain was lowered sufficiently to eliminate the postsaccadic drift. The learning rates used in this example were arbitrary and can be set over many orders of magnitude. 2.5 Triggering the Saccade. Although the saccadic eye movements presented thus far were manually triggered for testing purposes, the system has also been extensively used with visual input to close the sensorimotor loop. In the initial stages of this project, visual input was provided to the saccadic system in the form of a simplified analog VLSI model of the retinocollicular visual pathway. This enabled the system to trigger orienting saccades to temporally changing visual stimuli (Horiuchi, Bishofberger, & Koch, 1994). This visually driven chip computed the centroid of temporal
254
Timothy K. Horiuchi and Christof Koch
Figure 5: (A) Reduction of saccadic undershoot via on-chip learning. Saccade trajectories demonstrating the reduction of a backward postsaccadic drift by increasing the burst gain via a tunneling process that modifies the charge stored on a nonvolatile floating-gate memory circuit in an unsupervised manner. The circuit converged within eight practice saccades (not all shown). (B) Reduction of saccadic overshoot. Saccade trajectories showing the reduction of an onward postsaccadic drift by decreasing the burst gain via a hot-electron injection process, which modifies the charge on the same memory circuit as above but in the opposite direction. In this case, the performance converged in 41 practice saccades. The lower digital trace in each figure indicates the time of burst unit activity. The arrow indicates the progression from early saccade trials to later saccade trials. The floating-gate technology demonstrated here (Hasler et al., 1995; Diorio et al., 1995) provides us with a versatile, single transistor adaptive synapse.
activity in one dimension and triggered saccades when the sum of all the temporal-derivative signals on the array exceeded a threshold. In other work, we have also triggered saccades to auditory targets using an analog VLSI model of auditory localization based on the barn owl auditory localization system (Horiuchi, 1995). Auditory saccades are interesting because sounds are most easily localized in the head-based coordinate frame, but eye movements (at the level of the saccadic burst generator) are specified in essentially retinotopic coordinates. A coordinate transform that compensates for different starting eye positions must be performed to specify saccades correctly to auditory targets. 3 Smooth Pursuit Eye Movements While the saccadic system provides the primate with an effective alerting and orienting system to place targets in the fovea, in many cases, smooth
Modeling of the Primate Oculomotor System
255
tracking of objects may be desirable to retain the high visual acuity of the fovea. The ability to move the eyes smoothly to stabilize wide-field visual motion is fairly universal, but the ability to select only a portion of the visual field to stabilize is highly developed only in primates. To accomplish this task, some mechanism is needed to define where the object is and from what part of the scene to extract motion information. 3.1 Visual Attention and Eye Movements. A number of studies have revealed the involvement of selective visual attention in the generation of both saccadic (Kowler, Anderson, Dosher, & Blaser, 1995; Hoffman & Subramaniam, 1995; Rafal, Calabresi, Brennan, & Scioltio, 1989; Shimojo, Tanaka, Hikosaka, & Miyauchi, 1995) and smooth pursuit eye movements (Khurana & Kowler, 1987; Ferrera & Lisberger, 1995; Tam & Ono, 1994). Attentional enhancement occurs at the target location just before a saccade, as well as at the target location during smooth pursuit. In the case of saccades, attempts to dissociate attention from the target location disrupt their accuracy and latency (Hoffman & Subramaniam, 1995). It has been proposed that attention is involved in programming the next saccade by highlighting the target location. For smooth pursuit—driven by visual motion in a negative feedback loop (Rashbass, 1961)—spatial attention is thought to be involved in the extraction of the target’s motion. Since the cortical, motion-sensitive, middle temporal area (MT) and the middle superior temporal area (MST) have been strongly implicated in supplying visual motion information for pursuit by anatomical, lesion, and velocity-tuning studies, (see Lisberger, Morris, & Tychsen, 1987, for review) a mechanism to use the activity selectively from the neurons associated with the target at the correct time and place is actively being sought. The modulation of neural activity in these areas, for conditions that differ only by their instructions, has been investigated. Although only small differences in activity have been found in areas MT and MST during the initiation of smooth pursuit toward a target in the presence of a distractor (Ferrera & Lisberger, 1997), strong modulation of activity in MT and MST has been observed during an attentional task to discriminate between target and distractor motions (Treue & Maunsell, 1996). Koch and Ullman (1985) proposed a model of attentional selection based on the output of a single saliency map by combining the activity of elementary feature maps in a topographic manner. The most salient locations are where activity from many different feature maps coincides, or at locations where activity from a preferentially weighted feature map, such as temporal change, occurs. A winner-take-all (WTA) mechanism, acting as the center of the attentional spotlight, selects the location with the highest saliency. While the WTA mechanism captures the idea of attending a single point target, many experiments have demonstrated weighted vector averaging in both saccadic and smooth pursuit behavior (Lisberger & Ferrera, 1997; Groh, Born, & Newsome, 1997; Watamaniuk & Heinen, 1994). Ana-
256
Timothy K. Horiuchi and Christof Koch
log circuits that can account for this diversity of responses (that is, vector averaging, winner-take-all, vector summation) need to be investigated. 3.2 An Attentional Tracking Chip. A number of VLSI-based visual tracking sensors that use a WTA attentional model have been described (Morris & DeWeerth, 1996; Brajovic & Kanade, 1998). Building on the work of Morris and DeWeerth (1996) on modeling selective visual attention, this chip incorporates focal-plane processing to compute image saliency and select a target feature for tracking. The target position and the direction of motion are reported as the target moves across the array, providing control signals for tracking eye movements. The computational goal of the attentional tracking chip is the selection of a single target and the extraction of its retinal position and direction of motion. Figure 6 shows a block diagram of this computation. The first few upper stages of processing compute the saliency map from simple feature maps that drive the WTA-based selection of a target to track. Adaptive photoreceptor circuits (Delbruck, ¨ 1993) (at the top of Figure 6) transduce the incoming pattern of light into an array of voltages. The temporal (TD) and spatial (SD) derivatives are computed from these voltages and used to generate the saliency map and direction of motion. The saliency map is formed by summing the absolute value of each derivative (|TD| + |SD|). The direction-of-motion (DM) circuit computes a normalized product of the two derivatives: TD · SD . |TD| + |SD| Figure 7 shows an example stimulus and the computed features. Only circuits at the WTA-selected location send information off-chip. These signals include the retinal position, the direction of motion, and the type of target being tracked. The saccadic system uses the position information to foveate the target, and the smooth pursuit system uses the motion information to match the speed of the target. The target’s retinal position is reported by the position-to-voltage (P2V) circuit (DeWeerth, 1992) by driving the common position output line to a voltage representing its position in the array. The direction of motion is reported by a steering circuit that puts the local DM circuit’s current onto the common motion-output line. The saccadic triggering (ST) circuit indicates whether the position of the target warrants a recentering saccade based on the distance of the target from the center of the array. This acceptance “window” is externally specified. Figure 8 shows the response of the chip to a swinging edge stimulus. The direction of motion of the target (DM, upper trace) and the position of the target (P2V, lower trace) are displayed. As the target moves across the array, different direction-of-motion circuit outputs are switched onto the common output line. This switching is the primary cause of the noise seen on the motion output trace. At the end of the trace, the target slipped off
Modeling of the Primate Oculomotor System
257
Figure 6: Block diagram of the visual tracking chip. Images are projected directly onto the chip surface’s through a lens, and the output signals are sent to the oculomotor plant described in section 2. P = adaptive photoreceptor circuit, TD = temporal derivative circuit, SD = spatial derivative, DM = direction of motion, HYS WTA = hysteretic winner-take-all, P2V = position to voltage, ST = saccade trigger. The TD and SD signals are summed to form the saliency map from which the HYS WTA finds the maximum. Hysteresis is used locally to improve the tracking of moving targets, as well as to combat noise. The output of the HYS WTA steers both the direction of motion and the SD information onto global output lines. The HYS WTA also drives the P2V and ST circuits to convert the winning position to a voltage and to indicate when the selected pixel is outside a specified window located at the center of the array. The SD input control modulates the relative gain of the positive and negative spatial derivatives used in the saliency map. See the text for details.
258
Timothy K. Horiuchi and Christof Koch
Figure 7: Example stimulus. Traces from top to bottom: Photoreceptor voltage, absolute value of the spatial derivative, absolute value of the temporal derivative, and direction of motion. The stimulus is a high-contrast, expanding bar (shown on the right), which provides two edges moving in opposite directions. The signed, temporal, and spatial derivative signals are used to compute the direction of motion shown in the bottom trace. The three lower traces were current measurements, which shows some clocking noise from the scanners used to obtain the data.
the photoreceptor array, and the winning status shifted to a location with a small background signal. 3.3 Visually Guided Tracking Eye Movements. To demonstrate smooth pursuit behavior in our one-dimensional system, we mounted the attentional tracking chip on the oculomotor system and used its visual processing outputs to drive both smooth pursuit and saccadic eye movements. The motor component of the model used to drive smooth pursuit is based on a leaky integrator model described by McKenzie and Lisberger (1986) using only the target velocity input. Because retinal motion of the target serves as an eye-velocity error, direction-of-motion signals from the tracking chip are used as an eye acceleration command. Visual motion is thus temporally integrated to an eye velocity command and drives the oculomotor plant in parallel with the saccadic burst generator (see Figure 2). To implement the smooth pursuit integrator in this system, off-chip circuits were constructed. The direction-of-motion signal from the tracking chip was split into leftward and rightward motion channels and used as eye acceleration commands, which were integrated to eye velocity com-
Modeling of the Primate Oculomotor System
259
Figure 8: Extracting the target’s position and direction of motion from a swinging target. The WTA output voltage is used to switch the DM current onto a common current-sensing line. The output of this signal, converted to a voltage, is seen in the top trace. The zero-motion level is indicated by the flat line shown at 2.9 volts. The lower trace shows the target’s position from the position-tovoltage encoding circuits. The target’s position and direction of motion are used to drive saccades and smooth pursuit eye movement during tracking. The noise in the upper trace is due to switching transients as WTA circuits switch in the DM currents from different pixel locations.
mands by the pursuit integrators (see Figure 2). The integrator leak time constants were set to about 1 sec. Oscillation in the pursuit velocity around the target velocity (at around 6 Hz) is a common occurrence in primates (upper trace, Figure 10B) and is also seen in our system (but at about 4 Hz). Oscillations in this negative feedback system can occur from delays in visual processing and from large gain in the smooth pursuit integration stage. Our system has very little motion processing delay, but the gain on the integrator is large. Goldreich, Krauzlis, & Lisberger (1992) have shown that in the primate system, visual motion delays appear to be the dominant cause of this oscillation.
260
Timothy K. Horiuchi and Christof Koch
Figure 9: (A) Smooth pursuit and saccadic eye movements of a monkey in response to sinusoidal target motion at approximately 0.27 Hz, peak-to-peak amplitude 20 degrees. The target and eye position traces have been offset for clarity. (From Collewijn & Tamminga, 1984) (B) Smooth pursuit and saccadic eye movements in our VLSI model. A swinging target consisting of a bar with no distractors is tracked over a few cycles. The top trace shows the eye position over time, and the bottom trace shows the eye velocity.
When humans view natural scenes mixed with both stationary and moving objects, saccades and smooth pursuit are combined in an attempt to take in a scene quickly and scrutinize moving objects. How these two eye movements are behaviorally combined is still unclear. When humans or monkeys pursue fast, sinusoidally moving targets, the smooth pursuit eye movement becomes punctuated with catch-up saccades as the target speed exceeds the maximum pursuit speed and a retinal error builds up. While visual motion provides the largest contribution to eye acceleration during the initiation and maintenance of pursuit, the target’s retinal position and acceleration also contribute to driving the eye (Morris & Lisberger, 1985). In our hardware model, however, the pursuit system is driven by only the retinal velocity error. The saccadic system, dedicated to keeping targets near the center of the imager, uses position error to trigger and guide saccades. While these two motor control systems operate essentially independently, the visual target motion induced by the saccade must be suppressed at the input of the smooth pursuit integrator to prevent conflict between the two systems and to maintain the smooth pursuit eye velocity across saccades. Figure 9B exemplifies the integration of saccadic with smooth pursuit eye movements during the tracking of a sinusoidally swinging target. When the velocity of the target exceeds the peak velocity of the pursuit system, the target slips out of the central region of the chip’s field of view, and saccades are triggered to recenter the target. In another experiment, we used a step-ramp stimulus to illustrate the separation of the saccadic and smooth pursuit systems, activating both systems
Modeling of the Primate Oculomotor System
261
Figure 10: (A) Step-ramp experiment with a Macaque monkey. The upward arrow indicates the initialization of pursuit, which precedes the first saccade. (From Lisberger et al., 1987; with permission from Annual Review of Neuroscience, c 10, °1987, by Annual Reviews.) (B) Step-ramp experiment using the model. The target jumps from the fixation point to a new location and begins moving with constant velocity. An artificial delay of 100 msec has been added to simulate the saccadic triggering latency. Only the saccadic trigger is delayed; the target information is current.
at once but in different directions. In primates, visually triggered saccades (not including express saccades) have latencies from 150 to 250 msec; the pursuit system has a shorter latency, from 80 to 130 msec. With this stimulus, the pursuit system begins to move the eye in the direction of target motion before the saccade occurs (Lisberger et al., 1987). Because there is no explicit delay in the current saccadic triggering system, an artificial 100 msec delay was added in the saccadic trigger to mimic this behavior (see Figure 10A). The latency of the model pursuit system without adding additional delays is approximately 50 to 60 msec. Figure 10B shows comparison data from a step-ramp experiment in a macaque monkey. 4 Conclusions The two-chip primate oculomotor system model presented in this article is part of an ongoing exploration into the issues of systems-level neurobiological modeling using neuromorphic analog VLSI. This work focuses on feedback systems that involve sensory and motor interaction with natural environments. Within the analog VLSI framework, it has touched on various examples of sensorimotor control, learning, and the coordination of different eye movements. Saccadic and smooth pursuit eye movements have been integrated in the system, which has raised many questions about
262
Timothy K. Horiuchi and Christof Koch
how to model their interaction. Adaptation of saccadic parameters based on biologically constrained error measures has been demonstrated. The use of a simple WTA model of visual attention for target selection and selective motion extraction has been demonstrated, raising many questions about the interaction between reflexive (collicular) and volitional (cortical) eye movement systems. Our ongoing work seeks to address many of these questions. The main contribution of our work has been the demonstration of a realtime modeling system that brings together many different neural models to solve real-world tasks. To date, there are no other oculomotor modeling systems that use realistic burst generator circuits to drive an analog oculomotor plant with similar dynamics to the biological system. While other research groups have built biologically inspired, visual tracking systems, the problems they encounter are generally not similar to the problems faced by biological systems because they do not solve the task with hardware that has similar properties. By building circuits that compute with the representations of information found in the brain, the modeling system presented here is capable of replicating many of the behavioral, lesion, stimulation, and adaptation experiments performed on the primate oculomotor system. Armed with a continuously growing arsenal of circuits, we will be emulating much larger and more realistic sensorimotor systems in the future. Acknowledgments We thank Brooks Bishofberger for mechanical design of the oculomotor system and fabrication of some of the dynamics simulation electronics, Tobi Delbruck ¨ for advice on photoreceptor circuit layout, Paul Hasler for advice on the floating-gate structures, and Tonia Morris and Steven P. DeWeerth for their assistance and guidance in some important parts of the attentional selection circuits. We also thank Steven Lisberger, Rodney Douglas, and Terry Sejnowski for advice rendered over many years. The research reported here was supported by the Office of Naval Research and the Center for Neuromorphic Systems Engineering as part of the National Science Foundation Engineering Research Center Program. References Becker, W. (1989). Metrics. In R. H. Wurtz & M. E. Goldberg (Eds.), The neurobiology of saccadic eye movements (pp. 13–67). Amsterdam: Elsevier. Boahen, K. (1997). The retinomorphic approach: Pixel-parallel adaptive amplification, filtering and quantization. Analog Integ. Circ. Sig. Proc., 13, 53–68. Brajaovic, V., & Kanade, T. (1998). Computational sensor for visual tracking with attention. IEEE J. of Solid-State Circuits, 33:8, 1199–1207. Carley, L. R. (1989). Trimming analog circuits using floating-gate analog MOS memory. IEEE J. Solid State Circ., 24, 1569–1575. Collewijn, H., & Tamminga, E. P. (1984). Human smooth and saccadic eye movements during voluntary pursuit of different target motions on different back-
Modeling of the Primate Oculomotor System
263
grounds. J. Physiol., 351, 217–250. Collins, C. C., O’Meara, D., & Scott, A. B. (1975). Muscle tension during unrestrained human eye movements. J. Physiol., 245, 351–369. Delbruck, ¨ T. (1993). Investigations of analog VLSI visual transduction and motion processing. Ph.D. Thesis, Computation and Neural Systems program, California Institute of Technology. DeWeerth, S. P. (1992). Analog VLSI circuits for stimulus localization and centroid computation. Intl. J. Comp. Vis., 8, 191–202. Diorio, C., Hasler, P., Minch, B., & Mead, C. (1997). A complementary pair of four-terminal silicon synapses. Analog Integ. Circ. Sig. Proc., 13, 153–166. Diorio, C., Mahajan, S., Hasler, P., Minch, B., & Mead, C. (1995). A high-resolution non-volatile analog memory cell. In Proc. of the Intl. Symp. on Circuits and Systems (pp. 2233–2236). Seattle, WA. Douglas, R., Mahowald, M., & Mead, C. (1995). Neuromorphic analogue VLSI. In W. M. Cowan, E. M. Shooter, C. F. Stevens, & R. F. Thompson (Eds.), Annual reviews in neuroscience (Vol. 18, pp. 255–281). Palo Alto, CA: Annual Reviews. Elias, J. G. (1993). Artificial dendritic trees. Neural Computation, 5, 648–664. Ferrera, V., & Lisberger, S. (1995). Attention and target selection for smooth pursuit eye movements. J. Neurosci., 15, 7472–7484. Ferrera, V., & Lisberger, S. (1997). The effect of a moving distractor on the initiation of smooth-pursuit eye movements. Visual Neuroscience, 14, 323–338. Fukushima, K., Yamaguchi, Y., Yasuda, M., & Nagata, S. (1970). An electronic model of the retina. Proc. of the IEEE, 58, 1950–1951. Glasser, L. A. (1985). A UV write-enabled PROM. In H. Fuchs (Ed.), Chapel Hill Conference on VLSI (1985) (pp. 61–65). Rockville, MD: Computer Science Press. Godaux, E., & Cheron, G. (1996). The hypothesis of the uniqueness of the oculomotor neural integrator—Direct experimental evidence in the cat. J. Physiology London, 492, 517–527. Goldreich, D., Krauzlis, R. J., & Lisberger, S. G. (1992). Effect of changing feedback delay on spontaneous oscillations in smooth pursuit eye movements of monkeys. J. Neurophysiol., 67, 625–638. Groh, J. M., Born, R. T., & Newsome, W. T. (1997). How is a sensory map read out? Effects of microstimulation in visual area MT on saccades and smooth pursuit eye movements. J. Neurosci., 17, 4312–4330. Hasler, P., Diorio, C., Minch, B. A., & Mead, C. (1995). Single transistor learning synapses. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 817–824). Cambridge, MA: MIT Press. Hepp, K., Henn, V., Vilis, T., & Cohen, B. (1989). Brainstem regions related to saccade generation. In R. H. Wurtz & M. E. Goldberg (Eds.), The neurobiology of saccadic eye movements (pp. 105–212). Amsterdam: Elsevier. Hoffman, J., & Subramaniam, B. (1995). The role of visual attention in saccadic eye movements. Perception and Psychophysics, 57, 787–795. Horiuchi, T. (1995). An auditory localization and coordinate transform chip. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems 7 (pp. 787–794). Cambridge, MA: MIT Press. Horiuchi, T., Bishofberger, B., & Koch, C. (1994). An analog VLSI saccadic system. In J. D. Cowan, G. Tesauro, and J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 582–589). San Mateo, CA: Morgan Kaufmann.
264
Timothy K. Horiuchi and Christof Koch
Horiuchi, T. K., & Koch, C. (1996). Analog VLSI circuits for visual motion-based adaptation of post-saccadic drift. In Proc. 5th Intl. Conf. on Microelectronics for Neural Networks and Fuzzy Systems—MicroNeuro96 (pp. 60–66). Los Alamitos, CA: IEEE Computer Society Press. Jurgens, ¨ R., Becker, W., & Kornhuber, H. H. (1981). Natural and drug-induced variations of velocity and duration of human saccadic eye movements: Evidence for a control of the neural pulse generator by local feedback. Biol. Cybern., 39, 87–96. Kalayjian, Z., & Andreou, A. (1997). Asynchronous communication of 2D motion information using winner-take-all arbitration. Analog Integ. Circ. Sig. Proc., 13, 103–109. Keller, E. L. (1973). Accommodative vergence in the alert monkey: Motor unit analysis. Vision Res., 13, 1565–1575. Kerns, D. A. (1993). Experiments in very large-scale analog computation. Unpublished doctoral dissertation, California Institute of Technology. Khurana, B., & Kowler, E. (1987). Shared attentional control of smooth eye movement and perception. Vision Research, 27, 1603–1618. King, W. M., Lisberger, S. G., & Fuchs, A. F. (1986). Oblique saccadic eye movements. J. Neurophysiol., 56, 769–784. Koch, C. (1989). Seeing chips: Analog VLSI circuits for computer vision. Neural Computation, 1, 184–200. Koch, C., & Ullman, S. (1985). Shifts in selective visual attention: Towards the underlying neural circuitry. Human Neurobiology, 4, 219–227. Kowler, E., Anderson, E., Dosher, B., & Blaser, E. (1995). The role of attention in the programming of saccades. Vision Research, 35, 1897–1916. Lande, T. S., Ranjbar, H., Ismail, M., & Berg, Y. (1996). An analog floating-gate memory in a standard digital technology. In Proc. 5th Intl. Conf. on Microelectronics for Neural Networks and Fuzzy Systems—MicroNeuro96 (pp. 271–276). Los Alamitos, CA: IEEE Computer Society Press. Laughlin, S., van Steveninck, R. R. d., & Anderson, J. C. (1998). The metabolic cost of neural information. Nature Neurosci., 1, 36–41. Lisberger, S. G., & Ferrera, V. P. (1997). Vector averaging for smooth pursuit eye movements initiated by two moving targets in monkeys. J. Neurosci, 17, 7490–7502. Lisberger, S. G., Morris, E. J., & Tychsen, L. (1987). Visual motion processing and sensory-motor integration for smooth pursuit eye movements. In W. M. Cowan, E. M. Shooter, C. F. Stevens, & R. F. Thompson (Eds.), Annual reviews in neuroscience, vol. 10 (pp. 97–129). Palo Alto, CA: Annual Reviews. Mahowald, M. A. (1992). VLSI analogs of neuronal visual processing: A synthesis of form and function. Ph.D. Dissertation. Computer Science, California Insitute of Technology. McKenzie, A., & Lisberger, S. G. (1986). Properties of signals that determine the amplitude and direction of saccadic eye movements in monkeys. J. Neurophysiol., 56, 196–207. Mead, C. (1989a). Adaptive retina. In C. Mead & M. Ismail (Eds.), Analog VLSI implementation of neural systems (pp. 239–246). Boston: Kluwer. Mead, C. (1989b). Analog VLSI and neural systems. Menlo Park, CA: AddisonWesley.
Modeling of the Primate Oculomotor System
265
Mead, C. (1990). Neuromorphic electronic systems. Proc. IEEE, 78, 1629–1636. Morris, E. J., & Lisberger, S. G. (1985). A computer model that predicts monkey smooth pursuit eye movements on a millisecond timescale. Soc. Neurosci. Abstr., 11, 79. Morris, T. G., & DeWeerth, S. P. (1996). Analog VLSI circuits for covert attentional shifts. In Proc. 5th Intl. Conf. on Microelectronics for Neural Networks and Fuzzy Systems—MicroNeuro96 (pp. 30–37). Los Alamitos, CA: IEEE Computer Society Press. Mortara, A. (1997). A pulsed communication/computation framework for analog VLSI perceptive systems. Analog Integ. Circ. Sig. Proc., 13, 93–101. Nichols, M. J., & Sparks, D. L. (1995). Non-stationary properties of the saccadic system—New constraints on models of saccadic control. J. Neurophysiol., 73(1), 431–435. Niebur, E., & Erdos, ¨ P. (1993). Theory of the locomotion of nematodes: Control of the somatic motor neurons by interneurons. Mathematical Biosciences, 118, 51–82. Optican, L. M., & Miles, F. A. (1985). Visually induced adaptive changes in primate saccadic oculomotor control signals. J. Neurophysiol., 54, 940–958. Optican, L. M., & Robinson, D. A. (1980). Cerebellar-dependent adaptive control of the primate saccadic system. J. Neurophysiol., 44, 1058–1076. Rafal, R., Calabresi, P., Brennan, C., & Scioltio, T. (1989). Saccade preparation inhibits reorienting to recently attended locations. J. Exp. Psych: Hum. Percep. Perf., 15, 673–685. Rashbass, C. (1961). The relationship between saccadic and smooth tracking eye movements. J. Physiol., 159, 326–338. Robinson, D. (1973). Models of the saccadic eye movement control system. Kybernetik, 14, 71–83. Sarpeshkar, R. (1997). Efficient precise computation with noisy components: Extrapolating from electronics to neurobiology. Unpublished manuscript. Shimojo, S., Tanaka, Y., Hikosaka, O., & Miyauchi, S. (1995). Vision, attention, and action—Inhibition and facilitation in sensory motor links revealed by the reaction time and the line-motion. In T. Inui & J. L. McClelland (Eds.), Attention and performance XVI. Cambridge, MA: MIT Press. Strassman, A., Highstein, S. M., & McCrea, R. A. (1986). Anatomy and physiology of saccadic burst neurons in the alert squirrel monkey. I. Excitatory burst neurons. J. Comp. Neur., 249, 337–357. Tam, W. J., & Ono, H. (1994). Fixation disengagement and eye-movement latency. Perception and Psychophysics, 56, 251–260. Treue, S., & Maunsell, J. (1996). Attentional modulation of visual motion processing in cortical areas MT and MST. Nature, 382, 539–541. Watamaniuk, S. N. J., & Heinen, S. J. (1994). Smooth pursuit eye movements to dynamic random-dot stimuli. Soc. Neuroscience Abstr., 20, 317. Westheimer, G. (1954). Mechanism of saccadic eye movements. Arch. Ophthamol., 52, 710. Westheimer, G. A., & McKee, S. (1975). Visual acuity in the presence of retinal image motion. J. Opt. Soc. Am, 65, 847–850. Received December 1, 1997; accepted June 15, 1998.
LETTER
Communicated by Thomas Wachtler
JPEG Quality Transcoding Using Neural Networks Trained with a Perceptual Error Measure John Lazzaro John Wawrzynek Computer Science Division, University of California at Berkeley, Berkeley, CA 947201776, U.S.A.
A JPEG Quality Transcoder (JQT) converts a JPEG image file that was encoded with low image quality to a larger JPEG image file with reduced visual artifacts, without access to the original uncompressed image. In this article, we describe technology for JQT design that takes a pattern recognition approach to the problem, using a database of images to train statistical models of the artifacts introduced through JPEG compression. In the training procedure for these models, we use a model of human visual perception as an error measure. Our current prototype system removes 32.2% of the artifacts introduced by moderate compression, as measured on an independent test database of linearly coded images using a perceptual error metric. This improvement results in an average PSNR reduction of 0.634 dB. 1 Introduction JPEG is a lossy compression algorithm for digital images (Wallace, 1992). An image file format that uses JPEG compression, JFIF, has become the standard image file format for the World Wide Web and for digital cameras. The JPEG encoding algorithm gives users direct control over the compression process, supporting trade-offs between image quality and degree of compression. Higher compression ratios may result in undesirable visual artifacts in the decoded image. Given a JPEG-encoded image that was compressed to a small size at the expense of visual quality, how can we reduce visual artifacts in the decoded image? A substantial body of literature addresses this question (Wu & Gersho, 1992; Jarske, Haavisto, & Defee, 1994; Ahumada & Horng, 1994; Minami & Zakhor, 1995; Yang, Galasysanos, & Katsaggelos, 1995; O’Rourke & Stevenson, 1995). In these references, artifact reduction is undertaken as part of a JPEG decoder. In this article, we consider image artifact reduction as part of a different application: a JPEG Quality Transcoder (JQT). A JQT converts a JPEG image file that was encoded with low image quality to a larger JPEG image file with reduced visual artifacts, without access to the original uncompressed Neural Computation 11, 267–296 (1999)
c 1999 Massachusetts Institute of Technology °
268
John Lazzaro and John Wawrzynek
image. A JQT should perform only the lossless part of the JPEG decoding algorithm, followed by signal processing on the partially decompressed representation, followed by lossless JPEG encoding to produce the transcoded image. A JQT provides a simple way to improve image quality in situations where modifying the JPEG encoding or decoding operations is not possible. Applications of a JQT include enhancing the quality of JPEG images accessed from an Internet proxy server, reducing artifacts of video streamed from a motion-JPEG server, and improving the “number of stored photos” versus “image quality” trade-off of digital cameras. In contrast to most previous work in artifact reduction, we take a pattern recognition approach, using a database of images to train statistical models of artifacts. In the training procedure for these models, we use a model of human visual perception as an error measure. The article is organized as follows. In section 2, we review the JPEG compression system. Section 3 introduces the general architecture of the JQT. Section 4 describes the human visual system error measure. Section 5 explains the detailed architecture of our statistical artifact models. Section 6 details the training of the models. Sections 7 and 8 show data from a JQT using these models. Section 9 offers suggestions for further research. 2 JPEG and JFIF This section reviews JPEG compression and the JFIF file format (Wallace, 1992). Cathode ray tube (CRT) color computer display hardware has a natural color representation, RGB, consisting of three numbers that code the linear excitation intensity of red (R), green (G), and blue (B) phosphors at each pixel. High-quality display hardware uses an 8-bit value to encode each color plane (R, G, and B) for a 24-bit pixel encoding. The JPEG algorithm compresses each color plane of an image independently. Since cross-plane correlations cannot be captured in such a scheme, color representations with low correlation between planes are a good match to JPEG. The RGB coding has relatively high correlation between planes. A color scheme with lower cross-plane correlation, YCb Cr , is the color code for the JFIF file format. The YCb Cr code includes the luminance plane Y, which codes a monochrome version of the image in 8 bits. The 8-bit Cb and Cr planes code chrominance information. A linear transformation converts between RGB and YCb Cr coding. Following Fuhrman, Baro, and Cox (1995) we use a linear encoding of YCr Cb in this paper; an alternatie nonlinear encoding of YCr Cb (notated Y0 Cr0 Cb0 ) is more widely used in commercial JPEG applications. In addition to lower cross-plane correlation, the YCb Cr color code has another advantage for image compression. The human visual system is less sensitive to high spatial frequency energy in the chrominance planes of an image, relative to the luminance plane. To exploit this phenomenon, the JFIF file encoding process begins by subsampling the Cb and Cr planes of a
JPEG Quality Transcoding
269
YCb Cr image by a factor of two in both horizontal and vertical dimensions. This subsampling yields an immediate compression of nearly 60% with little degradation in image quality. After color transformation to YCb Cr and chrominance subsampling, JFIF file encoding continues by applying the JPEG encoding algorithm to each plane separately. This encoding begins by dividing the image plane into a grid of nonoverlapping blocks of 8 × 8 pixels; each block is coded independently. Encoding begins by taking the two-dimensional discrete cosine transform (DCT) on each pixel block P, yielding an 8×8 block of coefficients K, defined as k(u, v) =
7 7 X C(u)C(v) X p(x, y) 4 x=0 y=0
× cos((2x + 1)u(π/16)) cos((2y + 1)v(π/16)), (2.1) √ where u = 0, . . . , 7 and v = 0, . . . , 7. The term C(i) = 1/ 2 if i = 0, C(i) = 1 elsewise. In this equation, p(x, y) is the value at position (x, y) in the pixel block P, and k(u, v) is the value for frequency (u, v) in the coefficient block K. Coefficient k(0, 0) codes the DC energy in the block; other coefficients k(u, v) are AC coefficients, coding spatial frequency energy. Eleven-bit k(u, v) values are needed to code 8-bit p(x, y) values accurately. Most of the energy in real-world images lies in the lower spatial frequency coefficients. In addition, the sensitivity limits of the human visual system vary with spatial frequency. Careful quantization of k(u, v) values can exploit these two phenomena, yielding a considerable reduction in the bit size of a coefficient block while maintaining good image quality. Coefficient quantization is the sole lossy step in the JPEG encoding algorithm. Each coefficient k(u, v) is divided by the quantization divisor q(u, v); the dividend is rounded to the nearest integer, yielding scaled quantized coefficients. In baseline JPEG encoding, each plane of an image uses a single matrix Q of q(u, v) values to quantize all blocks in the plane. The JPEG encoding process concludes by lossless compression of the scaled quantized coefficients, yielding a bit-packed JFIF file that contains coefficient information for each block of each plane and the quantization matrix for each plane. JFIF file decoding begins with lossless decompression of the coefficient blocks and quantization matrices for each plane. For each coefficient block, each scaled quantized coefficient is multiplied by the appropriate quantiˆ v). The pixel zation divisor q(u, v), producing the quantized coefficient k(u, block is then reconstructed from the quantized coefficient block, via the Inverse DCT: 7 7 X 1X ˆ v) ˆ y) = C(u)C(v)k(u, p(x, 4 u=0 v=0 × cos((2x + 1)u(π/16)) cos((2y + 1)v(π/16)).
(2.2)
270
John Lazzaro and John Wawrzynek
In this way, a complete image for each color plane is reconstructed block by block. Replication of the subsampled Cb and Cr planes, and conversion from YCb Cr to RGB, complete the decoding process. 3 A JPEG Quality Transcoder Using the definitions of the last section, we now review artifact reduction algorithms for JPEG compression. If the reconstucted image is perceptually different from the original image, visual artifacts have been introduced during coefficient quantization. Published methods for artifact reduction as part of the JPEG decoding process use a combination of these methods to improve image quality: • Linear or nonlinear image processing on the reconstructed image, to lessen the visual impact of artifacts (Minami & Zakhor, 1995; Jarske et al., 1994). • Replacing the decoding algorithm as described in equation 2.1 with an iterative (Yang et al., 1995; O’Rourke & Stevenson, 1995) or codebook (Wu & Gersho, 1992) approach. • Preprocessing the quantized coefficients before proceeding to JPEG decoding as defined in equation 2.2 (Ahumada & Horng, 1994; Minami & Zakhor, 1995). The final method is the preferred approach for implementing artifact reduction in a JPEG quality transcoder; the first two methods would require the large overhead of decoding and reencoding the image. Previous work in preprocessing quantized coefficients for artifact reduction (Minami & Zakhor, 1995; Ahumada & Horng, 1994) has used the tools of iterative optimization. In this work, metrics were developed that measure the severity of a class of JPEG artifacts. These metrics are then used in an iterative optimization algorithm to calculate coefficient values that minimize image artifacts. In this article we pursue a different approach for preprocessing quantized coefficients for artifact reduction. The approach rests on the assumption that the information lost during quantization of coefficient k(u, v) in color plane C of block (i, j) of an image, expressed as kijC (u, v)− kˆijC (u, v), can be accurately estimated from other information in the compressed image. We use multilayer perceptrons to estimate kijC (u, v) − kˆijC (u, v). These networks are convolutional in input structure: the same network is used for each (i, j) block in an image, and inputs to the network are selected from the coefficient blocks of all three color planes in the neighborhood of block (i, j). Quantization divisor matrices are also used in the estimation process.
JPEG Quality Transcoding
271
We use 64 neural networks, each specialized in architecture (number of hidden units, selection of inputs, etc.) for a particular spatial frequency (u, v). Each network has three outputs—one for each color plane. We detail the network architecture and training procedure in sections 5 and 6. A key part of the training procedure is the computation of the error, as perceived by a human observer, between corresponding color pixels in an original image and a reconstructed image. In the next section, we review the literature of perceptual visual measures and present the perceptual error measure. 4 A Perceptual Error Metric In this section, we describe a pointwise perceptual metric, computed on a ˆ Cˆ b , Cˆ r ) pixel (Y, Cb , Cr ) in an original image and the corresponding pixel (Y, in a reconstructed image. In Appendix B, we present the exact formulation of the metric. Our goal is to develop a metric that is a good predictor for human sensitivity to the types of color imaging errors introduced in JPEG encoding. A recent article (Fuhrmann, Baro, & Cox, 1995) also addresses this issue, in the context of monochrome imaging (Y plane only). The article describes a set of psychophysical experiments that measures the threshold and suprathreshold sensitivity of subjects to JPEG-induced errors. The data from these experiments are compared with the predictions of a large collection of image ˆ 2 ) is shown not to metrics. While mean squared error (defined as |Y − Y| be a good predictor of human performance, distortion contrast (defined as ˆ |Y − Y|/(Y + Yˆ + C)) is highly predictive. We cannot use distortion contrast directly as our training metric because our task involves the measurement of error in color images. A good extension of monochrome contrast that has a firm basis in color science is the cone contrast metric (Cole, Hine, & McIlhagga, 1993). This metric is computed in the LMS color coordinate space. As the RGB color space is the coordinate system derived from the spectral sensitivity function of CRT screen phosphors, the LMS color space is the coordinate system derived from the spectral sensitivity of photopigments of the long-wavelength sensitive (L), mediumwavelength sensitive (M), and short-wavelength sensitive (S) cones in the human retina. A simple linear transformation, shown in appendix A, converts (Y, Cb , Cr ) ˆ To compute the cone ˆ M, ˆ S). ˆ Cˆ b , Cˆ r ) pixel values to (L, M, S) and (L, and (Y, contrast vector (1L/L, 1M/M, 1S/S) from the original pixel and reconstructed pixel LMS values, we use the following equations: 1L/L = 1M/M =
L − Lˆ L + Lo ˆ M−M M + Mo
272
John Lazzaro and John Wawrzynek
1S/S =
S − Sˆ . S + So
The constants Lo , Mo , and Co model a limitation of CRT displays: a pixel position that is programmed to produce the color black actually emits a dim gray color (Macintyre & Cowan, 1992). The constants Lo , Mo , and Co represent this gray in LMS space. Cone contrast space has an interesting psychophysical property (Cole et al., 1993) revealed by the experiment of briefly flashing a slightly off-white ˆ on the white background (L, M, S) and measuring the detecˆ M, ˆ S) color (L, tion threshold of the off-white color, for many different off-white shades. The detection threshold can be shown to be the result of three independent mechanisms, and each mechanism can be expressed as a linear weighting of the cone contrast representation (1L/L, 1M/M, 1S/S). These mechanisms correspond to the familiar opponent channels of red-green (RG), blue-yellow (BY), and black-white (BW). In our metric, we compute these three opponent channel values from the cone contrast vector. The BW channel is qualitatively similar to the distortion contrast metric in Fuhrmann et al. (1995). The other two channels (RG and BY) code chrominance information in a contrast framework. The opponent coding is a suitable representation for incorporating the effects of visual masking into our metric. Visual masking is the phenomenon of errors being less noticeable around an image edge and more noticeable around the smooth parts of an image. In our metric, we model only masking in the luminance plane. We weight the BW output by an activity function A(x, y) that is unity for pixel positions in the smooth regions of original image and less than unity for pixel positions near an edge (Kim, Lee, Eung, & Yeong, 1996). The activity function can be computed once for each pixel of each image in the database and reused to calculate the error of different reconstructions of a pixels. To complete our metric, we sum the weighted absolute values of the opˆ Cˆ b , ponent channel outputs, yielding the final error function E(Y, Cb , Cr ; Y, Cˆ r ). The weights correspond to the relative detection sensitivities of the underlying mechanisms, as measured in the Cole et al. (1993) study. Our use of the absolute values of the opponent channel outputs, rather than the square of these outputs, reflects the assumed independence of these mechanisms. The metric presented in van der Branden Lambrecht and Farrel (1996) shares several details with our work, including opponent channels and a masking model. Major differences include our use of cone contrast space to compute the opponent channels and our formulation of the model to be efficient in a neural network training loop. A more detailed human visual system model has been successfully applied to JPEG image coding in Westen et al. (1996).
JPEG Quality Transcoding
273
5 Network Architecture We use the perceptual metric described in the last section to train statistical models of the information lost during JPEG encoding. In this section, we describe these models in detail. Our system has 64 neural networks, each dedicated to modeling the information loss for a coefficient frequency (u, v). Figure 1 shows a typical network. The network has three outputs, OY (u, v), OCr (u, v), and OCb (u, v), that predict a normalized estimate of kijC (u, v) − kˆijC (u, v) for the three color planes for block position (i, j). These output neurons, as well as all hidden units, use the hyperbolic tangent function as the sigmoidal nonlinearity (output range, −1 to +1). In our implementation, network weights are stored as floating-point values, and network outputs are computed using floatingpoint math. The network outputs OY (u, v), OCr (u, v), OCb (u, v) predict the information lost during JPEG encoding. We use these outputs to compute coefficient values k˜C (u, v) with reduced artifacts, using the equation: k˜C (u, v) = kˆC (u, v) + 0.5qC (u, v)OC .
(5.1)
This approach ensures that only plausible predictions are made by the system. Recall that during JPEG encoding, each DCT coefficient k(u, v) is divided by the quantization divisor q(u, v) and rounded to the nearest integer. During decoding, integer multiplication by q(u, v) produces the quantized ˆ v). Note that this k(u, ˆ v) could have been produced by k(u, v) coefficient k(u, ˆ v) ± 0.5q(u, v). Equation 5.3 produces k˜C (u, v) values in the range of k(u, values only in this range. In this article, we model the information loss produced by a single set of quantization divisors: the quantization divisor tables recommended in section K.1 of CCITT Rec T.81 Standards Document that defines JPEG. This restriction simplifies the artifact reduction system by eliminating the need to include quantization divisor inputs in the neural networks. These quantization divisors, which we refer to as Qs here, produce good compression ratios with moderate visual artifacts and have been adopted by many popular applications. The output neurons in Figure 1 receive inputs from a pool of hidden-layer units. Each hidden-layer unit receives a set of coefficient inputs, selected from the coefficient blocks of one color plane in the neighborhood of block (i, j). We replicate the chrominance coefficient blocks to match the sampling pitch of the luminance block, to simplify the network input architecture. Each coefficient input is divided by its variance, as computed on the training set. Each hidden-layer and output unit has a bias input, not shown in Figure 1. In Figure 1, the chosen inputs are drawn in black on the coefficient block
274
John Lazzaro and John Wawrzynek
Y(u,v)
Cb(u,v)
Cr(u,v)
Cr
Cb
Y
Figure 1: A typical neural network for modeling information loss of coefficient (u, v). Outputs marked C(u, v) (and notated OC (u, v) in the text) predict a normalized estimate of kijC (u, v)− kˆ ijC (u, v) and receive input from all 12 hidden units. Hidden units specialize on a block edge for a single color plane. See the caption for Figure 2 for an explanation of the notation used to denote graphically hidden unit receptive fields. The network as drawn corresponds to architecture B in Figure 2.
grids. The receptive fields are drawn to be correct for coefficient 8 (u = 1, v = 2) as labeled in Figure 2b. Note that the receptive fields all include this coefficient. We handcrafted these receptive fields, guided by pilot training experiments. We believe the following ideas underlie the good performance of these receptive fields:
JPEG Quality Transcoding
275
ABCD 313
313 (a)
31 3
31 3 0 1 2 3 4 5 6 7 u 0 0 1 5 6 14152728 1 2 4 7 13 16 2629 42 2 3 8 12 17 25 30 41 43 9 11 18 24 31 40 44 53 v 34 10 19 23 32 39 45 52 54 5 20 22 33 38 46 51 55 60 6 21 3437 4750 5659 61 7 3536484957586263
(b)
Figure 2: (a) The four neural network architectures (A–D) used in the system. For each architecture, column entries note the number of hidden units of each receptive field type in the network. Column drawings show the four types of receptive fields for hidden units. These drawings show five adjacent coefficient blocks for a single color plane—(i, j), (i ± 1, j) and (i, j ± 1)—drawn as a cross. The selected inputs are drawn in black. The receptive fields are drawn to be correct for coefficient 8 (u = 1, v = 2) as labeled in Figure 2b. Note that the horizontal and vertical receptive fields pass through this coefficient. (b) The zig-zag coefficient numbering convention.
276
John Lazzaro and John Wawrzynek
• A brute force receptive field pattern would include all 64 coefficients for the center coefficient block and the four neighboring blocks, for a total of 320 coefficients. Experiments using these types of receptive fields yielded poor results. Most inputs were irrelevant for constructing a useful feature for the task, and the presence of these useless inputs confused the learning process. It was necessary to preselect a small subset of inputs from the universe of 320 that carried information for a certain class of features. • A natural way to divide hidden-unit space is to let hidden units specialize in artifacts occurring on one edge of the center coefficient block. These hidden units would receive inputs only from the center block and one adjacent block, paring the original universe of 320 potential inputs down to 128 inputs. All the receptive fields shown in Figure 1 have this characteristic. • Experiments using these specialized hidden units suggested that hidden units specializing in horizontal artifacts (the left and right edges of the block) can combine information over the full range of horizontal spatial frequency coefficients but have difficulty combining information over vertical spatial frequencies. A bar-shaped receptive field exploits this observation. We found that for a horizontally specialized hidden unit for coefficient (u, v) a horizontal bar centered on v produced the best results. The receptive fields in Figure 1 show this pattern. We used four different variants of the general architecture shown in Figure 1 in our work. Two variants differ in the number of copies of each of the 12 hidden units. We found that lower-frequency coefficients needed three copies of each hidden unit to model the visual artifacts best; conversely, higher-frequency coefficients sometimes worked best with a single copy of each hidden unit. These variants correspond to A and B in Figure 2a. The other two variants are used only for coefficients (0, u 6= 0) and (v 6= 0, 0). These coefficients have energy only in one spatial frequency axis (horizontal or vertical). For some of these coefficients, the presence of hidden units specialized for the opposite spatial frequency axis results in degraded performance. With these coefficients, we use neural networks with only hidden units that specialize in the preferred axis; three copies of each hidden unit are used in these networks. These variants correspond to C and D in Figure 2a. 6 Network Training We use backpropagation (Rumelhart, Hinton, & Williams, 1986) to train the networks. We train each of the 64 neural networks independently. Intuitively, one would expect simultaneous training of all 64 networks to pro-
JPEG Quality Transcoding
277
duce better performance. However, in pilot experiments using simultaneous training, we were not able to achieve good results. In this section, we describe the independent training method we use and offer reasons that we believe it works well in this application. To train the neural network for coefficient (u, v), we proceed on a perblock basis. We begin by JPEG encoding the three planes of pixel block in an original image; subsampled pixel blocks are used to encode Cb and Cr planes. We then compute the neural network outputs OY (u, v), OCr (u, v), OCb (u, v) for coefficient (u, v). ˜ y) in this block, under the asNext, we compute a reconstructed pixel p(x, sumption that only coefficient (u, v) has been quantized. This reconstruction can be computed efficiently using the equation p˜C (x, y) = pC (x, y) + Wxyuv (kˆC (u, v) − kC (u, v) + 0.5qC (u, v)OC (u, v)),
(6.1)
where Wxyuv is the appropriate DCT coefficient for the pixel (x, y) and the coefficient (u, v). Note that due to chrominance subsampling, the p˜Cb (x, y) and p˜Cr (x, y) values are on a coarser (x, y) grid than the p˜Y (x, y). Pixel replication of the chrominance planes is necessary to produce registered YCr Cb pixel values for the perceptual error calculation. We measure the perceptual error of this reconstructed pixel and update the weights of the neural network for coefficient (u, v) based on the error value. We repeat this reconstruct-measure-update loop for each of the 64 pixels in the block, to complete a training cycle for a block of an image. Note that we compute pC (x, y) as a floating-point number and retain floatingpoint precision for the perceptual error calculation. By training the 64 neural networks independently, we provide the system with a simple problem to solve: cancelling the effect of a single coefficient quantization, in isolation from other coefficient quantizations, and without the round-off noise of a complete inverse DCT computation. In addition, this approach offers 64-way parallelism for neural network training and allows the incremental improvement of the artifact reduction system by upgrading a few of the 64 neural networks without requiring retraining of the rest. Our image database consists of 699 color images, with an average dimension of 451 × 438 pixels. We collected these images from Internet archives. These images include natural scenes, face close-ups, and computer graphics images and have not undergone previous lossy compression or subsampling. We divide the database into three parts: a training set of 347 images, a cross-validation set of 176 images, and a final test set of 176 images. We built our training software on top of the public domain PVRG JPEG codec. To train one of the 64 neural networks, we used two different methods of choosing the quantization divisors for the training images. The first method (“constant divisor”) uses fixed divisor tables. We use the Qs divisor tables scaled by 1.5, to exaggerate the artifacts and simplify the learning task. Pilot
278
John Lazzaro and John Wawrzynek
experiments showed that networks trained using scaled table worked better on the artifacts induced by unscaled Qs compression than networks trained with the unscaled Qs tables. However, there is a large variance in the measured perceptual error over the training set database compressed with a fixed divisor table. We found that for many coefficients, training the neural network with images quantized using a fixed table resulted in suboptimal performance. As a result, we developed a second training method (“constant error”), where different images in the training set use different quantization divisors. We determined the quantization divisors using a multistep process. First, we measured the average perceptual error Eav over the entire training set using Qs . For each image i in the training set, we found the value of Ki (0.5 ≤ Ki ≤ 1.5, in steps of 0.02), so that compression using the divisor tables Ki Qs resulted in a perceptual error Ei such that 0.9Eav ≤ Ei ≤ Eav . We used the divisor tables Ki Qs for training image i; if no Ki could be found that met the inequality, the image was not used during training. This process resulted in a training set devoid of images with unusually large or small perceptual error; for most coefficients, the absence of these outlier images during training improved performance on the cross-validation set. To train one of the 64 neural networks using the constant divisor or the constant error method, we initialize the weights of the network to random ˆ Cˆ b , Cˆ r ) value for the crossvalues and measure the average E(Y, Cb , Cr ; Y, validation set (defined as E¯ cv (u, v)). We then train the network on each block of each image in the training set, using the per-block procedure described above. We set the initial learning rate to 1.0 for constant error training (0.001 for constant divisor training) and measure E¯ cv (u, v) at the end of each training pass. If E¯ cv (u, v) increases from the previous pass, we undo the weight updates from that training pass, reduce the learning rate by a factor of √ 10, and continue training. We terminate training when the learning rate falls below 0.0001 (0.00001 for coefficient (0, 0)) and measure the average ˆ Cˆ b , Cˆ r ) per pixel for the test set (defined as E¯ tst (u, v)). E(Y, Cb , Cr ; Y, In our current system, the neural network architecture for each coefficient is chosen as follows. For each coefficient (u, v), we train all applicable network variants for each coefficient (see section 5 and Figures 1 and 2 for details) with both training methods. We measure the cross-validation error E¯ cv (u, v) for each trained network and pick the network with the lowest error. No artifact reduction is applied to a coefficient if all network architectures have a higher cross-validation error than the baseline error for the coefficient (i.e., the measured error when quantized using Qs ). After choosing the 64 networks for the final system, we measure the test set error while correcting all 64 coefficients (defined as E¯ tst ; note the lack of a (u, v) specifier). In this final test, the coefficients for each block are computed using equation 5.1, and normal JPEG decoding (equation 2.2) is used to compute the pixels of the reconstructed images.
JPEG Quality Transcoding
279
We trained our networks using a workstation cluster and a 4-CPU multiprocessor as our compute engines, powered by 250 MHz and 300 MHz UltraSPARC II processors. Depending on the network architecture, coefficient number, and training method, it took 30 to 120 minutes of computing time on an unloaded processor to compute a single epoch of training, and 3 to 15 epochs to train a network. To train the baseline system, we used approximately 5500 hours of processor time. 7 Results: Perceptual Error Performance In this section, we describe the performance of the artifact reduction system on test set images compressed using the Qs quantization tables. The description of these tables in section K.1 of CCITT Rec T.81 Standards Document notes that images compressed with the quantization table Qs /2 (which we define to be Qe ) result in images that are usually indistinguishable from the source image. In the light of this observation, a reasonable benchmark for our artifact reduction system is its success in reducing the perceptual error of an image compressed with Qs to the perceptual error of the same image compressed with Qe . Figure 3 shows the performance of the artifact reduction system against this benchmark. It shows a plot of the perceptual error on the test set as a function of coefficient frequency. To produce these plots, we measure the error for quantizing one coefficient, while leaving the other coefficients unquantized. The frequency axis on this plot is a zig-zag scan of the (u, v) frequency space, as shown in Figure 2b. Figure 3 shows three measurements. The bottom thick curve (labeled Qe ) shows JPEG decoding without artifact reduction, using Qe divisor tables. The top thick curve (labeled Qs ) shows JPEG decoding without artifact reduction, using the Qs tables. The thin line (labeled JQT) shows the test set performance of the artifact reduction system, while processing images compressed with the Qs quantization tables. In Figure 4, we replot this JQT performance curve in percentage terms, relative to the Qs and Qe performance values, using the expression % reduced = 100
E(Qs ) − E(JQT) . E(Qs ) − E(Qe )
(7.1)
This plot shows 25% to 35% error reduction for most of the coefficients. Performance degrades for the lowest-frequency coefficients (due to the difficulty of the task) and for the highest-frequency coefficients (due to the limited amount of the training data, since natural images have very little energy at these spatial frequencies). The dip in performance for isolated midrange coefficient values corresponds to coefficients with the highest horizontal or vertical spatial frequency. This behavior can be seen more
280
John Lazzaro and John Wawrzynek
6 E Qs 4
JQT
2 Qe 0
20
k
40
60
Figure 3: Plot showing E¯ tst (u, v)(×10−3 ) for JPEG encoding with Qe quantizer tables (lower thick line), JPEG encoding with Qs quantizer tables (upper thick line), and the results of the artifact reduction system when applied to the Qs encoding (thin lines labeled JQT). Coefficient numbering scheme shown in Figure 2b.
clearly in Figure 5a, where we replot the percentage reduction data on the two-dimensional (u, v) coefficient grid. As described in section 6, we used two different training methods (constant divisor and constant error) for each architecture; cross-validation performance was used to pick the final networks. For most coefficients, the constant error training method produced the best cross-validation performance; the dots shown in Figure 4 mark the exceptional coefficients whose
JPEG Quality Transcoding
281
40 % 30 20 10 0 0
20
k
40
60
Figure 4: The percentage reduction of perceptual error achieved by the artifact reduction system. Zero percent corresponds to Qs error values; 100% corresponds to Qe error values. The dots indicate networks trained with the constant divisor training method. For all other coefficients, the cross-validation performance of a network trained with the constant error method was superior.
highest-performing network was the product of constant divisor training. This result shows the advantage of excluding outlier images from the training set. We found that for architecture A, constant divisor training is particularly ineffective for higher-frequency coefficients. To save training time, we trained only architecture A using constant divisor training for coefficients 0–43. Figure 6 shows further data concerning the architecture selection pro-
282
John Lazzaro and John Wawrzynek
% 0 18 36
0 1 2 3 4 5 6 7 v
0 1 2 3 4 5 6 7 v
0 1 2 3 4 5 6 7 v
0 1 2 3 4 5 6 7 u
(a) 0 1 2 3 4 5 6 7 u
(b) 0 1 2 3 4 5 6 7 u
(c)
Figure 5: (a) Percentage reduction for the artifact reduction system, plotted using gray scale on the (u, v) coordinate grid (See Figure 2b). The rule at the side of the figure shows the mapping of shading to percentage. (b) Percentage reduction performance using a linear network instead of the artifact reduction system. The network is trained with the perceptual error metric (see section 8.4). Negative percentages are mapped to white (0%). (c) Percentage reduction performance using a linear network instead of artifact reduction system; the network is trained using mean squared error (see section 8.4). Negative percentages are mapped to white (0%).
JPEG Quality Transcoding
283
50 % C&D 40 30 20
B A
10 0 0
20
k
40
60
Figure 6: Percentage reduction of perceptual error measured on the crossvalidation, for architectures A (thin line), B (thick line), C and D (dots). For each coefficient number, the architecture with the highest percentage reduction was chosen for the artifact reduction system.
cess. In this graph, we show the performance on the cross-validation set for architectures A–D, for the training method with the higher performance. We show the cross-validation results because performance on this set is used to select the network architecture of the final system. Figure 7 shows the architecture selected for each coefficient. The thin curve in Figure 6 shows the percentage error for architecture A (see Figure 2a), which has three copies of each hidden unit type. The thick curve shows the performance of architecture B, which has one copy of each
284
John Lazzaro and John Wawrzynek
0 1 2 3 4 5 6 7 v
01 2345 67u AAAACCAB AAAAABAA AAAAAAAB DAAAAABB AAAAAB B BAAABB A DBAAB B ABAAB B
Figure 7: Network architecture (A–D; see Figure 2a) chosen for each coefficient, plotted on the (u, v) coordinate grid. A blank box indicates no network is used for this coefficient.
hidden unit type. Architecture A works best for lower-frequency coefficients (below 25); to save training time, we did not train architecture B networks for the lowest-frequency coefficients (0–9). For higher-frequency coefficients, the best architecture is coefficient dependent. As the graph shows, for some coefficients, A is superior, and for others B works better. This behavior is consistent with the theory that lower-frequency artifacts are more complicated in nature and are better modeled by higher-parameter models, whereas higher-frequency coefficients have simpler artifact behavior, which may be overfit by a higher-parameter model. The dots in Figure 6 show the cross-validation performance of architectures C and D. Recall that these architectures are used for the 14 coefficients ((0, u 6= 0) and (v 6= 0, 0)), which have energy in only one spatial frequency axis. For the midfrequency coefficients 9, 14, 15, and 21, these architectures perform better than architectures A and B on the cross-validation set; note that the dots lie on or above the thin and thick lines at these locations in Figure 5. These networks generalized well, providing equivalent
JPEG Quality Transcoding
285
(14 and 15) or superior (9 and 21) performance to networks A and B on the test set. Finally, we measure the perceptual error on the test set if all 64 coefficients are quantized simultaneously. This test simulates the performance of the artifact reduction system in a JPEG transcoder application. Without artifact reduction, this error is 0.02555 for Qs quantization and 0.01965 for Qe quantization. The test set error of the artifact reduction system while processing images compressed with the Qs quantization table is 0.02365 (defined as E¯ tst in section 6). Equation 7.1 yields a percent reduction of 32.2% for this task. This result shows a good correlation between the improvements in single-coefficient performance shown in Figure 4. 8 Results: Standard Measures of Performance In section 7 we reported the performance of the artifact reduction system using the perceptual error metric. In this section, we characterize the system performance with techniques common in the image processing community. 8.1 Peak-Signal-to-Noise Ratio. A classical way to judge image processing systems is the peak-signal-to-noise ratio (PSNR). For a YCr Cb image of size N × M, with original image pixels (Y, Cr , Cb ) and degraded pixels ˆ Cˆ r , Cˆ b ), the PSNR is defined as (Y, 10 log10
(1/3)(1/NM)
P M,N
2552 . ˆ 2 + (Cr − Cˆ r )2 + (Cb − Cˆ b )2 ) ((Y − Y)
While this metric does not correlate well with human perception of artifacts (Fuhrmann et al., 1995), it is a common figure of merit in the image processing community. For each test set image in the database used in section 7, we measured the difference between the PSNR for compression using Qs and compression using Qe . The average PSNR difference between the two compression levels for an image in the test set database is 2.5 dB. We also measured the difference between the PSNR for compression using Qs and compression using Qs followed by the artifact reduction system. The average PSNR difference is 0.63 dB, a significant portion of the 2.5 dB PSNR that corresponds to the perceptually indistinguishable Qe tables. Figure 8 tabulates this result, along with PSNR measurements, for three color photos often used in the image processing community: Lena, Parrot, and Peppers. Note that linearly coded versions of these popular test images (YCr Cb , not the more common YCr0 Cb0 ) are used. These images are not in our training or cross-validation data sets. 8.2 Bit-Rate Savings. Another way to characterize the artifact reduction system is to measure the equivalent savings in bit rate. In this approach, we compress a test set image using divisor table Qs and measure the size of the
286
John Lazzaro and John Wawrzynek
Testset
Perceptual
Parrot
Lena
Peppers
768 x 512 512 x 512 512 x 512
Qs
0.0255
0.0252
0.0362
0.0440
Qe
0.0196
0.0191
0.0298
0.0347
JQT
0.0236 32.2%
0.0221 50.8%
0.0327 55.2%
0.0386 57.8%
0.0248,12% 0.0237, 25% Lin: Perc. MSE 0.025,10.2% 0.0239, 21% PSNR 2.50 dB 2.26 dB Qs - Qe 0.79 dB Qs - JQT 0.634 dB 0.132 dB 0.22 dB Lin: Perc. MSE 0.129 dB 0.23 dB bits/pixel 1.110 0.517 Ss 0.160 0.135 Ss - Seq 14.4% 20.7% 0.0503,5.5% 0.051, 9% Lin: Perc. MSE 0.0407,4.4% 0.040,7.2%
0.035,24% 0.0420, 21% 0.035,24% 0.0425,15%
1.74 dB 0.70 dB 0.18 dB 0.21 dB
2.20 dB 0.68 dB 0.10 dB 0.11 dB
0.704 0.192 21.4%
0.744 0.215 22.4%
0.066,8.5% 0.0682, 8.4% 0.066,8.5% 0, 0%
Figure 8: Tabulation of the artifact reduction system performance for the linearly-coded test data set and for three images from the test set that are commonly used in the image processing community; these images are reproduced at the top of the figure. Results for the baseline nonlinear system and for two linear systems are shown. Three metrics (perceptual error, PSNR, and bits/pixel) are used, grouped by shading. See sections 7 and 8 for details.
JPEG Quality Transcoding
287
Table 1: Artifact Reduction System Performance.
Perceptual KQs Qe JQT PSNR KQs − Qe KQs − JQT Bits/pixel Ss Ss − Seq , %
K = 0.6
K = 0.8
K = 1.0
K = 1.2
K = 1.4
0.0208 0.0196 0.0195, 111%
0.0234 0.0196 0.0217, 46.5%
0.0255 0.0196 0.0236, 32.2%
0.0273 0.0196 0.0253, 26.1%
0.0290 0.0196 0.0265, 26.2%
0.613 dB 0.504 dB
1.666 dB 0.577 dB
2.505 dB 0.634 dB
3.075 dB 0.648 dB
3.583 dB 0.645 dB
1.54 0.170, 11.3%
1.29 0.160, 12.3%,
1.11 0.160, 14.3%
1.00 0.140, 14.2%
0.897 0.129, 14.4%
compressed file in terms of bits per color pixel (Ss ). We decode the image, apply the artifact reduction system, and measure the perceptual error of the image (Es ). We then use a search technique to find the divisor table KQs that results in a compressed image whose perceptual error is equal to Es , without applying the artifact reduction system. In this search, K is quantized in steps of 0.02, modeling the quantization of this scaling parameter in many JPEG applications. We measure the bits per pixel Seq of the file compressed with KQs , and consider the difference Seq − Ss to be the bits gained by artifact reduction. Figure 8 shows this measure for both the entire test set and specific images. We tabulate the average values of Ss , Seq − Ss , and the average percentage bit savings, defined to be the average of (Ss − Seq )/Seq over the data set. For the test set, an average percentage bit savings of 14.4% is achieved. 8.3 Scaling Performance. We targeted the artifact reduction system to work well for the Qs quantization tables. In Table 1, we tabulate the performance of the system on the scaled quantization tables KQs for K < 1.0 (higher image quality) and K > 1.0 (lower image quality), using the perceptual error, PSNR, and bit-rate savings metrics. We measured this performance because in practice, JPEG end users often manually scale the Qs tables to achieve a certain perceptual quality versus file size trade-off, and so a practical JQT would need to work reasonably well for a range of scalings. Table 1 shows reasonable performance over the range of K scalings on all three metrics. The relative performance of the system as a function of K is metric dependent. In terms of bit-rate savings, the system performs best for K > 1.0. However, in terms of percentage reduction of perceptual error relative to the perceptually Qe , the system performs best for K < 1.0. 8.4 Comparison with Linear Networks. Another way to characterize the artifact reduction system is to compare its performance with a linear system. To perform this comparison, we trained an artifact reduction system that replaced the 64 multilayer perceptron neural networks with 64
288
John Lazzaro and John Wawrzynek
single-layer linear networks. These networks use the same 47 coefficient inputs of architectures A and B (see Figure 2a). The outputs of the linear networks are clipped to the range of plausible values, as indicated by the quantization divisor. We trained the linear networks with gradient descent, using the annealing methods described in section 6. Separate linear networks were trained using the constant divisor and constant error methods for each coefficient, and cross-validation performance was used to select the best network. We trained two systems, one using the perceptual error metric to compute the gradient and one using the conventional mean squared error (MSE) metric to compute the gradient. Figure 9 shows the performance of the linear networks for singlecoefficient artifact reduction, using the percentage metric of equations 7.1. We reproduced the single-coefficient performance curve shown in Figure 4 for reference. Figure 9 shows that the linear networks are markedly inferior in performance for the lowest coefficients and unable to provide any artifact reduction for all other coefficients. The negative percentages on this graph indicate the poor generalization of the linear networks: the cross-validation results for these coefficients showed improvements over the Qs error, but test results were inferior to the Qs error. The linear network trained with the perceptual error metric (thin line) performs better than the MSE-trained networks (dots) on the key low-frequency coefficients 1 and 2. In Figures 5b and 5c, we replot the performance of the linear networks on the two-dimensional (u, v) coefficient grid, mapping all negative percentages to white (0%). The single coefficient results in Figure 9 are confirmed when the linear networks are used for artifact reduction on all 64 coefficients simultaneously. To show the linear networks in the best possible light, we included networks only for coefficients 0–9, eliminating the negative effects of highfrequency poor generalization shown in Figure 9. Figure 8 shows the poor performance of the linear network artifact reduction system, on both the perceptual error measure, PSNR, and bit savings, on both the full test set data and on selected images. The linear network trained with the perceptual error metric performs marginally better than the network trained with MSE, reflecting the better performance on coefficients 1 and 2 shown in Table 1. 8.5 Images. Finally, we present color images to show qualitatively the performance of the artifact reduction system. Figure 10 shows a closeup of a parrot head in the Parrot image for the five values of K tabulated in Table 1. The top row shows the original image, the upper middle row shows the image compressed with KQs (from left to right, K = 0.6, 0.8, 1.0, 1.2, 1.4), and the lower middle row shows the results of the artifact reduction system on the KQs compressed images. This closeup was chosen to highlight the high-frequency performance of the system. Note that the corona of artifacts around the parrot’s head and beak is significantly reduced by the artifact reduction system. This improvement corresponds to the high performance figures for midrange of coefficients (3–50) in Figure 4.
JPEG Quality Transcoding
289
40 % 20 0 -20 0
20
k
40
60
Figure 9: Comparison of the percentage reduction of the artifact reduction system (heavy curve, data reproduced from Figure 4) with systems that replace the multilayer perceptron neural networks with single-layer linear networks. Thin line shows results from a linear net trained with the perceptual error metric; dots show results from a linear net trained with mean squared error (computed only for coefficients 0–31).
Figure 11 shows a closeup of the cheek and nose of the Lena image. The format of the figure is identical to Figure 10. Note that the facial discoloration and lip artifacts are modestly reduced at each scaling value. The modest improvement in these low-frequency artifacts corresponds to the modest performance figures for the lowest-frequency coefficients (0–2) in Figure 4.
290
John Lazzaro and John Wawrzynek
Figure 10: Closeup from Parrot image, showing high-frequency system performance at 5 scalings of Qs . From left to right, scalings are 0.6Qs , 0.8Qs , 1.0Qs , 1.2As , and 1.4Qs . From top to bottom: original image, image compressed with KQs , image compression with KQs followed by artifact reduction system, and difference image (see text for details).
In both figures, we computed the signed difference between the artifacts present in the upper middle row and the artifacts present in the lower middle row (artifacts were computed by subtraction from the original images). This difference image was scaled (by three for Figure 10, by four for Figure 11) and added to a neutral gray to produce the bottom row of each image. Nongray parts of this image indicate areas where the artifact reduction system significantly altered the compressed image. Readers can use this difference image to guide comparisons of the middle rows. 9 Discussion Examining the results presented in sections 7 and 8, we see several promising avenues for improving the performance of the artifact reduction system. One avenue concerns improving the accuracy of the perceptual error metric. For example, the current metric does not model chromatic masking or the spatial-frequency dependence of the relative weightings of opponent chan-
JPEG Quality Transcoding
291
Figure 11: Closeup from Lena image, showing low-frequency system performance at 5 scalings of Qs . Format identical to Figure 10.
nel outputs. In addition, the per-coefficient training method provides an implicit model of spatial-frequency sensitivity, but more explicit modeling of this phenomena may produce better results. Another promising avenue for research is improving system performance through more appropriate neural network architectures. Possible improvements include more hidden layers to model the complexity of artifacts and an automatic method for choosing inputs relevant for each coefficient. These improvements need to focus on the lowest-frequency coefficients, where the current system shows only modest performance improvements. Apart from performance improvements, the work presented in this article requires other enhancements in order to be used in a practical system. A practical JQT implementation must also include a method of quantization. As implemented here, the enhanced coefficients k˜C (u, v) are maintained as floating-point values. To create the transcoder JPEG file, a JQT must decide, for each coefficient, how many bits of precision should be maintained.
292
John Lazzaro and John Wawrzynek
Finally, for many applications, a version of the JQT that uses nonlinear image coding (YCr0 Cb0 ) is needed. Initial experiments with a JQT trained with a Y0 Cr0 Cb0 image database and modified perceptual models (see appendix A) show good results using the techniques described in this article (31% perceptual error improvement, 0.56dB PSNR, 11.5% bit-rate savings on an independent test database). 10 Conclusions We have presented a neural network image processing system that operates directly on a compressed representation of an image and uses a perceptual error metric to guide supervised learning on a large image database. We believe this approach has more general application in image processing, beyond the artifact reduction problem. An advantage of this approach is the ability to define a variant of a general problem by customizing the training database of images. A JQT customized for photographs of faces can be specified by assembling an image database with a heavy representation of these types of photos. A JFIF transcoder that corrects for artifacts caused by an inexpensive analog-to-digital conversion in a consumer digital camera can be trained by collecting a database using a prototype camera that has auxiliary high-quality conversion circuitry. This ease of customization may be the deciding factor for using a pattern recognition approach for a particular problem in digital imaging. Appendix A: Color Space Transformation: YCr Cb to LMS In this appendix, we derive the transformation from YCr Cb color space to LMS color space. The YCr Cb color space, as used in the JPEG/JFIF standards, uses 8-bit integer values. Substituting this scaling into the suggested YCr Cb to RGB conversion in CCIR Recommendation 601-1 yields: R = (Y/255) + 1.402(Cr /255) G = (Y/255) − 0.3441(Cb /255) − 0.7141(Cr /255) B = (Y/255) + 1.772(Cb /255). These equations assume Y ranges from 0 to 255, and Cr and Cb range from −128 to 127. The R, G, and B values are valid between 0.0 and 1.0. Since some values of YCr Cb may produce RGB values outside the valid range, we clamp the RGB values to a maximum of 1.0 and a minimum of 0.0. If linearly coded YCr Cb images are used, these RGB values will be linear. However, if nonlinearly coded inputs are used (Y0 Cr0 Cb0 ), these RGB values will be nonlinear, and must be converted to linear RGB before proceeding. Following CCIR Recommendation 709 (D65 white point), we convert
JPEG Quality Transcoding
293
RGB to 1931 2-deg CIE XYZ tristimulus values using the equations: X = 0.4124R + 0.3576G + 0.1804B Y = 0.2127R + 0.7152G + 0.07217B Z = 0.01933R + 0.1192G + 0.9502B. We use the following equations, derived for typical spectral power distributions of the phosphors in a Sony 17-inch color monitor (Tjan, 1996), to convert CIE 1931 XYZ values to Judd-Vos tristimulus values Xp Yp Zp : Xp = 0.9840X + 0.00822Y − 0.00459Z Yp = 0.00028X + 0.9992Y + 0.00519Z Zp = −0.00177X + 0.00388Y + 0.9215Z. To complete the transformation to LMS space, we convert Judd-Vos tristimulus values to Smith-Pokorny cone excitations (Tjan, 1996): L = 0.1551Xp + 0.5431Yp − 0.03286Zp M = −0.1551Xp + 0.4568Yp + 0.03286Zp S = 0.00801Zp . These operations can be collapsed into two sets of linear equations and a clipping operation. Appendix B includes this compact form of the YCr Cb to LMS transformation. Appendix B: Computing the Perceptual Error Metric In this appendix, we defined a pointwise perceptual metric, computed on a ˆ Cˆ b , Cˆ r ) pixel (Y, Cb , Cr ) in an original image and the corresponding pixel (Y, in a reconstructed image. We begin by converting both points from YCb Cr color space to LMS space, as derived in appendix A. We assume Y ranges from 0 to 155, and Cb and Cr range from −128 to 127. R = 0.003922Y + 0.005498Cr G = 0.003922Y − 0.001349Cb − 0.002800Cr B = 0.003922Y + 0.006949Cb . Clamp R, G, and B to lie between 0.0 and 1.0 linearize if needed (see appendix A), then compute LMS values as: L = 0.17816R + 0.4402G + 0.04005B M = 0.03454R + 0.2750G + 0.03703B S = 0.0001435R + 0.008970G + 0.007014B.
294
John Lazzaro and John Wawrzynek
ˆ values, we compute cone contrast vectors as: ˆ M, ˆ S) Using (L, M, S) and (L, 1L/L =
L − Lˆ L + Lo
ˆ M−M M + Mo S − Sˆ 1S/S = . S + So 1M/M =
Using the recommendations in Macintyre and Cowan (1992) for the dark pixel characteristics of commercial CRT monitors, we set Lo = 0.01317, Mo = 0.006932, So = 0.0001611. Consult Macintyre and Cowan (1992) for details on tuning these values to a particular CRT monitor. Using these cone contrast values, we compute opponent space values as: BW = A(x, y)(1L/L + 1M/M) RG = 1L/L − 1M/M BY = 1S/S − 0.5(1L/L + 1M/M) where A(x, y) is the activity function value for the pixel position under comparison. Using these opponent values, we compute the error function ˆ Cˆ b , Cˆ r ) = 0.07437|BW| + 0.8205|RG| + 0.1051|BY|. E(Y, Cb , Cr ; Y, The channel weightings in the error function are the averaged relative sensitivities of the three subjects measured in Cole et al. (1993). The error function sums the absolute values of the opponent channels, rather than the squared values used in a Euclidean metric. This choice reflects the assumed independence of the underlying mechanisms. Straightforward application of the chain rule yields the partial derivaˆ ∂E()/∂ Cˆ b , ∂E()/∂ Cˆ r used in the backpropagation learning tives ∂E()/∂ Y, algorithm. To compute the A(x, y) function over the original image, we use the Y component of YCb Cr pixels directly, without converting to LMS space. We take this approach because BW in opponent space and Y in YCb /Cr space are qualitatively similar measures of luminance. To compute A(x, y), we first compute the mean luminance value Ym in a 5 × 5 block centered around pixel position (x, y). We then compute the contrast Y/Ym for each pixel in a 5 × 5 block centered around pixel position (x, y), and clamp the lower limit of this contrast at 10/Ym . If a contrast is less than 1, we take its reciprocal. We sum these 25 modified contrast values, divide by 25, and take the reciprocal of the result to produce A(x, y). The function is an average measure of edge activity in a region, which takes a value of 1 for smooth areas and a value less than one for regions around an edge.
JPEG Quality Transcoding
295
Acknowledgments We thank the three referees for many helpful suggestions. We also thank Krste Asanovic, Dan Hammerstrom, John Hauser, Yann LeCun, Richard Lyon, John Platt, and Larry Yaeger for useful comments and Terry Sejnowski for his invitation to submit to this anniversary issue. This research was funded by DARPA contract number DABT63-96-C-0048. References Ahumada, A. J., & Horng, R. (1994). Smoothing DCT compression artifacts. In 1994 SID International Symposium Digest of Technical Papers (pp. 708–711). Santa Ana, CA: SID. Cole, G. R., Hine, T., & McIlhagga, W. (1993). Detection mechanisms in L-, M-, and S-cone contrast space. Journal Optical Society of America, 10, 38–51. Fuhrmann, D. R., Baro, J. A., & Cox, J. R. (1995). Experimental evaluation of psychophysical distortion metrics for JPEG-encoded images. Journal of Electronic Imaging, 4, 397–406. Jarske, T., Haavisto, P., & Defee, I. (1994). Post-filtering methods for reducing blocking artifacts from coded images. IEEE Transactions on Consumer Electronics, 40, 521–526. Kim, K. M., Lee, C. S., Eung, J. L., & Yeong, H. H. (1996). Color image quantization and dithering method based on human visual system characteristics. Journal of Imaging Science and Technology, 40, 502–509. Macintyre, B., & Cowan, W. B. (1992). A practical approach to calculating luminance contrast on a CRT. ACM Transaction on Graphics, 11, 336–347. Minami, S., & Zakhor, A. (1995). An optimization approach for removing blocking artifacts in transform coding. IEEE Transactions on Circuits and Systems for Video Technology, 5, 74–81. O’Rourke, T. P., & Stevenson, R. L. (1995). Improved image decompression for reduced transform coding artifacts. IEEE Transactions on Circuits and Systems for Video Technology, 5, 490–499. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error propagation. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing: Explorations in the microstructure of cognition, Vol. 1: Foundations. Cambridge, MA: MIT Press. Tjan, B. S. (1996). Color spaces for human observer (Tech. Memo). Minneapolis: Minnesota Laboratory for Low-Vision Research, University of Minnesota. Available online at: http://vision.psych.umn.edu/www/people/ bosco/techs.html. van der Branden Lambrecht, C. J., & Farrell, J. E. (1996). Perceptual quality metric for digitally coded color images. In Proceedings of EUSIPCO-96. Trieste, Italy. Wallace, G. K. (1992). The JPEG still picture compression standard. IEEE Transactions on Consumer Electronics, 38, 18–34. Westen, S. J. P., Lagendijk, R. L., and Biemond, J. (1996). Optimization of JPEG color image coding using a human visual system model. Proceedings of the
296
John Lazzaro and John Wawrzynek
SPIE, 2657, 370–381. Wu, S.-W., & Gersho, A. (1992). Improved decoder for transform coding with application to the JPEG baseline system. IEEE Transactions of Communications, 40, 251–254. Yang, Y., Galasysanos, N. P., & Katsaggelos, A. K. (1995). Projection-based spatially adaptive reconstruction of block-transform compressed images. IEEE Transactions on Image Processing, 4, 896–908. Received January 5, 1998; accepted June 25, 1998.
REVIEW
Communicated by Steven Nowlan
A Unifying Review of Linear Gaussian Models Sam Roweis∗ Computation and Neural Systems, California Institute of Technology, Pasadena, CA 91125, U.S.A.
Zoubin Ghahramani∗ Department of Computer Science, University of Toronto, Toronto, Canada
Factor analysis, principal component analysis, mixtures of gaussian clusters, vector quantization, Kalman filter models, and hidden Markov models can all be unified as variations of unsupervised learning under a single basic generative model. This is achieved by collecting together disparate observations and derivations made by many previous authors and introducing a new way of linking discrete and continuous state models using a simple nonlinearity. Through the use of other nonlinearities, we show how independent component analysis is also a variation of the same basic generative model. We show that factor analysis and mixtures of gaussians can be implemented in autoencoder neural networks and learned using squared error plus the same regularization term. We introduce a new model for static data, known as sensible principal component analysis, as well as a novel concept of spatially adaptive observation noise. We also review some of the literature involving global and local mixtures of the basic models and provide pseudocode for inference and learning for all the basic models. 1 A Unifying Review Many common statistical techniques for modeling multidimensional static data sets and multidimensional time series can be seen as variants of one underlying model. As we will show, these include factor analysis, principal component analysis (PCA), mixtures of gaussian clusters, vector quantization, independent component analysis models (ICA), Kalman filter models (also known as linear dynamical systems), and hidden Markov models (HMMs). The relationships between some of these models has been noted in passing in the recent literature. For example, Hinton, Revow, and Dayan (1995) note that FA and PCA are closely related, and Digalakis, Rohlicek, and Ostendorf (1993) relate the forward-backward algorithm for HMMs to ∗ Present address: {roweis, zoubin}@gatsby.ucl.ac.uk. Gatsby Computational Neuroscience Unit, University College London, 17 Queen Square, London WCIN 3AR U.K.
Neural Computation 11, 305–345 (1999)
c 1999 Massachusetts Institute of Technology °
306
Sam Roweis and Zoubin Ghahramani
Kalman filtering. In this article we unify many of the disparate observations made by previous authors (Rubin & Thayer, 1982; Delyon, 1993; Digalakis et al., 1993; Hinton et al., 1995; Elliott, Aggoun, & Moore, 1995; Ghahramani & Hinton, 1996a,b, 1997; Hinton & Ghahramani, 1997) and present a review of all these algorithms as instances of a single basic generative model. This unified view allows us to show some interesting relations between previously disparate algorithms. For example, factor analysis and mixtures of gaussians can be implemented using autoencoder neural networks with different nonlinearities but learned using a squared error cost penalized by the same regularization term. ICA can be seen as a nonlinear version of factor analysis. The framework also makes it possible to derive a new model for static data that is based on PCA but has a sensible probabilistic interpretation, as well as a novel concept of spatially adaptive observation noise. We also review some of the literature involving global and local mixtures of the basic models and provide pseudocode (in the appendix) for inference and learning for all the basic models. 2 The Basic Model The basic models we work with are discrete time linear dynamical systems with gaussian noise. In such models we assume that the state of the process in question can be summarized at any time by a k-vector of state variables or causes x that we cannot observe directly. However, the system also produces at each time step an output or observable p-vector y to which we do have access. The state x is assumed to evolve according to simple first-order Markov dynamics; each output vector y is generated from the current state by a simple linear observation process. Both the state evolution and the observation processes are corrupted by additive gaussian noise, which is also hidden. If we work with a continuous valued state variable x, the basic generative model can be written1 as: xt+1 = Axt + wt = Axt + w•
w• ∼ N (0, Q)
(2.1a)
yt = Cxt + vt = Cxt + v•
v• ∼ N (0, R)
(2.1b)
where A is the k × k state transition matrix and C is the p × k observation, measurement, or generative matrix. The k-vector w and p-vector v are random variables representing the state evolution and observation noises, respectively, which are independent 1 All vectors are column vectors. To denote the transpose of a vector or matrix, we use the notation xT . The determinant of a matrix is denoted by |A| and matrix inversion by A−1 . The symbol ∼ means “distributed according to.” A multivariate ¡normal ¢ (gaussian) distribution with mean µ and covariance matrix Σ is written as N µ, Σ . The same
¡
¢
gaussian evaluated at the point z is denoted N µ, Σ |z .
A Unifying Review of Linear Gaussian Models
307
of each other and of the values of x and y. Both of these noise sources are temporally white (uncorrelated from time step to time step) and spatially gaussian distributed2 with zero mean and covariance matrices, which we denote Q and R, respectively. We have written w• and v• in place of wt and vt to emphasize that the noise processes do not have any knowledge of the time index. The restriction to zero-mean noise sources is not a loss of generality.3 Since the state evolution noise is gaussian and its dynamics are linear, xt is a first-order Gauss-Markov random process. The noise processes are essential elements of the model. Without the process noise w• , the state xt would always either shrink exponentially to zero or blow up exponentially in the direction of the leading eigenvector of A; similarly in the absence of the observation noise v• the state would no longer be hidden. Figure 1 illustrates this basic model using both the engineering system block form and the network form more common in machine learning. Notice that there is degeneracy in the model. All of the structure in the matrix Q can be moved into the matrices A and C. This means that we can, without loss of generality, work with models in which Q is the identity matrix.4 Of course, R cannot be restricted in the same way since the values yt are observed, and hence we are not free to whiten or otherwise rescale them. Finally, the components of the state vector can be arbitrarily reordered; this corresponds to swapping the columns of C and A. Typically we choose an ordering based on the norms of the columns of C, which resolves this degeneracy. The network diagram of Figure 1 can be unfolded in time to give separate units for each time step. Such diagrams are the standard method of illustrating graphical models, also known as probabilistic independence networks, a category of models that includes Markov networks, Bayesian (or belief) networks, and other formalisms (Pearl, 1988; Lauritzen & Spiegelhalter, 1988; Whittaker, 1990; Smyth et al., 1997). A graphical model is a representation of the dependency structure between variables in a multivariate probability distribution. Each node corresponds to a random variable, and the absence of an arc between two variables corresponds to a particular conditional independence relation. Although graphical models are beyond 2
An assumption that is weakly motivated by the central limit theorem but more strongly by analytic tractability. 3 Specifically we could always add a k + 1st dimension to the state vector, which is fixed at unity. Then augmenting A with an extra column holding the noise mean and an extra row of zeros (except unity in the bottom right corner) takes care of a nonzero mean for w• . Similarly, adding an extra column to C takes care of a nonzero mean for v• . 4 In particular, since it is a covariance matrix, Q is symmetric positive semidefinite and thus can be diagonalized to the form EΛET (where E is a rotation matrix of eigenvectors and Λ is a diagonal matrix of eigenvalues). Thus, for any model in which Q is not the identity matrix, we can generate an exactly equivalent model using a new state vector −1/2 T −1/2 T 1/2 1/2 x0 = Λ E x with A0 = (Λ E )A(EΛ ) and C0 = C(EΛ ) such that the new 0 0 covariance of x is the identity matrix: Q = I.
308
Sam Roweis and Zoubin Ghahramani
x
t
A
z ,1
+
C
y
+
t
v
w
v y
t
C x
w
t
A
z ,1
Figure 1: Linear dynamical system generative model. The z−1 block is a unit delay. The covariance matrix of the input noise w is Q and the covariance matrix of the output noise v is R. In the network model below, the smaller circles represent noise sources and all units are linear. Outgoing weights have only been drawn from one hidden unit. This model is equivalent to a Kalman filter model (linear dynamical system).
the scope of this review, it is important to point out that they provide a very general framework for working with the models we consider here. In this review, we unify and extend some well-known statistical models and signal processing algorithms by focusing on variations of linear graphical models with gaussian noise. The main idea of the models in equations 2.1 is that the hidden state sequence xt should be an informative lower dimensional projection or explanation of the complicated observation sequence yt . With the aid of the dynamical and noise models, the states should summarize the underlying causes of the data much more succinctly than the observations themselves. For this reason, we often work with state dimensions much smaller than the number of observables—in other words, k ¿ p.5 We assume that both 5
More precisely, in a model where all the matrices are full rank, the problem of inferring
A Unifying Review of Linear Gaussian Models
309
A and C are of rank k and that Q, R, and Q1 (introduced below) are always full rank. 3 Probability Computations The popularity of linear gaussian models comes from two fortunate analytical properties of gaussian processes: the sum of two independent gaussian distributed quantities is also gaussian distributed,6 and the output of a linear system whose input is gaussian distributed is again gaussian distributed. This means that if we assume the initial state x1 of the system to be gaussian distributed, ¡ ¢ x1 ∼ N µ1 , Q1 ,
(3.1)
then all future states xt and observations yt will also be gaussian distributed. In fact, we can write explicit formulas for the conditional expectations of the states and observables: P (xt+1 |xt ) = N (Axt , Q) |xt+1 , ¢ ¡ P yt |xt = N (Cxt , R) |yt .
(3.2a) (3.2b)
Furthermore, because of the Markov properties of the model and the gaussian assumptions about the noise and initial distributions, it is easy to write an expression for the joint probability of a sequence of τ states and outputs: τY −1 τ Y ¢ ¡ ¢ ¡ P (xt+1 |xt ) P yt |xt . P {x1 , . . . , xτ }, {y1 . . . yτ } = P (x1 ) t=1
(3.3)
t=1
The negative log probability (cost) is just the sum of matrix quadratic forms: ¢ ¡ −2 log P {x1 , . . . , xτ }, {y1 , . . . , yτ } τ X [(yt − Cxt )T R−1 (yt − Cxt ) + log |R|] = t=1
the state from a sequence of τ consecutive observations is well defined as long k ≤ τp (a notion related to observability in systems theory; Goodwin & Sin, 1984). For this reason, in dynamic models it is sometimes useful to use state-spaces of larger dimension than the observations, k > p, in which case a single state vector provides a compact representation of a sequence of observations. 6 In other words the convolution of two gaussians is again a gaussian. In particular, ¡ ¢ ¡ ¢ ¡ ¢ the convolution of N µ1 , Σ1 and N µ2 , Σ2 is N µ1 + µ2 , Σ1 + Σ2 . This is not the same as the (false) statement that the sum of two gaussians is a gaussian but is the same as the (Fourier domain equivalent) statement that the multiplication of two gaussians is a gaussian (although no longer normalized).
310
Sam Roweis and Zoubin Ghahramani
+
τ −1 X [(xt+1 − Axt )T Q−1 (xt+1 − Axt ) + log |Q|] t=1
+ (x1 − µ1 )T Q−1 1 (x1 − µ1 ) + log |Q1 | + τ (p + k) log 2π.
(3.4)
4 Learning and Estimation Problems Latent variable models have a wide spectrum of application in data analysis. In some scenarios, we know exactly what the hidden states are supposed to be and just want to estimate them. For example, in a vision problem, the hidden states may be the location or pose of an object; in a tracking problem, the states may be positions and momenta. In these cases, we can often write down a priori the observation or state evolution matrices based on our knowledge of the problem structure or physics. The emphasis is on accurate inference of the unobserved information from the data we do have—for example, from an image of an object or radar observations. In other scenarios, we are trying to discover explanations or causes for our data and have no explicit model for what these causes should be. The observation and state evolution processes are mostly or entirely unknown. The emphasis is instead on robustly learning a few parameters that model the observed data well (assign it high likelihood). Speech modeling is a good example of such a situation; our goal is to find economical models that perform well for recognition tasks, but the particular values of hidden states in our models may not be meaningful or important to us. These two goals—estimating the hidden states given observations and a model and learning the model parameters—typically manifest themselves in the solution of two distinct problems: inference and system identification. 4.1 Inference: Filtering and Smoothing. The first problem is that of inference or filtering and smoothing, which asks: Given fixed model parameters {A, C, Q, R, µ1 , Q1 }, what can be said about the unknown hidden state sequence given some observations? This question is typically made precise in several ways. A very basic quantity we would like to be able to compute is the total likelihood of an observation sequence: P({y1 , . . . , yτ }) Z =
P({x1 , . . . , xτ }, {y1 , . . . , yτ }) d{x1 , . . . , xτ }.
(4.1)
all possible {x1 ,...,xτ }
This marginalization requires an efficient way of integrating (or summing) the joint probability (easily computed by equation 3.4 or similar formulas) over all possible paths through state-space. Once this integral is available, it is simple to compute the conditional distribution for any one proposed hidden state sequence given the observations
A Unifying Review of Linear Gaussian Models
311
by dividing the joint probability by the total likelihood of the observations: ¢ ¡ ¢ P {x1 , . . . , xτ }, {y1 , . . . , yτ } ¡ ¡ ¢ . P {x1 , . . . , xτ }|{y1 , . . . , yτ } = P {y1 , . . . , yτ }
(4.2)
Often we are interested in the distribution of the hidden state at a particular time t. In filtering, we attempt to compute this conditional posterior probability, ¢ ¡ P xt |{y1 , . . . , yt } ,
(4.3)
given all the observations up to and including time t. In smoothing, we compute the distribution over xt , ¢ ¡ P xt |{y1 , . . . , yτ } ,
(4.4)
given the entire sequence of observations. (It is also possible to ask for the conditional state expectation given observations that extend only a few time steps into the future—partial smoothing—or that end a few time steps before the current time—partial prediction.) These conditional calculations are closely related to the computation of equation 4.1 and often the intermediate values of a recursive method used to compute that equation give the desired distributions of equations 4.3 or 4.4. Filtering and smoothing have been extensively studied for continuous state models in the signal processing community, starting with the seminal works of Kalman (1960; Kalman & Bucy, 1961) and Rauch (1963; Rauch, Tung, & Striebel, 1965), although this literature is often not well known in the machine learning community. For the discrete state models, much of the literature stems from the work of Baum and colleagues (Baum & Petrie, 1966; Baum & Eagon, 1967; Baum, Petrie, Soules, & Weiss, 1970; Baum, 1972) on HMMs and of Viterbi (1967) and others on optimal decoding. The recent book by Elliott and colleagues (1995) contains a thorough mathematical treatment of filtering and smoothing for many general models. 4.2 Learning (System Identification). The second problem of interest with linear gaussian models is the learning or system identification problem: given only an observed sequence (or perhaps several sequences) of outputs {y1 , . . . , yτ } find the parameters {A, C, Q, R, µ1 , Q1 } that maximize the likelihood of the observed data as computed by equation 4.1. The learning problem has been investigated extensively by neural network researchers for static models and also for some discrete state dynamic models such as HMMs or the more general Bayesian belief networks. There is a corresponding area of study in control theory known as system identification, which investigates learning in continuous state models. For linear gaussian models, there are several approaches to system identification
312
Sam Roweis and Zoubin Ghahramani
(Ljung & Soderstr ¨ om, ¨ 1983), but to clarify the relationship between these models and the others we review in this article, we focus on system identification methods based on the expectation-maximization (EM) algorithm. The EM algorithm for linear gaussian dynamical systems was originally derived by Shumway and Stoffer (1982) and recently reintroduced (and extended) in the neural computation field by Ghahramani and Hinton (1996a,b). Digalakis et al. (1993) made a similar reintroduction and extension in the speech processing community. Once again we mention the book by Elliott et al. (1995), which also covers learning in this context. The basis of all the learning algorithms presented by these authors is the powerful EM algorithm (Baum & Petrie, 1966; Dempster, Laird, & Rubin, 1977). The objective of the algorithm is to maximize the likelihood of the observed data (equation 4.1) in the presence of hidden variables. Let us denote the observed data by Y = {y1 , . . . , yτ }, the hidden variables by X = {x1 , . . . , xτ }, and the parameters of the model by θ. Maximizing the likelihood as a function of θ is equivalent to maximizing the log-likelihood: Z
L(θ) = log P(Y|θ) = log
P(X, Y|θ ) dX.
(4.5)
X
Using any distribution Q over the hidden variables, we can obtain a lower bound on L: Z log
Z P(X, Y|θ ) dX (4.6a) P(Y, X|θ) dX = log Q(X) Q(X) X X Z P(X, Y|θ ) dX (4.6b) Q(X) log ≥ Q(X) Z ZX = Q(X) log P(X, Y|θ ) dX− Q(X) log Q(X) dX (4.6c) X
X
= F (Q, θ),
(4.6d)
where the middle inequality is known as Jensen’s inequality and can be proved using the concavity of the log function. If we define the energy of a global configuration (X, Y) to be − log P(X, Y|θ ), then some readers may notice that the lower bound F (Q, θ) ≤ L(θ ) is the negative of a quantity known in statistical physics as the free energy: the expected energy under Q minus the entropy of Q (Neal & Hinton, 1998). The EM algorithm alternates between maximizing F with respect to the distribution Q and the parameters θ, respectively, holding the other fixed. Starting from some initial parameters θ0 : E-step:
Qk+1 ← arg max F (Q, θk ) Q
(4.7a)
A Unifying Review of Linear Gaussian Models
M-step:
θk+1 ← arg max F (Qk+1 , θ). θ
313
(4.7b)
It is easy to show that the maximum in the E-step results when Q is exactly the conditional distribution of X: Qk+1 (X) = P(X|Y, θk ), at which point the bound becomes an equality: F (Qk+1 , θk ) = L(θk ). The maximum in the M-step is obtained by maximizing the first term in equation 4.6c, since the entropy of Q does not depend on θ : Z M-step:
θk+1 ← arg max θ
X
P(X|Y, θk ) log P(X, Y|θ ) dX.
(4.8)
This is the expression most often associated with the EM algorithm, but it obscures the elegant interpretation of EM as coordinate ascent in F (Neal & Hinton, 1998). Since F = L at the beginning of each M-step and since the E-step does not change θ, we are guaranteed not to decrease the likelihood after each combined EM-step. Therefore, at the heart of the EM learning procedure is the following idea: use the solutions to the filtering and smoothing problem to estimate the unknown hidden states given the observations and the current model parameters. Then use this fictitious complete data to solve for new model parameters. Given the estimated states obtained from the inference algorithm, it is usually easy to solve for new parameters. For linear gaussian models, this typically involves minimizing quadratic forms such as equation 3.4, which can be done with linear regression. This process is repeated using these new model parameters to infer the hidden states again, and so on. We shall review the details of particular algorithms as we present the various cases; however, we now touch on one general point that often causes confusion. Our goal is to maximize the total likelihood (see equation 4.1) (or equivalently maximize the total log likelihood) of the observed data with respect to the model parameters. This means integrating (or summing) over all ways in which the generative model could have produced the data. As a consequence of using the EM algorithm to do this maximization, we find ourselves needing to compute (and maximize) the expected log-likelihood of the joint data, where the expectation is taken over the distribution of hidden values predicted by the current model parameters and the observations. Thus, it appears that we are maximizing the incorrect quantity, but doing so is in fact guaranteed to increase (or keep the same) the quantity of interest at each iteration of the algorithm. 5 Continuous-State Linear Gaussian Systems Having described the basic model and learning procedure, we now focus on specific linear instances of the model in which the hidden state variable x is continuous and the noise processes are gaussian. This will allow us to
314
Sam Roweis and Zoubin Ghahramani
elucidate the relationship among factor analysis, PCA, and Kalman filter models. We divide our discussion into models that generate static data and those that generate dynamic data. Static data have no temporal dependence; no information would be lost by permuting the ordering of the data points yt ; whereas for dynamic data, the time ordering of the data points is crucial. 5.1 Static Data Modeling: Factor Analysis, SPCA, and PCA. In many situations we have reason to believe (or at least to assume) that each point in our data set was generated independently and identically. In other words, there is no natural (temporal) ordering to the data points; they merely form a collection. In such cases, we assume that the underlying state vector x has no dynamics; the matrix A is the zero matrix, and therefore x is simply a constant (which we take without loss of generality to be the zero vector) corrupted by noise. The new generative model then becomes: A=0
⇒
x• = w•
w• ∼ N (0, Q)
(5.1a)
y• = Cx• + v•
v• ∼ N (0, R) .
(5.1b)
Notice that since xt is driven only by the noise w• and since yt depends only on xt , all temporal dependence has disappeared. This is the motivation for the term static and for the notations x• and y• above. We also no longer use a separate distribution for the initial state: x1 ∼ x• ∼ w• ∼ N (0, Q). This model is illustrated in Figure 2. We can analytically integrate equation 4.1 to obtain the marginal distribution of y• , which is the gaussian, ´ ³ y• ∼ N 0, CQCT + R .
(5.2)
Two things are important to notice. First, the degeneracy mentioned above persists between the structure in Q and C.7 This means there is no loss of generality in restricting Q to be diagonal. Furthermore, there is arbitrary sharing of scale between a diagonal Q and C. Typically we either restrict the columns of C to be unit vectors or make Q the identity matrix to resolve this degeneracy. In what follows we will assume Q = I without loss of generality. Second, the covariance matrix R of the observation noise must be restricted in some way for the model to capture any interesting or informative projections in the state x• . If R were not restricted, learning could simply choose C = 0 and then set R to be the sample covariance of the data, thus trivially achieving the maximum likelihood model by explaining all of the 7
If we ³ diagonalize Q and rewrite the ´ covariance of y• , the degeneracy becomes clear: 1/2
y• ∼ N 0, (CEΛ CE.
1/2 T )
)(CEΛ
+ R . To make Q diagonal, we simply replace C with
A Unifying Review of Linear Gaussian Models
+
0
w
x
C
315
+
y
v
v
y C w
x
Figure 2: Static generative model (continuous state).The covariance matrix of the input noise w is Q and the covariance matrix of the output noise v is R. In the network model below, the smaller circles represent noise sources and all units are linear. Outgoing weights have only been drawn from one hidden unit. This model is equivalent to factor analysis, SPCA and PCA models depending on the output noise covariance. For factor analysis, Q = I and R is diagonal. For SPCA, Q = I and R = αI. For PCA, Q = I and R = lim²→0 ²I.
structure in the data as noise. (Remember that since the model has reduced to a single gaussian distribution for y• , we can do no better than having the covariance of our model equal the sample covariance of our data.) Note that restricting R, unlike making Q diagonal, does constitute some loss of generality from the original model of equations 5.1. There is an intuitive spatial way to think about this static generative model. We use white noise to generate a spherical ball (since Q = I) of density in k-dimensional state-space. This ball is then stretched and rotated into p-dimensional observation space by the matrix C, where it looks like a k-dimensional pancake. The pancake is then convolved with the covariance density of v• (described by R) to get the final covariance model for y• . We want the resulting ellipsoidal density to be as close as possible to the ellipsoid given by the sample covariance of our data. If we restrict the shape of the v• covariance by constraining R, we can force interesting information to appear in both R and C as a result. Finally, observe that all varieties of filtering and smoothing reduce to the same problem in this static model because there is ¢ no time dependence. We ¡ are seeking only the posterior probability P x• |y• over a single hidden state given the corresponding single observation. This inference is easily done by
316
Sam Roweis and Zoubin Ghahramani
linear matrix projection, and the resulting density is itself gaussian: ¢ ¡ ¢ ¡ P y• |x• P (x• ) N (Cx• , R) |y• N (0, I) |x• ¡ ¢ ¢ ¡ = P x• |y• = P y• N 0, CCT + R |y• ¢ ¢ ¡ ¡ β = CT (CCT + R)−1 , P x• |y• = N β y• , I − β C |x• ,
(5.3a) (5.3b)
from which we obtain not only the expected value β y• of the unknown state but also an estimate of the uncertainty in this value in the form of the covariance I − β C. Computing the likelihood of a data point y• is merely an evaluation under the gaussian in equation 5.2. The learning problem now consists of identifying the matrices C and R. There is a family of EM algorithms to do this for the various cases discussed below, which are given in detail at the end of this review. 5.2 Factor Analysis. If we restrict the covariance matrix R that controls the observation noise to be diagonal (in other words, the covariance ellipsoid of v• is axis aligned) and set the state noise Q to be the identity matrix, then we recover exactly a standard statistical model known as maximum likelihood factor analysis. The unknown states x are called the factors in this context; the matrix C is called the factor loading matrix, and the diagonal elements of R are often known as the uniquenesses. (See Everitt, 1984, for a brief and clear introduction.) The inference calculation is done exactly as in equation 5.3b. The learning algorithm for the loading matrix and the uniquenesses is exactly an EM algorithm except that we must take care to constrain R properly (which is as easy as taking the diagonal of the unconstrained maximum likelihood estimate; see Rubin & Thayer, 1982; Ghahramani & Hinton, 1997). If C is completely free, this procedure is called exploratory factor analysis; if we build a priori zeros into C, it is confirmatory factor analysis. In exploratory factor analysis, we are trying to model the covariance structure of our data with p + pk − k(k − 1)/2 free parameters8 instead of the p(p + 1)/2 free parameters in a full covariance matrix. The diagonality of R is the key assumption here. Factor analysis attempts to explain the covariance structure in the observed data by putting all the variance unique to each coordinate in the matrix R and putting all the correlation structure into C (this observation was first made by Lyttkens, 1966, in response to work by Wold). In essence, factor analysis considers the axis rotation in which the original data arrived to be special because observation noise (often called sensor noise) is independent along the coordinates in these axes. However, the original scaling of the coordinates is unimportant. If we were to change the units in which we measured some of the components of y, factor analysis could merely rescale the corresponding entry in R and 8 The correction k(k − 1)/2 comes in because of degeneracy in unitary transformations of the factors. See, for example, Everitt (1984).
A Unifying Review of Linear Gaussian Models
317
row in C and achieve a new model that assigns the rescaled data identical likelihood. On the other hand, if we rotate the axes in which we measure the data, we could not easily fix things since the noise v is constrained to have axis aligned covariance (R is diagonal). EM for factor analysis has been criticized as being quite slow (Rubin & Thayer, 1982). Indeed, the standard method for fitting a factor analysis model (Joreskog, ¨ 1967) is based on a quasi-Newton optimization algorithm (Fletcher & Powell, 1963), which has been found empirically to converge faster than EM. We present the EM algorithm here not because it is the most efficient way of fitting a factor analysis model, but because we wish to emphasize that for factor analysis and all the other latent variable models reviewed here, EM provides a unified approach to learning. Finally, recent work in online learning has shown that it is possible to derive a family of EM-like algorithms with faster convergence rates than the standard EM algorithm (Kivinen & Warmuth, 1997; Bauer, Koller, & Singer, 1997). 5.3 SPCA and PCA. If instead of restricting R to be merely diagonal, we require it to be a multiple of the identity matrix (in other words, the covariance ellipsoid of v• is spherical), then we have a model that we will call sensible principal component analysis (SPCA) (Roweis, 1997). The columns of C span the principal subspace (the same subspace found by PCA), and we will call the scalar value on the diagonal of R the global noise level. Note that SPCA uses 1 + pk − k(k − 1)/2 free parameters to model the covariance. Once again, inference is done with equation 5.3b and learning by the EM algorithm (except that we now take the trace of the maximum likelihood estimate for R to learn the noise level; see (Roweis, 1997)). Unlike factor analysis, SPCA considers the original axis rotation in which the data arrived to be unimportant: if the measurement coordinate system were rotated, SPCA could (left) multiply C by the same rotation, and the likelihood of the new data would not change. On the other hand, the original scaling of the coordinates is privileged because SPCA assumes that the observation noise has the same variance in all directions in the measurement units used for the observed data. If we were to rescale one of the components of y, the model could not be easily corrected since v has spherical covariance (R = ²I). The SPCA model is very similar to the independently proposed probabilistic principal component analysis (Tipping & Bishop, 1997). If we go even further and take the limit R = lim²→0 ²I (while keeping the diagonal elements of Q finite)9 then we obtain the standard principal component analysis (PCA) model. The directions of the columns of C are
9 Since isotropic scaling of the data space is arbitrary, we could just as easily take the limit as the diagonal elements of Q became infinite while holding R finite or take both limits at once. The idea is that the noise variance becomes infinitesimal compared to the scale of the data.
318
Sam Roweis and Zoubin Ghahramani
known as the principal components. Inference now reduces to simple least squares projection:10 ¡ ¢ P(x• |y• ) = N β y• , I − β C |x• , β = lim CT (CCT + ²I)−1 ²→0 ´ ³ T −1 T P(x• |y• ) = N (C C) C y• , 0 |x• = δ(x• − (CT C)−1 CT y• ).
(5.4a)
(5.4b)
Since the noise has become infinitesimal, the posterior over states collapses to a single point, and the covariance becomes zero. There is still an EM algorithm for learning (Roweis, 1997), although it can learn only C. For PCA, we could just diagonalize the sample covariance of the data and take the leading k eigenvectors multiplied by their eigenvalues to be the columns of C. This approach would give us C in one step but has many problems.11 The EM learning algorithm amounts to an iterative procedure for finding these leading eigenvectors without explicit diagonalization. An important final comment is that (regular) PCA does not define a proper density model in the observation space, so we cannot ask directly about the likelihood assigned by the model to some data. We can, however, examine a quantity that is proportional to the negative log-likelihood in the limit of zero noise. This is the sum squared deviation of each data point from its projection. It is this “cost” that the learning algorithm ends up minimizing and is the only available evaluation of how well a PCA model fits new data. This is one of the most critical failings of PCA: translating points by arbitrary amounts inside the principal subspace has no effect on the model error.
10 Recall that if C is p×k with p > k and is rank k, then left multiplication by CT (CCT )−1 (which appears not to be well defined because CCT is not invertible) is exactly equivalent to left multiplication by (CT C)−1 CT . This is the same as the singular value decomposition idea of defining the “inverse” of the diagonal singular value matrix as the inverse of an element unless it is zero, in which case it remains zero. The intuition is that although CCT truly is not invertible, the directions along which it is not invertible are exactly those that CT is about to project out. 11 It is computationally very hard to diagonalize or invert large matrices. It also requires an enormous amount of data to make a large sample covariance matrix full rank. If we are working with patterns in a large (thousands) number of dimensions and want to extract only a few (tens) principal components, we cannot naively try to diagonalize the sample covariance of our data. Techniques like the snapshot method (Sirovich, 1987) attempt to address this but still require the diagonalization of an N × N matrix where N is the number of data points. The EM algorithm approach solves all of these problems, requiring no explicit diagonalization whatsoever and the inversion of only a k×k matrix. It is guaranteed to converge to the true principal subspace (the same subspace spanned by the principal components). Empirical experiments (Roweis, 1997) indicate that it converges in a few iterations, unless the ratio of the leading eigenvalues is near unity.
A Unifying Review of Linear Gaussian Models
319
5.4 Time-Series Modeling: Kalman Filter Models. We use the term dynamic data to refer to observation sequences in which the temporal ordering is important. For such data, we do not want to ignore the state evolution dynamics, which provides the only aspect of the model capable of capturing temporal structure. Systems described by the original dynamic generative model, shown in equations 2.1a and 2.2b, are known as linear dynamical systems or Kalman filter models and have been extensively investigated by the engineering and control communities for decades. The emphasis has traditionally been on inference problems: the famous discrete Kalman filter (Kalman, 1960; Kalman & Bucy, 1961) gives an efficient recursive solution to the optimal filtering and likelihood computation problems, while the RTS recursions (Rauch, 1963; Rauch et al., 1965) solve the optimal smoothing problem. Learning of unknown model parameters was studied by Shumway and Stoffer (1982) (C known) and by Ghahramani and Hinton (1996a) and Digalakis et al. (1993) (all parameters unknown). Figure 1 illustrates this model, and the appendix gives pseudocode for its implementation. We can extend our spatial intuition of the static case to this dynamic model. As before, any point in state-space is surrounded by a ball (or ovoid) of density (described by Q), which is stretched (by C) into a pancake in observation space and then convolved with the observation noise covariance (described by R). However, unlike the static case, in which we always centered our ball of density on the origin in state-space, the center of the state-space ball now “flows” from time step to time step. The flow is according to the field described by the eigenvalues and eigenvectors of the matrix A. We move to a new point according to this flow field; then we center our ball on that point and pick a new state. From this new state, we again flow to a new point and then apply noise. If A is the identity matrix (not the zero matrix), then the “flow” does not move us anywhere, and the state just evolves according to a random walk of the noise set by Q. 6 Discrete-State Linear Gaussian Models We now consider a simple modification of the basic continuous state model in which the state at any time takes on one of a finite number of discrete values. Many real-world processes, especially those that have distinct modes of operation, are better modeled by internal states that are not continuous. (It is also possible to construct models that have a mixed continuous and discrete state.) The state evolution is still first-order Markovian dynamics, and the observation process is still linear with additive gaussian noise. The modification involves the use of the winner-take-all nonlinearity WTA[·], defined such that WTA[x] for any vector x is a new vector with unity in the position of the largest coordinate of the input and zeros in all other positions. The discrete-state generative model is now simply: xt+1 = WTA[Axt + wt ] = WTA[Axt + w• ] yt = Cxt + vt = Cxt + v•
(6.1a) (6.1b)
320
Sam Roweis and Zoubin Ghahramani
x
t
A
z ,1
WTA[ ]
C
+
y
t
v
+
w
v y
t
C x
w
t
A
z ,1
Figure 3: Discrete state generative model for dynamic data. The WTA[·] block implements the winner-take-all nonlinearity. The z−1 block is a unit delay. The covariance matrix of the input noise w is Q and the covariance matrix of the output noise v is R. In the network model below, the smaller circles represent noise sources and the hidden units x have a winner take all behaviour (indicated by dashed lines). Outgoing weights have only been drawn from one hidden unit. This model is equivalent to a hidden Markov model with tied output covariances.
where A is no longer known as the state transition matrix (although we will see that matrix shortly). As before, the k-vector w and p-vector v are temporally white and spatially gaussian distributed noises independent of each other and of x and y. The initial state x1 is generated in the obvious way: ¡ ¢ (6.2) x1 = WTA[N µ1 , Q1 ] (though we will soon see that without loss of generality Q1 can be restricted to be the identity matrix). This discrete state generative model is illustrated in Figure 3. 6.1 Static Data Modeling: Mixtures of Gaussians and Vector Quantization. Just as in the continuous-state model, we can consider situations in
A Unifying Review of Linear Gaussian Models
321
which there is no natural ordering to our data, and so set the matrix A to be the zero matrix. In this discrete-state case, the generative model becomes: A=0
⇒
x• = WTA[w• ]
w• ∼ N (µ, Q)
(6.3)
y• = Cx• + v•
v• ∼ N (0, R) .
(6.4)
Each state x• is generated independently12 according to a fixed discrete probability histogram ¡ ¢ controlled by the mean and covariance of w• . Specifically, πj = P x• = ej is the probability assigned by the gaussian N (µ, Q) to the region of k-space in which the jth coordinate is larger than all others. (Here ej is the unit vector along the jth coordinate direction.) Notice that to obtain nonuniform priors πj with the WTA[·] nonlinearity, we require a nonzero mean µ for the noise w• . Once the state has been chosen, the corresponding output y• is generated from a gaussian whose mean is the jth column of C and whose covariance is R. This is exactly the standard mixture of gaussian clusters model except that the covariances ¡of all the ¢ clusters are constrained to be the same. The probabilities πj = P x• = ej correspond to the mixing coefficients of the clusters, and the columns of C are the cluster means. Constraining R in various ways corresponds to constraining the shape of the covariance of the clusters. This model is illustrated in Figure 4. To compute the likelihood of a data point, we can explicitly perform the sum equivalent to the integral in equation 4.1 since it contains only k terms, k k ¢ X ¡ ¡ ¢ X P x• = ej , y• = N (Ci , R) |y• P (x• = ei ) P y• = i=1
=
k X
i=1
N (Ci , R) |y• π i ,
(6.5)
i=1
where Ci denotes the ith column of C. Again, all varieties of inference and filtering and we are simply seeking the set of discrete probabil¡ are the same, ¢ ities P x• = ej |y• j = 1, . . . , k. In other words, we need to do probabilistic classification. The problem is easily solved by computing the responsibilities xˆ • that each cluster has for the data point y• : ¡ ¡ ¢ ¢ ¢ P x• = ej , y• ¡ P x• = ej , y• ¢ ¡ = Pk (ˆx• )j = P x• = ej |y• = ¡ ¢ P y• i=1 P x• = ei , y• ¡ ¢ ¢ ¡ N Cj , R |y• P x• = ej (ˆx• )j = Pk i=1 N (Ci , R) |y• P (x• = ei )
(6.6a)
12 As in the continuous static case, we again dispense with any special treatment of the initial state.
322
Sam Roweis and Zoubin Ghahramani
+ WTA[ ] x C
0
w
y
+
v
v
y x
w
C
Figure 4: Static generative model (discrete state). The WTA[·] block implements the winner-take-all nonlinearity. The covariance matrix of the input noise w is Q and the covariance matrix of the output noise v is R. In the network model below, the smaller circles represent noise sources and the hidden units x have a winner take all behaviour (indicated by dashed lines). Outgoing weights have only been drawn from one hidden unit. This model is equivalent to a mixture of Gaussian clusters with tied covariances R or to vector quantization (VQ) when R = lim²→0 ²I.
¡ ¢ N Cj , R |y• πj
= Pk
i=1 N
(Ci , R) |y• π i
.
(6.6b)
The mean xˆ • of the state vector given a data point is exactly the vector of responsibilities for that data point. This quantity defines the entire posterior distribution of the discrete hidden state given the data point. As a measure of the randomness or uncertainty in the hidden state, one could evaluate the entropy or normalized entropy13 of the discrete distribution corresponding to xˆ • . Although this may seem related to the variance of the posterior in factor analysis, this analogy is deceptive. Since xˆ • defines the entire distribution, no other “variance” measure is needed. Learning consists of finding the cluster means (columns of C), the covariance R, and the mixing coefficients πj . This is easily done with EM and corresponds exactly to maximum likelihood competitive learning (Duda & Hart, 1973; Nowlan, 1991), except
13 The entropy of the distribution divided by the logarithm of k so that it always lies between zero and one.
A Unifying Review of Linear Gaussian Models
323
that all the clusters share the same covariance. Later we introduce extensions to the model that remove this restriction. As in the continuous-state case, we can consider the limit as the observation noise becomes infinitesimal compared to the scale of the data. What results is the standard vector quantization model. The inference (classification) problem is now solved by the one-nearest-neighbor rule, using Euclidean distance if R is a multiple of the identity matrix, or Mahalanobis distance in the unscaled matrix R otherwise. Similarly to PCA, since the observation noise has disappeared, the posterior collapses to have all of its mass on one cluster (the closest), and the corresponding uncertainty (entropy) becomes zero. Learning with EM is equivalent to using a batch version of the k-means algorithm such as that proposed by Lloyd (1982). As with PCA, vector quantization does not define a proper density in the observation space. Once again, we examine the sum squared deviation of each point from its closest cluster center as a quantity proportional to the likelihood in the limit of zero noise. Batch k-means algorithms minimize this cost in lieu of maximizing a proper likelihood. 6.2 Time-Series Modeling: Hidden Markov Models. We return now to the fully dynamic discrete-state model introduced in equations 6.1a and 6.2b. Our key observation is that the dynamics described by equation 6.1a are exactly equivalent to the more traditional discrete ¡ Markov chain¢dynamics using a state transition matrix T, where Tij = P xt+1 = ej |xt = ei . It is easy to see how to compute the equivalent state transition matrix T given A and Q above: Tij is the probability assigned by the gaussian whose mean is the ith column of A (and whose covariance is Q) to the region of k-space in which the jth coordinate is larger than all others. It is also true that for any transition matrix T (whose rows each sum to unity), there exist matrices A and Q such that the dynamics are equivalent.14 Similarly, the initial probability mass function for x1 is easily computed from µ1 and Q1 and for any desired histogram over the states for x1 there exist a µ1 and Q1 that achieve it. Similar degeneracy exists in this discrete-state model as in the continuousstate model except that it is now between the structure of A and Q. Since for any noise covariance Q, the means in the columns of A can be chosen to set any equivalent transition probabilities Tij , we can without loss of generality restrict Q to be the identity matrix and use only the means in the columns
14 Although harder to see. Sketch of proof: Without loss of generality, always set the covariance to the identity matrix. Next, set the dot product of the mean vector with the k-vector having unity in all positions to be zero since moving along this direction does not change the probabilities. Now there are (k − 1) degrees of freedom in the mean and also in the probability model. Set the mean randomly at first (except that it has no projection along the all-unity direction). Move the mean along a line defined by the constraint that all probabilities but two should remain constant until one of those two probabilities has the desired value. Repeat this until all have been set correctly.
324
Sam Roweis and Zoubin Ghahramani
of A to set probabilities. Equivalently, we can restrict Q1 = I and use only the mean µ1 to set the probabilities for the initial state x1 . Thus, this generative model is equivalent to a standard HMM except that the emission probability densities are all constrained to have the same covariance. Likelihood and filtering computations are performed with the socalled forward (alpha) recursions, while complete smoothing is done with the forward-backward (alpha-beta) recursions. The EM algorithm for learning is exactly the well-known Baum-Welch reestimation procedure (Baum & Petrie, 1966; Baum et al., 1970). There is an important and peculiar consequence of discretizing the state that affects the smoothing problem. The state sequence formed by taking the most probable state of the posterior distribution at each time (as computed by the forward-backward recursions given the observed data and model parameters) is not the single state sequence most likely to have produced the observed data. In fact, the sequence of states obtained by concatenating the states that individually have maximum posterior probability at each time step may have zero probability under the posterior. This creates the need for separate inference algorithms to find the single most likely state sequence given the observations. Such algorithms for filtering and smoothing are called Viterbi decoding methods (Viterbi, 1967). Why was there no need for similar decoding in the continuous-state case? It turns out that due to the smooth and unimodal nature of the posterior probabilities for individual states in the continuous case (all posteriors are gaussian), the sequence of maximum a posteriori states is exactly the single most likely state trajectory, so the regular Kalman filter and RTS smoothing recursions suffice. It is possible (see, for example, Rabiner & Juang, 1986) to learn the discrete-state model parameters based on the results of the Viterbi decoding instead of the forward-backward smoothing—in other words, to maximize the joint likelihood of the observations and the single most likely state sequence rather than the total likelihood summed over all possible paths through state-space.
7 Independent Component Analysis There has been a great deal of recent interest in the blind source separation problem that attempts to recover a number of “source” signals from observations resulting from those signals, using only the knowledge that the original sources are independent. In the “square-linear” version of the problem, the observation process is characterized entirely by a square and invertible matrix C. In other words, there are as many observation streams as sources, and there is no delay, echo, or convolutional distortion. Recent experience has shown the surprising result that for nongaussian distributed sources, this problem can often be solved even with no prior knowledge about the sources or about C. It is widely believed (and beginning to be proved theo-
A Unifying Review of Linear Gaussian Models
325
retically; see MacKay, 1996) that high kurtosis source distributions are most easily separated. We will focus on a modified, but by now classic, version due to Bell and Sejnowski (1995) and Baram and Roth (1994) of the original independent component analysis algorithm (Comon, 1994). Although Bell and Sejnowski derived it from an information-maximization perspective, this modified algorithm can also be obtained by defining a particular prior distribution over the components of the vector xt of sources and then deriving a gradient learning rule that maximizes the likelihood of the data yt in the limit of zero output noise (Amari, Cichocki, & Yang, 1996; Pearlmutter & Parra, 1997; MacKay, 1996). The algorithm, originally derived for unordered data, has also been extended to modeling time series (Pearlmutter & Parra, 1997). We now show that the generative model underlying ICA can be obtained by modifying slightly the basic model we have considered thus far. The modification is to replace the WTA[·] nonlinearity introduced above with a general nonlinearity g(·) that operates componentwise on its input. Our generative model (for static data) then becomes: x• = g(w• )
w• ∼ N (0, Q)
(7.1a)
y• = Cx• + v•
v• ∼ N (0, R) .
(7.1b)
The role of the nonlinearity is to convert the gaussian distributed prior for w• into a nongaussian prior for x• . Without loss of generality, we can set Q = I, since any covariance structure in Q can be be obtained by a linear transformation of a N (0, I) random variable, and this linear transformation can be subsumed into the nonlinearity g(·). Assuming that the generative nonlinearity g(·) is invertible and differentiable, any choice of the generative nonlinearity results in a corresponding prior distribution on each source given by the probability density function: px (x) =
N (0, 1) |g−1 (x) . |g0 (g−1 (x))|
(7.2)
It is important to distinguish this generative nonlinearity from the nonlinearity found in the ICA learning rule. We call this the learning rule nonlinearity, f (·), and clarify the distinction between the two nonlinearities below. Classic ICA is defined for square and invertible C in the limit of vanishing noise, R = lim²→0 ²I. Under these conditions, the posterior density of x• given y• is a delta function at x• = C−1 y• , and the ICA algorithm can be defined in terms of learning the recognition (or unmixing) weights W = C−1 , rather than the generative (mixing) weights C. The gradient learning rule to increase the likelihood is 1W ∝ W−T + f (Wy• )yT• ,
(7.3)
326
Sam Roweis and Zoubin Ghahramani
where the learning rule nonlinearity f (·) is the derivative of the implicit d log px (x) log prior: f (x) = (MacKay, 1996). Therefore, any generative nondx linearity g(·) results in a nongaussian prior px (·), which in turn results in a nonlinearity f (·) in the maximum likelihood learning rule. Somewhat frustratingly from the generative models perspective, ICA is often discussed in terms of the learning rule nonlinearity without any reference to the implicit prior over the sources. A popular choice for the ICA learning rule nonlinearity f (·) is the tanh(·) function, which corresponds to a heavy tailed prior over the sources (MacKay, 1996): px (x) =
1 . π cosh(x)
(7.4)
From equation 7.2, we obtain a general relationship between the cumulative distribution function of the prior on the sources, cdfx (x), and of the zeromean, unit variance noise w, cdfx (g(w)) = cdfw (w) =
√ 1 1 + erf(w/ 2), 2 2
(7.5)
√ Rz 2 for monotonic g, where erf(z) is the error function 2/ π 0 e−u du. This relationship can often be solved to obtain an expression for g. For example, 1 , we find that setting if px (x) = π cosh(x) ³ ³π ³ √ ´´´ 1 + erf(w/ 2) g(w) = ln tan 4
(7.6)
causes the generative model of equations 7.1 to generate vectors x in which each component is distributed exactly according to 1/(π cosh(x)). This nonlinearity is shown in Figure 5. ICA can be seen either as a linear generative model with nongaussian priors for the hidden variables or as a nonlinear generative model with gaussian priors for the hidden variables. It is therefore possible to derive an EM algorithm for ICA, even when the observation noise R is nonzero and there are fewer sources than observations. The only complication is that the posterior distribution of x• given y• will be the product of a nongaussian prior and a gaussian likelihood term, which can be difficult to evaluate. Given this posterior, the M step then consists of maximizing the expected log of the joint probability as a function of C and R. The M-step for C is C ← arg max C
X i
® log P(x) + log P(yi |x, C, R) i ,
(7.7)
where i indexes the data points and h·ii denotes expectation with respect to the posterior distribution of x given yi , P(x|yi , C, R). The first term does not
A Unifying Review of Linear Gaussian Models
327
15 10
g(w)
5 0 −5 −10 −15 −5 −4 −3 −2 −1
0 w
1
2
3
4
5
Figure 5: The nonlinearity g(·) from equation 7.6 which converts a gaussian distributed source w ∼ N (0, 1) into one distributed as x = g(w) ∼ 1/(π cosh(x)).
depend on C, and the second term is a quadratic in C, so taking derivatives with respect to C, we obtain a linear system of equations that can be solved in the usual manner: Ã C←
X i
!Ã T
yi hx ii
X hxxT ii
!−1 .
(7.8)
i
A similar M-step can be derived for R. Since, given x• , the generative model is linear, the M-step requires only evaluating the first and second moments of the posterior distribution of x: hxii and hxxT ii . It is not necessary to know anything else about the posterior if its first two moments can be computed. These may be computed using Gibbs sampling or, for certain source priors, using table lookup or closed-form computation.15 In particular, Moulines et al. (1997) and Attias and Schreiner (1998) have independently proposed using a gaussian mixture to model the prior for each component of the source, x. The posterior distribution over x is then also a gaussian mixture, which can be evaluated analytically and used to derive an EM algorithm for both the mixing matrix and the source densities. The only caveat is that the number of gaussian components in the posterior grows exponentially in the number of sources,16 which limits the applicability of this method to models with only a few sources. 15 In the limit of zero noise, R = 0, the EM updates derived in this manner degenerate to C ← C and R ← R. Since this does not decrease the likelihood, it does not contradict the convergence proof for the EM algorithm. However, it also does not increase the likelihood, which might explain why no one uses EM to fit the standard zero-noise ICA model. 16 If each source is modeled as a mixture of k gaussians and there are m sources, then there are km components in the mixture.
328
Sam Roweis and Zoubin Ghahramani
Alternatively, we can compute the posterior distribution of w• given y• , which is the product of a gaussian prior and a nongaussian likelihood. Again, this may not be easy, and we may wish to resort to Gibbs sampling (Geman & Geman, 1984) or other Markov chain Monte Carlo methods (Neal, 1993). Another option is to employ a deterministic trick recently used by Bishop, Svenson, and Williams (1998) in the context of the generative topographic map (GTM), which is a probabilistic version of Kohonen’s (1982) self-organized topographic map. We approximate the gaussian prior via a finite number (N) of fixed points (this is the trick). In other words, ˜ •) = P(w• ) = N (0, I) ≈ P(w
N X
δ(w• − wj ),
(7.9)
j=1
where the wj ’s are a finite sample from N (0, I). The generative model then takes these N points, maps them through a fixed nonlinearity g, an adaptable linear mapping C, and adds gaussian noise with covariance R to produce y• . The generative model is therefore a constrained mixture of N gaussians, where the constraint comes from the fact that the only way the centers can move is by varying C. Then, computing the posterior over w• amounts to computing the responsibility under each of the N gaussians for the data point. Given these responsibilities, the problem is again linear in C, which means that it can be solved using equation 7.8. For the traditional zero-noise limit of ICA, the responsibilities will select the center closest to the data point in exactly the same manner as standard vector quantization. Therefore, ICA could potentially be implemented using EM for GTMs in the limit of zero output noise. 8 Network Interpretations and Regularization Early in the modern history of neural networks, it was realized that PCA could be implemented using a linear autoencoder network (Baldi & Hornik, 1989). The data are fed as both the input and target of the network, and the network parameters are learned using the squared error cost function. In this section, we show how factor analysis and mixture of gaussian clusters can also be implemented in this manner, albeit with a different cost function. To understand how a probabilistic model can be learned using an autoencoder it is very useful to make a recognition-generation decomposition of the autoencoder (Hinton & Zemel, 1994; Hinton, Dayan, & Revow, 1997). An autoencoder takes an input y• , produces some internal representation in the hidden layer xˆ • , and generates at its output a reconstruction of the input yˆ • in Figure 6. We call the mapping from hidden to output layers the generative part of the network since it generates the data from a (usually more compact) representation under some noise model. Conversely, we call the mapping from input to hidden units the recognition part of the network
A Unifying Review of Linear Gaussian Models
generative weights recognition weights
329
x^
C
y^
y
Figure 6: A network for state inference and for learning parameters of a static data model. The input y• is clamped to the input units (bottom), and the mean xˆ • of the posterior of the estimated state appears on the hidden units above. The covariance of the state posterior is constant at I − βC which is easily computed if the weights β are known. The inference computation is a trivial linear projection, but learning the weights of the inference network is difficult. The input to hidden weights are always constrained to be a function of the hidden to output weights, and the network is trained as an autoencoder using self-supervised learning. Outgoing weights have only been drawn from one input unit and one hidden unit.
since it produces some representation in the hidden variables given the input. Because autoencoders are usually assumed to be deterministic, we will think of the recognition network as computing the posterior mean of the hidden variables given the input. The generative model for factor analysis assumes that both the hidden states and the observables are normally distributed, from which we get the posterior probabilities for the hidden states in equation 5.3b. If we assume that the generative weight matrix from the hidden units to the outputs is C and the noise model at the output is gaussian with covariance R, then the posterior mean of the hidden variables is xˆ • = β y• , where β = CT (CCT + R)−1 . Therefore, the hidden units can compute the posterior mean exactly if they are linear and the weight matrix from input to hidden units is β . Notice that β is tied to C and R, so we only need to estimate C and R during learning. We denote expectations under the posterior state distribution by h·i, for example, Z hx• i = x• P(x• |y• ) dx• = xˆ • . From the theory of the EM algorithm (see section 4.2), we know that one way to maximize the likelihood is to maximize the expected value of the log of the joint probability under the posterior distribution of the hidden variables: hlog P(x• , y• |C, R)i.
330
Sam Roweis and Zoubin Ghahramani
Changing signs and ignoring constants, we can equivalently minimize the following cost function:
C = h(y• − Cx• )T R−1 (y• − Cx• )i + log |R| = yT• R−1 y• − 2yT• R−1 Chx• i + hxT• CT R−1 Cx• i + log |R| = (y• − Cˆx• )T R−1 (y• − Cˆx• ) + log |R| + trace[CT R−1 CΣ].
(8.1a) (8.1b) (8.1c)
Here we have defined Σ to be the posterior covariance of x• ,
Σ ≡ hx• xT• i − hx• ihx• iT = I − β C, and in the last step we have reorganized terms and made use of the fact that hxT• CT R−1 Cx• i = trace[CT R−1 Chx• xT• i]. The first two terms of cost function in equation 8.1c are just a squared error cost function evaluated with respect to a gaussian noise model with covariance R. They are exactly the terms minimized when fitting a standard neural network with this gaussian noise model. The last term is a regularization term that accounts for the posterior variance in the hidden states given the inputs.17 When we take derivatives of this cost function, we do not differentiate xˆ and Σ with respect to C and R. As is usual for the EM algorithm, we differentiate the cost given the posterior distribution of the hidden variables. Taking derivatives with respect to C and premultiplying by R, we obtain a weight change rule, 1C ∝ (y• − Cx• )xT• − CΣ.
(8.2)
The first term is the usual delta rule. The second term is simply a weightdecay term decaying the columns of C with respect to the posterior covariance of the hidden variables. Intuitively, the higher the uncertainty in a hidden unit, the more its outgoing weight vector is shrunk toward zero. To summarize, factor analysis can be implemented in an autoassociator by tying the recognition weights to the generative weights and using a particular regularizer in addition to squared reconstruction error during learning. We now analyze the mixture of gaussians model in the same manner. The recognition network is meant to produce the mean of the hidden variable given the inputs. Since we assume that the discrete hidden variable is represented as a unit vector, its mean is just the vector of probabilities of being in each of its k settings given the inputs, that is, the responsibilities. Assuming equal mixing coefficients, P(x• = ej ) = P(x• = ei ) ∀ij, the responsibilities
17 PCA assumes infinitesimal noise, and therefore the posterior “distribution” over the hidden states has zero variance (Σ → 0) and the regularizer vanishes (CT R−1 CΣ → I).
A Unifying Review of Linear Gaussian Models
331
defined in equation 6.6b are exp{− 12 (y• − Cj )T R−1 (y• − Cj )} (ˆx• )j = P(x• = ej |y• ) = Pk 1 T −1 i=1 exp{− 2 (y• − Ci ) R (y• − Ci )} exp{βj y• − αj } , = Pk i=1 exp{β i y• − αi }
(8.3a)
(8.3b)
where we have defined βj = Cj R−1 and αj = 12 CjT R−1 Cj . Equation 8.3b describes a recognition model that is linear followed by the softmax nonlinearity, 8, written in full matrix notation: xˆ • = 8(β y• − α). In other words, a simple network could do exact inference with linear input to hidden weights β and softmax hidden units. Appealing again to the EM algorithm, we obtain a cost function that when minimized by an autoassociator will implement the mixture of gaussians.18 The log probability of the data given the hidden variables can be written as −2 log P(y• |x• , C, R) = (y• − Cx• )T R−1 (y• − Cx• ) + log |R| + const. Using this and the previous derivation, we obtain the cost function,
C = (y• − Cˆx• )T R−1 (y• − Cˆx• ) + log |R| + trace[CT R−1 CΣ],
(8.4)
where Σ = hx• xT• i − hx• ihx• iT . The second-order term, hx• xT• i, evaluates to a matrix with xˆ • along its diagonal and zero elsewhere. Unlike in factor analysis, Σ now depends on the input. To summarize, the mixture of gaussians model can also be implemented using an autoassociator. The recognition part of the network is linear, followed by a softmax nonlinearity. The cost function is the usual squared error penalized by a regularizer of exactly the same form as in factor analysis. Similar network interpretations can be obtained for the other probabilistic models. 9 Comments and Extensions There are several advantages, both theoretical and practical, to a unified treatment of the unsupervised methods reviewed here. From a theoretical viewpoint, the treatment emphasizes that all of the techniques for inference in the various models are essentially the same and just correspond to probability propagation in the particular model variation. Similarly, all the learning procedures are nothing more than an application of the EM 18 Our derivation assumes tied covariance and equal mixing coefficients. Slightly more complex equations result for the general case.
332
Sam Roweis and Zoubin Ghahramani
algorithm to increase the total likelihood of the observed data iteratively. Furthermore, the origin of zero-noise-limit algorithms such as vector quantization and PCA is easy to see. A unified treatment also highlights the relationship between similar questions across the different models. For example, picking the number of clusters in a mixture model or state dimension in a dynamical system or the number of factors or principal components in a covariance model or the number of states in an HMM are all really the same question. From a practical standpoint, a unified view of these models allows us to apply well-known solutions to hard problems in one area to similar problems in another. For example, in this framework it is obvious how to deal properly with missing data in solving both the learning and inference problems. This topic has been well understood for many static models (Little & Rubin, 1987; Tresp, Ahmad, & Neuneier, 1994; Ghahramani & Jordan, 1994) but is typically not well addressed in the linear dynamical systems literature. As another example, it is easy to design and work with models having a mixed continuous- and discrete-state vector, (for example, hidden filter HMMs (Fraser & Dimitriadis, 1993), which is something not directly addressed by the individual literatures on discrete or continuous models. Another practical advantage is the ease with which natural extensions to the basic models can be developed. For example, using the hierarchical mixture-of-experts formalism developed by Jordan and Jacobs (1994) we can consider global mixtures of any of the model variants discussed. In fact, most of these mixtures have already been considered: mixtures of linear dynamical systems are known as switching state-space models (see Shumway & Stoffer, 1991; Ghahramani & Hinton, 1996b); mixtures of factor analyzers (Ghahramani and Hinton, 1997) and of pancakes (PCA) (Hinton et al., 1995); and mixtures of HMMs (Smyth, 1997). A mixture of m of our constrained mixtures of gaussians each with k clusters gives a mixture model with mk components in which there are only m possible covariance matrices. This “tied covariance” approach is popular in speech modeling to reduce the number of free parameters. (For k = 1, this corresponds to a full “unconstrained” mixture of gaussians model with m clusters.) It is also possible ¢ to consider “local mixtures” in which the conditional ¡ probability P yt |xt is no longer a single gaussian but a more complicated density such as a mixture of gaussians. For our (constrained) mixture of gaussians model, this is another way to get a “full” mixture. For HMMs, this is a well-known extension and is usually the standard approach for emission density modeling (Rabiner & Juang, 1986). It is even possible to use constrained mixture models as the output density model for an HMM (see, for example, Saul & Rahim, 1998, which uses factor analysis as the HMM output density). However, we are not aware of any work that considers this variation in the continuous-state cases, for either static or dynamic data. Another important natural extension is spatially adaptive observation noise. The idea here is that the observation noise v can have different statis-
A Unifying Review of Linear Gaussian Models
333
tics in different parts of (state or observation) space rather than being described by a single matrix R. For discrete-mixture models, this idea is well known, and it is achieved by giving each mixture component a private noise model. However, for continuous-state models, this idea is relatively unexplored and is an interesting area for further investigation. The crux of the problem is how to parameterize a positive definite matrix over some space. We propose some simple ways to achieve this. One possibility is replacing the single covariance shape Q for the observation noise with a conic19 linear blending of k “basis” covariance shapes. In the case of linear dynamical systems or factor analysis, this amounts to a novel type of model in which the local covariance matrix R is computed as a conic linear combination of several “canonical” covariance matrices through a tensor product between the current state vector x (or equivalently the “noiseless” observation Cx) and a master noise tensor R.20 Another approach would be to drop the conic restriction (allow general linear combinations) and then add a multiple of the identity matrix to the resulting noise matrix in order to make it positive definite. A third approach is to represent the covariance shape as the compression of an elastic sphere due to a spatially varying force field. This representation is easier to work with because the parameterization of the field is unconstrained, but it is hard to learn the local field from measurements of the effective covariance shape. Bishop (1995, sec. 6.3) and others have considered simple nonparametric methods for estimating input-dependent noise levels in regression problems. Goldberg, Williams, and Bishop (1998) have explored this idea in the context of gaussian processes. It is also interesting to consider what happens to the dynamic models when the output noise tends to zero. In other words, what are the dynamic analogs of PCA and vector quantization? For both linear dynamical systems and HMMs, this causes the state to no longer be hidden. In linear dynamical systems, the optimal observation matrix is then found by performing PCA on the data and using the principal components as the columns of C; for HMMs, C is found by vector quantization of the data (using the codebook vectors as the columns of C). Given these observation matrices, the state is no longer hidden. All that remains is to identify a first-order Markov dynamics in state-space: this is a simple AR(1) model in the continuous case or a firstorder Markov chain in the discrete case. Such zero-noise limits are not only interesting models in their own right, but are also valuable as good choices for initialization of learning in linear dynamical systems and HMMs.
19
A conic linear combination is one in which all the coefficients are positive. For mixtures of gaussians or hidden Markov models, this kind of linear “blending” merely selects the jth submatrix of the tensor if the discrete state is ej . This is yet another way to recover the conventional “full” or unconstrained mixture of gaussians or hidden Markov model emission density in which each cluster or state has its own private covariance shape for observation noise. 20
334
Sam Roweis and Zoubin Ghahramani
Appendix In this appendix we review in detail the inference and learning algorithms for each of the models. The goal is to enable readers to implement these algorithms from the pseudocode provided. For each class of model, we first present the solution to the inference problem, and then the EM algorithm for learning the model parameters. For this appendix only, we adopt the notation that the transpose of a vector or matrix is written as x0 , not xT . We use T instead of τ to denote the length of a time series. We also define the binary operator ¯ to be the element-wise product of two equal-size vectors or matrices. Comments begin with the symbol %. A.1 Factor Analysis, SPCA, and PCA. A.1.1 Inference. For factor analysis and related models, the posterior probability of the hidden state given the observations, P(x• |y• ), is gaussian. The inference problem therefore consists of computing the mean and covariance of this gaussian, xˆ • and V = Cov[x• ]: FactorAnalysisInference(y• ,C,R) β ← C0 (CC0 + R)−1 xˆ • ← β y• V ← I − βC return xˆ • , V
Since the observation noise matrix R is assumed to be diagonal and x• is of smaller dimension than y• , β can be more efficiently computed using the matrix inversion lemma: ³ ´ β = C0 R−1 I − C(I + C0 R−1 C)−1 C0 R−1 . Computing the (log) likelihood of¡ an observation is nothing more than an ¢ evaluation under the gaussian N O, CC0 + R . The sensible PCA (SPCA) algorithm is a special case of factor analysis in which the observation noise is assumed to be spherically symmetric: R = αI. Inference in SPCA is therefore identical to inference for factor analysis. The traditional PCA algorithm can be obtained as a limiting case of factor analysis: R = lim²→0 ²I. The inverse used for computing β in factor analysis is no longer well defined. However, the limit of β is well defined: β = (C0 C)−1 C0 . Also, the posterior collapses to a single point, so V = Cov[x• ] = I − β C = 0. PCAInference(y• ,C) % Projection onto principal components β ← (C0 C)−1 C0 xˆ • ← β y• return xˆ •
A Unifying Review of Linear Gaussian Models
335
A.1.2 Learning. The EM algorithm for learning the parameters of a factor analyzer with k factors from a zero-mean data set Y = [y1 , . . . , yn ] (each column of the p × n matrix Y is a data point) is FactorAnalysisLearn(Y,k,² ) initialize C, R compute sample covariance S of Y while change in log likelihood > ² % E step ˆ V ← FactorAnalysisInference(Y,C,R) X,
δ ← YXˆ 0 γ ← Xˆ Xˆ 0 + nV % M step C ← δγ −1
set diagonal elements of R to Rii ← (S − Cδ 0 /n)ii
end return C, R
Here FactorAnalysisInference(Y,C,R) has the obvious interpretation of the inference function applied to the entire matrix of observations. Since β and V do not depend on Y, this can be computed efficiently in matrix form. Since the data appear only in outer products, we can run factor analysis learning with just the sample covariance. Note also that the log-likelihood is computed as − 12 y0 (CC0 + R)−1 y + n2 log |CC0 + R| + const. The EM algorithm for SPCA is identical to the EM algorithm for factor analysis, except that since the observation noise covariance is spherically Pp symmetrical, the M-step for R is changed to R ← αI, where α ← j=1 (S − Cδ 0 )jj /p. The EM algorithm for PCA can be obtained in a similar manner: PCALearn(Y,k,² ) initialize C while change in squared reconstruction error > ² % E step Xˆ ← PCAInference(Y,C) δ ← YXˆ 0 γ ← Xˆ Xˆ 0
% M step C ← δγ −1 end return C
336
Sam Roweis and Zoubin Ghahramani
Since PCA is not a probability model (it assumes zero noise), the likelihood is undefined, so convergence is assessed by monitoring the squared reconstruction error. A.2 Mixtures of Gaussians and Vector Quantization. A.2.1 Inference. We begin by discussing the inference problem for mixtures of gaussians and then discuss the inference problem in vector quantization as a limiting case. The hidden variable in a mixture of gaussians is a discrete variable that can take on one of k values. We represent this variable using a vector x• of length k, where each setting of the hidden variable corresponds to x• taking a value of unity in one dimension and zero elsewhere. The probability distribution of the discrete hidden variable, which has k − 1 degrees of freedom (since it must sum to one), is fully described by the mean of x• . Therefore, the inference problem is limited to computing the posterior mean of x• given a data point y• and the model parameters, which are π (the prior mean of x• ), C (the matrix whose k columns are the means of y• given each of the k settings of x• ) and R (the observation noise covariance matrix). MixtureofGaussiansInference(y• ,C,R,π ) % compute % responsibilities α←0 for i = 1 to k ∆i ← (y• − Ci )0 R−1 (y• − Ci ) © ª γ i ← π i exp − 12 ∆i α ← α + γi end xˆ • ← γ /α return xˆ •
A measure of the randomness of the hidden state can be obtained by evaluating the entropy of the discrete distribution corresponding to xˆ • . Standard VQ corresponds to the limiting case R = lim²→0 ²I and equal priors π i = 1/k. Inference in this case is performed by the well known (1-)nearest-neighbor rule. VQInference(y• ,C) % 1-nearest-neighbor for i = 1 to k ∆i ← (y• − Ci )0 (y• − Ci ) end xˆ • ← ej for j = arg min ∆i return xˆ •
A Unifying Review of Linear Gaussian Models
337
As before, ej is the unit vector along the jth coordinate direction. The squared distances ∆i can be generalized to a Mahalanobis metric with respect to some matrix R, and nonuniform priors π i can easily be incorporated. As was the case with PCA, the posterior distribution has zero entropy. A.2.2 Learning. The EM algorithm for learning the parameters of a mixture of gaussian is: MixtureofGaussiansLearn(Y,k,² )% ML soft competitive learning initialize C, R, π while change in log likelihood > ² initialize δ ← 0, γ ← 0, α ← 0 % E step for i = 1 to n xi ← MixtureofGaussiansInference(yi ,C,R,π ) δ ← δ + yi x0i γ ← γ + xi end % M step for j = 1 to k Cj ← δj /γ j for i = 1 to n α ← α + xij (yi − Cj )(yi − Cj )0 end end R ← α/n π ← γ /n end return C, R, π
We have assumed a common covariance matrix R for all the gaussians; the extension to different covariances for each gaussian is straightforward. The k-means vector quantization learning algorithm results when we take the appropriate limit of the above algorithm: VQLearn(Y,k,² ) % k-means initialize C while change in squared reconstruction error > ² % E step initialize δ ← 0, γ ← 0 for i = 1 to n xi ← VQInference(yi ,C) δ ← δ + yi x0i γ ← γ + xi
338
Sam Roweis and Zoubin Ghahramani
end % M step for j = 1 to k Cj ← δj /γ j end end return C
A.3 Linear Dynamical Systems. A.3.1 Inference. Inference in a linear dynamical system involves computing the posterior distributions of the hidden state variables given the sequence of observations. As in factor analysis, all the hidden state variables are assumed gaussian and are therefore fully described by their means and covariance matrices. The algorithm for computing the posterior means and covariances consists of two parts: a forward recursion, which uses the observations from y1 to yt , known as the Kalman filter (Kalman, 1960), and a backward recursion, which uses the observations from yT to yt+1 (Rauch, 1963). The combined forward and backward recursions are known as the Kalman or Rauch-Tung-Streibel (RTS) smoother. To describe the smoothing algorithm it will be useful to define the following quantities: xst and Vst are, respectively, the mean and covariance matrix ˆ t ≡ VT are the “full of xt given observations {y1 , . . . ys }; xˆ t ≡ xTt and V t smoother” estimates. To learn the A matrix using EM, it is also necessary to compute the covariance across time between xt and xt−1 : LDSInference(Y,A,C,Q,R,x01 ,V01 ) % Kalman smoother for t = 1 to T % Kalman filter (forward pass) ← Axt−1 xt−1 t t−1 if t > 1 0 ← AVt−1 Vt−1 t t−1 A + Q if t > 1 t−1 0 0 −1 Kt ← Vt−1 t C (CVt C + R)
xtt ← xt−1 + Kt (yt − Cxt−1 t t ) − Kt CVt−1 Vtt ← Vt−1 t t end ˆ T,T−1 = (I − KT C)AVT−1 initialize V T−1 for t = T to 2 % Rauch recursions (backward pass) t−1 −1 0 Jt−1 ← Vt−1 t−1 A (Vt )
xt − Axt−1 xˆ t−1 ← xt−1 t−1 + Jt−1 (ˆ t−1 ) t−1 ˆ t − Vt−1 )J0 ˆ t−1 ← V + Jt−1 (V V t−1
t
t−1
ˆ t,t−1 ← Vt J0 + Jt (V ˆ t+1,t − AVt )J0 V t t−1 t t−1 if t < T
A Unifying Review of Linear Gaussian Models
339
end ˆ t, V ˆ t,t−1 for all t return xˆ t , V
A.3.2 Learning. The EM algorithm for learning a linear dynamical system (Shumway & Stoffer, 1982; Ghahramani & Hinton, 1996a) is given below, assuming for simplicity that we have only a single sequence of observations: LDSLearn(Y,k,² ) initialize A, C, Q, R, x01 , V01 P set α ← t yt y0t while change in log likelihood > ² LDSInference(Y,A,C,Q,R,x01 ,V01 ) % E step initialize δ ← 0, γ ← 0, β ← 0 for t = 1 to T δ ← δ + yt xˆ 0t ˆt γ ← γ + xˆ t xˆ 0 + V t
ˆ t,t−1 if t > 1 β ← β + xˆ t xˆ 0t−1 + V end ˆT γ 1 ← γ − xˆ T xˆ 0T − V 0 ˆ1 γ 2 ← γ − xˆ 1 xˆ − V 1
% M step C ← δγ −1 R ← (α − Cδ 0 )/T A ← βγ −1 1 Q ← (γ 2 − Aβ 0 )/(T − 1) x01 ← xˆ 1 ˆ1 V0 ← V 1
end return A, C, Q, R, x01 , V01
A.4 Hidden Markov Models. A.4.1 Inference. The forward-backward algorithm computes the posterior probabilities of the hidden states in an HMM and therefore forms the basis of the inference required for EM. We use the following standard definitions,
αt = P(xt , y1 , . . . , yt )
(A.1a)
β t = P(yt+1 , . . . , yT |xt ),
(A.1b)
340
Sam Roweis and Zoubin Ghahramani
where both α and β are vectors of the same length as x. We present the case where the observations yt are real-valued p-dimensional vectors and the probability density of an observation given the corresponding state (the “output model”) is assumed to be a single gaussian with mean Cxt and covariance R. The parameters of the model are therefore a k × k transition matrix T, initial state prior probability vector π , an observation mean matrix C, and an observation noise matrix R that is tied across states: HMMInference(Y,T,π ,C,R) % forward–backward algorithm for t = 1 to T % forward pass for i = 1 to ©k ª bt,i ← exp − 12 (yt − Ci )0 R−1 (yt − Ci ) |R|−1/2 (2π )−p/2 end if (t = 1) αt ← π ¯ b1 else αt ← [T0 αt−1 ] ¯ bt end P ρt ← i αt,i αt ← αt /ρt end β T ← 1/ρT for t = T − 1 to 1 % backward pass β t ← T [β t+1 ¯ bt+1 ]/ρt end γ t ← αt ¯ β t /(α0t β t ) ξ t ← αt [β t+1 ¯ bt+1 ]0 P ξ t ← ξ t /( ij ξ tij ) return γ t , ξ t , ρt for all t
The definitions of α and β in equations A.1a and A.1b correspond to running the above algorithm without the scaling factors ρt . These factors, however, are essential to the numerical stability of the algorithm; otherwise, for long sequences, both α and β become vanishingly small. Furthermore, from the ρ’s we can compute the log-likelihood of the sequence
log P(y1 , . . . , yT ) =
T X
log ρt ,
t=1
which is why it is useful for the above function to return them. A.4.2 Learning. Again, we assume for simplicity that we have a single sequence of observations from which we wish to learn the parameters of an HMM. The EM algorithm for learning these parameters, known as the
A Unifying Review of Linear Gaussian Models
341
Baum-Welch algorithm, is: HMMLearn(Y,k,² ) % Baum-Welch initialize T, π , C, R while change in log likelihood > ² HMMInference(Y,T,π ,C,R) % E step % M step π ← γ1 P T ← t ξt P Tij ← Tij / ` Ti` for all i, j P P Cj ← t γ t,j yt / t γ t,j for all j P R ← t,j γ t,j (yt − Cj )(yt − Cj )0 /T return T, π , C, R
Acknowledgments We thank Carlos Brody, Sanjoy Mahajan, and Erik Winfree for many fruitful discussions in the early stages, the anonymous referees for helpful comments, and Geoffrey Hinton and John Hopfield for providing outstanding intellectual environments and guidance. S.R. was supported in part by the Center for Neuromorphic Systems Engineering as a part of the National Science Foundation Engineering Research Center Program under grant EEC9402726 and by the Natural Sciences and Engineering Research Council of Canada under an NSERC 1967 Award. Z.G. was supported by the Ontario Information Technology Research Centre. References Amari, S., Cichocki, A., & Yang, H. H. (1996). A new learning algorithm for blind signal separation. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 757–763). Cambridge, MA: MIT Press. Attias, H., & Schreiner, C. (1998). Blind source separation and deconvolution: The dynamic component analysis algorith. Neural Computation, 10, 1373– 1424. Baldi, P., & Hornik, K. (1989). Neural networks and principal components analysis: Learning from examples without local minima. Neural Networks, 2, 53–58. Baram, Y., & Roth, Z. (1994). Density shaping by neural networks with application to classification, estimation and forecasting (Tech. Rep. TR-CIS-94-20). Haifa, Israel: Center for Intelligent Systems, Technion, Israel Institute for Technology. Bauer, E., Koller, D., & Singer, Y. (1997). Update rules for parameter estimation in Bayesian networks. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence (UAI-97). San Mateo, CA: Morgan Kaufmann. Baum, L. E. (1972). An inequality and associated maximization technique in sta-
342
Sam Roweis and Zoubin Ghahramani
tistical estimation of probabilistic functions of a Markov process. Inequalities, 3, 1–8. Baum, L. E., & Eagon, J. A. (1967). An inequality with applications to statistical estimation for probabilistic functions of Markov processes and to a model for ecology. Bulletin of the American Mathematical Society, 73, 360–363. Baum, L. E., & Petrie, T. (1966). Statistical inference for probabilistic functions of finite state Markov chains. Annals of Mathematical Statistics, 37, 1554–1563. Baum, L. E., Petrie, T., Soulds, G., & Weiss, N. (1970). A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. Annals of Mathematical Statistics, 41(1), 164–171. Bell, A. J., & Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7(6), 1129– 1159. Bishop, C. (1995). Neural networks for pattern recognition. Oxford: Clarendon Press. Bishop, C. M., Svensen, M., & Williams, C. K. I. (1998). GTM: A principled alternative to the self-organizing map. Neural Computation, 10, 215–234. Comon, P. (1994). Independent component analysis: A new concept. Signal Processing, 36, 287–314. Delyon, B. (1993). Remarks on filtering of semi-Markov data (Tech. Rep. 733). Beaulieu, France: Institute de Recherche en Informatique et Systems Aleatiores. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society B, 39, 1–38. Digalakis, V., Rohlicek, J. R., & Ostendorf, M. (1993). ML estimation of a stochastic linear system with the EM algorithm and its application to speech recognition. IEEE Transactions on Speech and Audio Processing, 1(4), 431–442. Duda, R. O., & Hart, P. E. (1973). Pattern classification and scene analysis. New York: Wiley. Elliott, R. J., Aggoun, L., & Moore, J. B. (1995). Hidden Markov models: Estimation and control, New York: Springer-Verlag. Everitt, B. S. (1984). An introduction to latent variable models. London: Chapman and Hill. Fletcher, R., & Powell, M. J. D. (1963). A rapidly convergent descent method for minimization. Computing Journal, 2, 163–168. Fraser, A. M., & Dimitriadis, A. (1993). Forecasting probability densities by using hidden Markov models with mixed states. In A. S. Weigend & N. A. Gershenfeld (Eds.), Time series prediction: Forecasting the future and understanding the past (pp. 265–282). Reading, MA: Addison-Wesley. Geman, S., & Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 721–741. Ghahramani, Z., & Hinton, G. (1996a). Parameter estimation for linear dynamical systems (Tech. Rep. CRG-TR-96-2). Toronto: Department of Computer Science, University of Toronto. Available from ftp://ftp.cs.toronto.edu/ pub/zoubin/. Ghahramani, Z., & Hinton, G. (1996b). Switching state-space models (Tech. Rep.
A Unifying Review of Linear Gaussian Models
343
CRG-TR-96-3). Toronto: Department of Computer Science, University of Toronto. Submitted for publication. Ghahramani, Z., & Hinton, G. (1997). The EM algorithm for mixtures of factor analyzers (Tech. Rep. CRG-TR-96-1). Toronto: Department of Computer Science, University of Toronto. Available from ftp://ftp.cs.toronto.edu/pub/zoubin/. Ghahramani, Z., & Jordan, M. I. (1994). Supervised learning from incomplete data via an EM approach. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 120–127). San Mateo, CA: Morgan Kaufmann. Goldberg, P., Williams, C., & Bishop, C. (1998). Regression with input-dependent noise: A gaussian process treatment. In M. Kearns, M. Jordan, & T. Solla (Eds.), Advances in neural information processing systems, 10 (pp. 493–499). Cambridge, MA: MIT Press. Goodwin, G. C., & Sin, K. S. (1984). Adaptive filtering prediction and control. Englewood Cliffs, NJ: Prentice Hall. Hinton, G. E., Dayan, P., & Revow, M. (1997). Modelig the manifolds of images of handwritten digits. IEEE Transactions on Neural Networks, 8, 65–74. Hinton, G., & Ghahramani, Z. (1997). Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society B, 352, 1177–1190. Hinton, G. E., Revow, M., & Dayan, P. (1995). Recognizing handwritten digits using mixtures of linear models. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 1015–1022). Cambridge, MA: MIT Press. Hinton, G. E., & Zemel, R. S. (1994). Autoencoders, minimum description length, and Helmholtz free energy. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 3–10). San Mateo, CA: Morgan Kaufmann. Jordan, M. I., & Jacobs, R. A. (1994). Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6(2), 181–214. Joreskog, ¨ K. G. (1967). Some contributions to maximum likelihood factor analysis. Psychometrica, 32, 443–482. Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. Trans. Am. Soc. Mech. Eng., Series D, Journal of Basic Engineering, 82, 35–45. Kalman, R. E., & Bucy, R. S. (1961). New results in linear filtering and prediction theory. Trans. Am. Soc. Mech. Eng., Series D, Journal of Basic Engineering, 83, 95–108. Kivinen, J., & Warmuth, M. K. (1997). Exponentiated gradient versus gradient descent for linear predictors. Journal of Information and Computation, 132(1), 1–64. Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological Cybernetics, 43, 59–69. Lauritzen, S. L., & Spiegelhalter, D. J. (1988). Local computations with probabilities on graphical structures and their application to expert systems. J. Royal Statistical Society B, 50, 157–224. Little, R. J. A., & Rubin, D. B. (1987). Statistical analysis with missing data. New
344
Sam Roweis and Zoubin Ghahramani
York: Wiley. Ljung, L., & Soderstr ¨ om, ¨ T. (1983). Theory and practice of recursive identification. Cambridge, MA: MIT Press. Lloyd, S. P. (1982). Least square quantization in PCM. IEEE Transactions on Information Theory, 28, 128–137. Lyttkens, E. (1966). On the fixpoint property of Wold’s iterative estimation method for principal components. In P. Krishnaiah (Ed.), Paper in multivariate analysis. New York: Academic Press. MacKay, D. J. C. (1996). Maximum likelihood and covariant algorithms for independent component analysis (Tech. Rep. Draft 3.7). Cambridge: Cavendish Laboratory, University of Cambridge. Moulines, E., & Cardoso, J.-F., & Gassiat, E. (1997). Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (Vol. 5), pp. 3617–3620. Neal, R. M. (1993). Probabilistic inference using Markov chain Monte Carlo methods (Tech. Rep. CRG-TR-93-1). Toronto: Department of Computer Science, University of Toronto. Neal, R. M., & Hinton, G. E. (1998). A view of the EM algorithm that justifies incremental, sparse and other variants. In M. I. Jordan (Ed.), Learning in graphical models (pp. 355–368). Dordrecht, MA: Kluwer. Nowlan, S. J. (1991). Maximum likelihood competitive learning. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems, 3 (pp. 574–582). San Mateo, CA: Morgan Kaufmann. Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Mateo, CA: Morgan Kaufmann. Pearlmutter, B. A., & Parra, L. C. (1997). Maximum likelihood blind source separation: A context-sensitive generalization of ICA. In M. Mozer, M. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems, 9 (pp. 613– 619). Cambridge, MA: MIT Press. Rabiner, L. R., & Juang, B. H. (1986). An introduction to hidden Markov models. IEEE ASSP Magazine, 3(1), 4–16. Rauch, H. E. (1963). Solutions to the linear smoothing problem. IEEE Transactions on Automatic Control, 8, 371–372. Rauch, H. E., Tung, F., & Striebel, C. T. (1965). Maximum likelihood estimates of linear dynamic systems. AIAA Journal, 3(8), 1445–1450. Roweis, S. T. (1998). EM algorithms for PCA and SPCA. In M. Kearns, M. Jordan, and S. Solla (Eds.), Advances in neural information processing systems, 10 (626– 632). Cambridge, MA: MIT Press. Also Tech Report CNS-TR-97-02, Computation and Neural Systems, Calif. Institute of Technology. Rubin, D. B., & Thayer, D. T. (1982). EM algorithms for ML factor analysis. Psychometrika, 47(1), 69–76. Saul, L. and Rahim, M. (1998). Modeling acoustic correlations by factor analysis. In M. Kearns, M. Jordan, & S. Solla (Eds.), Advances in neural information processing systems, 10 (pp. 749–755). Cambridge, MA: MIT Press. Shumway, R. H., & Stoffer, D. S. (1982). An approach to time series smoothing and forecasting using the EM algorithm. Journal of Time Series Analysis, 3(4),
A Unifying Review of Linear Gaussian Models
345
253–264. Shumway, R. H., & Stoffer, D. S. (1991). Dynamic linear models with switching. Journal of the American Statistical Association, 86(415), 763–769. Sirovich, L. (1987). Turbulence and the dynamics of coherent structures. Quarterly Applied Mathematics, 45(3), 561–590. Smyth, P. (1997). Clustering sequences with hidden Markov models. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 9 (pp. 648–654). Cambridge, MA: MIT Press. Smyth, P., Heckerman, D., & Jordan, M. (1997). Probabilistic independence networks for hidden Markov probability models. Neural Computation, 9, 227–269. Tipping, M. E., & Bishop, C. M. (1999). Mixtures of probabilistic principal component analyzers. Neural Computation, 11, 443–482. Also Tech. Report TRNCRG/97/003, Neural Computing Research Group, Aston University. Tresp, V., Ahmad, S., & Neuneier, R. (1994). Training neural networks with deficient data. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 128–135). San Mateo, CA: Morgan Kaufmann. Viterbi, A. J. (1967). Error bounds for convolutional codes and an asymptotically optimal decoding algorithm. IEEE Transactions on Information Theory, IT-13, 260–269. Whittaker, J. (1990). Graphical models in applied multivariate statistics. Chichester, England: Wiley. Received September 5, 1997; accepted April 23, 1998.
ARTICLE
Communicated by Richard Zemel
Implicit Learning in 3D Object Recognition: The Importance of Temporal Context Suzanna Becker Department of Psychology, McMaster University, Hamilton, Ontario, Canada L8S 4K1
A novel architecture and set of learning rules for cortical self-organization is proposed. The model is based on the idea that multiple information channels can modulate one another’s plasticity. Features learned from bottom-up information sources can thus be influenced by those learned from contextual pathways, and vice versa. A maximum likelihood cost function allows this scheme to be implemented in a biologically feasible, hierarchical neural circuit. In simulations of the model, we first demonstrate the utility of temporal context in modulating plasticity. The model learns a representation that categorizes people’s faces according to identity, independent of viewpoint, by taking advantage of the temporal continuity in image sequences. In a second set of simulations, we add plasticity to the contextual stream and explore variations in the architecture. In this case, the model learns a two-tiered representation, starting with a coarse view-based clustering and proceeding to a finer clustering of more specific stimulus features. This model provides a tenable account of how people may perform 3D object recognition in a hierarchical, bottom-up fashion. 1 Introduction: Context, Coherence, and Plasticity Context effects, both spatiotemporal and top-down, are ubiquitous in behavior and can also be observed at the neuronal level. The ability of context to influence perception has been demonstrated in many domains. For example, letters are recognized more quickly and accurately in the context of words (see, e.g., McClelland & Rumelhart, 1981), and words are recognized more efficiently when preceded by related isolated words (see, e.g., Neely, 1991), sentences, or passages (Hess, Foss, & Carroll, 1995). In the compelling McGurk effect (McGurk & MacDonald, 1976; MacDonald & McGurk, 1978), a person is presented with a videotape of auditory information for one utterance simultaneously paired with visual information for another utterance. However, the mismatch typically goes unnoticed. What happens is that for some sound pairs, the person’s percept tends to be dominated by the auditory cues, in other cases the visual cues dominate, and in still other cases, various fusions and alternations of the two sources are perceived. Apparently when the two modalities provide contradictory information, people Neural Computation 11, 347–374 (1999)
c 1999 Massachusetts Institute of Technology °
348
Suzanna Becker
choose which modality to believe and which to ignore, or whether to fuse the modalities, according to the context. The importance of contextual information in modulating neuronal response profiles is becoming increasingly apparent. For example, some visual cortical cells (in the deepest layer of area V1) have been found that are excited by an oriented stimulus in the center of their receptive field and show an enhanced response to a similarly oriented stimulus in the surrounding region; on the other hand, the response is suppressed by an orthogonally oriented stimulus in the surround (Cudeiro & Sillito, 1996). In contrast, some cells show just the opposite pattern: they are antagonized by a similarly oriented stimulus in the surround, and facilitated by an orthogonally oriented stimulus (Sillito, Grieve, Jones, Cudeiro, & Davis, 1995). On the other hand, about 40% of complex cells (in the superficial layers of area V1) are facilitated by the conjunction of a line segment in their classical receptive field and a colinear line segment placed nearby, outside their classical receptive field (Gilbert, Das, Ito, Kapadia, & Westheimer, 1996). Moreover, even in primary visual cortex, cells’ tuning curves (in all cortical layers) are sensitive to the temporal history of the input signal and can show bimodal peaks and even complete reversals in tuning over time (Ringach, Hawken, & Shapley, 1997). These examples demonstrate that neuronal responses can be modulated by secondary sources of information in complex ways. Why would contextual modulation be such a pervasive phenomenon? One obvious reason is that if context can influence processing, it can help in disambiguating or cleaning up noisy stimuli. However, an overreliance on contextual cues leaves the system open to the possibility of information loss, for example, by smearing information across discontinuities. A less obvious reason that context is so pervasive may be that if context can influence learning, it may lead to more compact and powerful representations, whereby units encode complex stimulus configurations. In this article, we focus particularly on temporal context. Most unsupervised classifiers are insensitive to temporal context; that is, they group patterns together solely on the basis of spatial overlap. This may be reasonable if there is very little shift or other form of distortion between one time step and the next, but it is not a reasonable assumption about the sensory input to the cortex. Precortical stages of sensory processing, certainly in the visual system and probably in other modalities, tend to remove low-order correlations in space and time (see, e.g., Dong & Atick’s, 1995, model of lateral geniculate nucleus cells). Consider the images in Figure 1. The top row shows a series of snapshots of one person’s face being rotated through 180 degrees. The bottom row shows a series of snapshots of another person’s face, also being rotated through 180 degrees. They have been preprocessed by a simple edge filter, so that successive views of the same face have relatively little pixel overlap. Even in these low-resolution images, we can see certain regularities in the features of each individual. For example, each person’s head shape remains consistent across changes in viewpoint. With respect to raw pixel
Implicit Learning in 3D Object Recognition
349
Figure 1: Two sequences of 48 × 48 pixel images digitized with an IndyCam and preprocessed with an edge filter using SGI’s Image Works. Eleven views of each of 4 to 10 faces were used in the simulations reported here. The alternate (odd) views of 2 of the faces are shown above.
overlap, however, two snapshots of a given individual’s face taken from very different viewpoints often have less in common than snapshots of two different individuals’ faces taken from the same viewpoint. This creates a difficult challenge for unsupervised learning systems. Unsupervised learning procedures like principal component analysis and clustering can model only lower-order structure (e.g., covariance or Euclidean proximity). How could a self-organizing system discover the higher-order structure shared by radically different views of the same object, and ignore the lower-order structure shared by identical views of different objects? Clearly we have a long way to go in understanding what sort of learning procedures are employed by the brain, to form distributed representations and account for our high-level perceptual abilities. One powerful cue for real vision systems is the temporal continuity of objects. Novel objects typically are encountered from a variety of angles, as the position and orientation of the observer, or objects, or both, vary smoothly over time. It would be very surprising if the visual system did not capitalize on this temporal continuity in learning to group together visual events that co-occur in time. In section 7, we mention several lines of empirical evidence in support of this notion. In the model of cortical selforganization proposed here, we postulate that contextual modulation plays a critical role in guiding unsupervised class formation. The term context is used very generally here to mean any secondary source of input; it could be from a different sensory modality, a different input channel within the same modality, a temporal history of the input, or top-down information from descending pathways. Although in the simulations reported here we specifically focus on temporal context in the visual system, the same ideas should be applicable to a variety of other sources of context in a variety of cortical areas.
350
Suzanna Becker
2 Maximum Likelihood Cost Function Given that we have identified context as an important cue in learning, the next step is to formalize this notion. We propose maximizing a log-likelihood cost function, as in Nowlan (1990) and Jacobs, Jordan, Nowlan, and Hinton (1991). In this framework, the network is viewed as a probabilistic, generative model of the data. The learning serves to adjust the weights so as to maximize the log-likelihood of the model having generated the data: L = log P(data | model).
(2.1)
If the training patterns, I(α) , are independent, L = log
n Y
P(I(α) | model)
α=1
=
n X
log P(I(α) | model).
(2.2)
α=1
However, this assumption of independence is not valid under natural viewing conditions. If one view of an object is encountered, a similar view of the same object is likely to be encountered next. In this article, we propose an extension to the above model in which the independence assumption is relaxed, so that the inputs are only assumed to be independent given the context. In the most general case, the context could be any additional source of information. In the simulations reported here, we explore the special case where the temporal history of the input acts as the context. There are several advantages to this approach. First, having a global cost function for the learning provides a principled basis for deriving learning rules in a network. Second, the maximum likelihood cost function sets up a very reasonable goal for the learning: modeling the probability distribution of the data. Third, by choosing an appropriate parametric form for the model, that is, the network architecture and associated statistical assumptions, we can incorporate the added goal of allowing contextual input to modulate the learning. 2.1 Maximum Likelihood Competitive Learning. In maximum likelihood competitive learning (MLCL) (Nowlan, 1990), the units have gaussian activations, yi , and the network forms a mixture-of-gaussians model of the data. The result is a simple and elegant network implementation of a widely used statistical clustering algorithm. A “soft competition” among the units, rather than a winner-take-all, “hard competition,” determines the relative activation levels of the units and hence their learning rates for each pattern. This causes each unit to become selective for a different region of the input space.
Implicit Learning in 3D Object Recognition
351
The following cost function forms the basis for MLCL, L=
n X
log
α=1
=
n X
" m X
# P(I
(α)
i=1
log
α=1
" m X
| submodeli ) P(submodeli ) #
yi
(α)
πi ,
(2.3)
i=1
where the πi s are positive mixing coefficients that sum to one, and the yi s are the unit activations, E i , 6i ), yi (α) = N(IE(α) , w
(2.4)
E i and covariance where N() is the gaussian density function, with mean w matrix 6i . Here and throughout the article, we use the term submodel to refer to a gaussian component in the mixture model. So yi represents the probability of the input vector under the ith submodel, a gaussian centered E i . The ith mixing coefficient, πi , represents on the ith unit’s weight vector, w the prior probability of the ith gaussian having generated the data. In MLCL, E i , are obtained by maximizing over L, and the mixing the gaussian means, w coefficients are either fixed to equal values or alternatingly reestimated after each update of the model parameters as in the expectation-maximization (EM) algorithm (Dempster, Laird, & Rubin, 1977). For simplicity, Nowlan typically used a single global variance parameter for all input dimensions and allowed it to shrink during learning. L can be maximized by on-line gradient ascent1 with learning rate ε: 1wij = ε
³ ´ X πi yi (α) ∂L P =ε Ij (α) − wij . (α) ∂wij k πk yk α
(2.5)
The term πi yi (α) P (α) k πk yk represents the ith submodel’s probability given the current pattern and context. It is normalized over all competing units (submodels), hence the term soft competition. A long-time average of this probability over many data items represents πi , the overall probability of the ith submodel. Thus, this rule is quite biologically plausible. It consists of a Hebbian update rule with weight decay, using normalized postsynaptic unit activations. 1 Nowlan (1990) used a slightly different on-line weight update rule that more closely approximates the batch update rule of the EM algorithm.
352
Suzanna Becker
2.2 Contextually Modulated Competitive Learning. MLCL assumes that the input patterns are independent. If we remove this restriction, allowing for temporal dependencies among the input patterns, the log-likelihood function becomes: L = log P(data | model) X log P(I(α) | I(1) , . . . , I(α−1) , model). =
(2.6)
α
To incorporate a contextual information source into the learning equation, we extend MLCL by introducing a contextual input stream into the likelihood function: L = log P(data | model, context) X log P(I(α) | I(1) , . . . , I(α−1) , model, context). =
(2.7)
α
Unlike the model underlying standard MLCL, we want to deal with input streams that may contain arbitrarily complex temporal dependencies. Suppose the input and context represent two separate streams of observable data, with unknown interdependencies. This situation is depicted in Figure 2a. Taken together, the input and context can be viewed as an ordered sequence of pairs, (I(α) , C(α) ), where C(α) is the contextual input pattern on training case α. We now consider several simplifying assumptions that result in a tractable model. Our first assumption is that the model consists of a mixture of submodels. The log-likelihood then becomes: L=
X α
X log P(I(α) | I(1) , . . . , I(α−1) , C(1) , . . . , C(α) , submodelj ) j
i P(submodelj | I(1) , . . . , I(α−1) , C(1) , . . . , C(α) ) .
(2.8)
Second, let us assume that the probability of observing a particular input pattern is independent of other patterns when conditioned on the context sequence, and vice versa. In other words, all of the temporal dependencies in the input stream can be accounted for by knowing the context, and vice versa. This situation is depicted in Figure 2b. Now we have: L=
X α
X log P(I(α) | C(1) , . . . , C(α) , submodelj ) j
i P(submodelj | I(1) , . . . , I(α−1) , C(1) , . . . , C(α) ) .
(2.9)
Implicit Learning in 3D Object Recognition
353
Figure 2: The conditional dependencies among the observable variables (context and input) are depicted in three situations. (a) The long-range dependencies within the two sequences. (b) The interdependencies within the two sequences disappear when each element in the top sequence is conditioned on the bottom sequence, and vice versa. (c) The sequences become independent of each other when conditioned on the hidden variables (the “submodel” index).
Finally, let us assume that given the submodel, the input and context are independent. In other words, all the remaining dependencies in the observable data are explained away by knowing which submodel generated the data at each point in time. This situation is depicted in Figure 2c. Now the likelihood equation simplifies to: L=
X α
" log
X
P(I(α) | submodelj )
j
# (1)
P(submodelj | I , . . . , I =
n X α=1
X log yj (α) gj (α) ,
(α−1)
(1)
(α)
,C ,...,C
)
(2.10)
j
where gj (α) represents the probability of the jth submodel given the input and context, and yj (α) represents the probability of the input under the jth submodel.
354
Suzanna Becker
Figure 3: A neural circuit for implementing CMCL.
3 Network Implementation The contextually modulated competitive learning (CMCL) cost function given in equation 2.10 could be implemented in a variety of architectures, depending on how much computational power is allocated to individual units. In section 7, we explore this issue further and consider the potential advantage of more powerful units with nonlinear synaptic interactions. In the simulations reported here, we used multilayer circuits consisting of an input layer, a layer of clustering units, and a layer of gating units, as in Figure 3. We chose the term gating units because their role here is analogous to that of the gating network in the competing experts model (Jacobs et al., 1991). In fact, the model proposed here could be viewed as an unsupervised version of the mixture of competing experts architecture. Jacobs et al.’s competing experts network performs supervised learning and can be interpreted as fitting a mixture of gaussians model of the training signal. In contrast, the clustering units (experts) here are fitting a mixture model to the input signal, while the gating units simultaneously are adapting to the context signal, in order to help the clustering units divide up the input space. This is very different from a model that separately clusters the input and context signals because contextual features are used here to modulate the partitioning of the input space. As our simulations show, this results in a very different clustering of the inputs. The clustering units receive the primary source of input to the network. As in MLCL, each clustering unit produces an output yi (α) proportional to the probability of the input pattern, I(α) , given the ith submodel (this would be exactly equal to the probability if it were normalized). Each yi (α) is computed as a gaussian function of its current input, yi (α) = e−kI
(α)
−wEi k2 /σ 2 i
,
(3.1)
where k · k is the L2 norm, wEi is the weight vector for the ith clustering
Implicit Learning in 3D Object Recognition
355
unit representing the mean of the ith gaussian, and σ 2 i is the variance of that gaussian, assuming all gaussians are spherical. The gating units receive the contextual stream of input and produce outputs gi (α) representing the probability of the ith submodel given the current context, C(α) . For the simulations reported here, the gating units compute their outputs according to a “softmax function” (Bridle, 1990) of their weighted summed inputs xi (α) : (α)
e xi gi (α) = P x (α) , j je X Ck (α) vik , xi (α) =
(3.2) (3.3)
k
where j indexes over all gating units in the network, and vik is the weight on the connection from the kth contextual input to the ith gating unit. Here, we have made a further simplifying assumption that the prior probabilities of the submodels (the p(submodeli ) terms in equation 2.10) are all equal and fixed and can therefore be folded into the gating units’ activations gi . Alternatively, assuming the probabilities of choosing each submodel form a Markov chain—that is, they depend on knowledge only one step back in time—one could then estimate the true probabilities of submodels under a hidden Markov model (HMM) (as suggested by Hinton, personal communication). This would allow for temporal dependencies between the submodels over time to be modeled explicitly. Cacciatore and Nowlan (1994) have extended the mixture of competing experts model in this way, to allow recurrent gating networks. (See section 7 for further comments on the relation between HMMs and our model.) 4 The Learning Equations Given the likelihood function defined by equation 2.10, on-line learning rules for the clustering and gating units can be derived by differentiating L with respect to their weights. The variances of each of the gaussians, σi2 , could be approximated by their maximum likelihood estimates under a mixture model, as in the EM algorithm (Dempster et al., 1977). Instead, we used a simple on-line approximation to the true variance of the input vector about each clustering unit’s weight vector, σi 2
(α)
= k
´ X³ w2ij + (I(α)j )2 ,
(4.1)
j
where k is a constant. This approximation would be exact, to within a constant factor, if the input vectors were of fixed length and uncorrelated with the weight vectors. In the first set of simulations reported here, k = 0.05, and in the second set, k = 0.03. The main role of the adaptive variance in
356
Suzanna Becker
the learning is to scale the clustering unit activations, to prevent them from overfitting the training patterns. The learning rule for the weight from the jth input to the ith clustering unit for input pattern α is: 1wij = ε
∂L ∂yi (α) ∂yi (α) ∂wij
1 gi (α) yi (α) = ε P (α) (α) (α) 2 yk σi k gk
à Ij
(α)
k I(α) − wi k2 − wij +wij P (α) )2 + wik 2 k (Ik
! , (4.2)
where ε is a learning-rate constant. The learning rule for the weight from the jth contextual input to the ith gating unit for input pattern α is: ∂L ∂ gi (α) ∂ gi (α) ∂vij µ ¶ gi (α) yi (α) (α) = ε P (α) (α) − gi Ij (α) . yk k gk
1vij = ε
(4.3)
As a consequence of the multiplicative interaction between the gating and clustering units’ activations in the cost function (see equation 2.10), each gating unit’s activation modulates the corresponding clustering unit’s learning. Thus, the clustering units are encouraged to discover features that agree with the current contextual gating signal (and vice versa). At any given moment in time, if their contextual gating signal is weak or if they fail to capture enough activation from their bottom-up input, they will do very little learning. Only when a clustering unit’s weight vector is sufficiently close to the current input vector and its corresponding gating unit is strongly active will it do substantial learning. 5 Simulations with Network 1 Our first set of simulations was designed to demonstrate the utility of temporal context in contributing to higher-order feature extraction and viewpointinvariant object recognition. For these simulations, the gating connection weights were held fixed. Our second set of simulations was designed to generalize these findings to a network with adaptive links in the gating layer and to show that by varying the architectural constraints, the network could develop pose-tuned rather than viewpoint-invariant face-tuned units. For our first set of simulations, we used networks of the form shown in Figure 4. The network is subdivided into modules, each consisting of one or more clustering units and one gating unit. In our second set of simulations, modules contain multiple gating units and only one clustering unit.
Implicit Learning in 3D Object Recognition
357
The contextual inputs are time-delayed, temporally blurred versions of the outputs of a module (including both gating and clustering units’ outputs). The gating units’ outputs are softmax functions of their weighted summed blurred inputs. The temporal blurring on the contextual input lines was achieved by accumulating the activation on each connection as follows: Ci (α) = 0.5(Ci (α−1) + inputi (α−1) ),
(5.1)
where inputi (α) is the ith input to the gating unit before blurring for pattern α; this input could be equal to the output of either a clustering unit in the layer below or the gating unit itself (see Figure 4). More general forms of context are possible, as noted in section 7. We have deviated from the general form of the architecture shown in Figure 3 in an important way: There is now a many-to-one mapping from clustering units to gating units, so that clustering units within the same module i receive a shared gating signal, gi , and produce outputs yij . Thus, clustering units in the same module are responsible for learning different submodels, but they predict the same contextual feature. The likelihood equation now becomes: m l X X 1 (α) (α) log gi yij . L= l j=1 α=1 i=1 n X
(5.2)
To relate this to the original mixture model given by equation 2.10, we still have a single mixture of gaussian submodels, with each clustering unit corresponding to a submodel. However, the probabilities over submodels (the gi s) given the inputs and contexts have some equality constraints imposed, so that clustering units in the same module share the same submodel probability. One might predict that clustering units with a shared source of contextual input would all come to detect exactly the same feature. Fortunately, there is a disincentive for them to do so: They would then do poorly at modeling the input. Thus, clustering units in the same module should come to encode a common part of the context but detect different features. Our network architecture was designed with several goals in mind. First, the modular, layered architecture is meant to constrain the network to develop hierarchical representations and functional modularity, as observed in the cortical laminae and columns respectively (see, e.g., Calvin, 1995). That is, we should see a progression from simple to higher-order features in the clustering and gating layers, with functional groupings of similar features in units within the same module. Second, we expect the temporal context to influence the sort of features learned by the clustering layer; each clustering unit should detect a different range of temporally correlated features. To test the predictions of our model, we performed simulations with networks like the one shown in Figure 4 trained on sequences of patterns like
358
Suzanna Becker
Figure 4: The architecture used in the first set of simulations reported here. The gating units received all their inputs across unit delay lines with fixed weights of 1.0. For these simulations, some of the networks had an architecture with 4 modules exactly as shown here and were trained on sequences of images of 4 individuals’ faces. For the remaining simulations, the networks had 10 modules like the ones shown above and were trained on sequences of 10 individuals’ faces.
the ones shown in Figure 1. The training patterns consisted of a set of image sequences of 10 centered, gradually rotating faces. In our first set of simulations, there were 4 modules, and only 4 of the 10 faces were used; in the final simulations, the generality of our findings was extended by training a larger network of 10 modules like the ones shown in Figure 4 on all 10 faces. In both cases, there were three clustering units per module. It was predicted that the clustering units should discover “features” such as temporally correlated views of specific faces. Further, different views of the same face should be represented by different clustering units within the same module because they will be observed in the same temporal context, while the gating units should respond to particular individuals’ faces, independent of viewpoint. The training and testing pattern sets were created by repeatedly visiting each of the 10 faces in random order. For each face, an ordered sequence of views was presented to the network by randomly choosing either a leftfacing or right-facing view as the initial view in the sequence, and then presenting the remaining views of that face in an ordered sequence. For a given face sequence, views were presented in an ascending order and then
Implicit Learning in 3D Object Recognition
359
a descending order (e.g., rotating through 180 degrees to the right and then to the left), so the initial view was always the final view in each sequence. At the end of each face sequence, a new face and starting view were randomly selected. The network had no knowledge of when a new face would occur or that the training set actually contained ordered sequences. Thus, although the network assumes that temporal context is smooth everywhere, in these data, it is actually discontinuous across the boundaries between sequences. Gating units had self-links, as well as links from all the clustering units within the same module, all of which had unit time delays. All the gating unit connections had fixed weights of 1.0. Thus, each gating unit received a temporal history of its own output and the outputs of the clustering units in the same module. Tuning curves for all units in the network in a typical run are plotted in Figures 5 and 6. The clustering units became specialized for detecting particular faces in a narrow range of views, as shown in Figure 5. Simply by accumulating a temporal history of the clustering units’ activations within a module, each gating unit was then able to respond to an individual face, independent of viewpoint, as shown in Figure 6. Of course, the tuning curves for the gating layer shown here depend on there being continuity in the context signal during both training and testing. One might wonder how much of the network’s ability to discriminate faces was due to the temporal context, and how much to unsupervised clustering, independent of the contextual modulation. To answer this question, the baseline effect of the temporal context on clustering performance was assessed by comparing the network shown in Figure 4 to the same network with all connections into the gating layer removed. The latter is equivalent to MLCL with fixed, equal mixing proportions (πi ). First, networks with four modules were trained on sequences of four faces. To quantify clustering performance, each unit was assigned to predict the face class for which it most frequently won (was the most active). Then for each pattern, the layer’s activity vector was counted as correct if the winner correctly predicted the face identity. Generalization performance was assessed by training the network on only the odd-numbered views and testing classification performance on the even-numbered views. The results are summarized in Table 1. As one would expect, the temporal context provides incentive for the clustering units to group successive instances of the same face together, and the gating layer can therefore do very well at classifying the faces with a much smaller number of units—independent of viewpoint. In contrast, the clustering units without the contextual signal are more likely to group together instances of different people’s faces. Next, a network like the one shown in Figure 4 but with 10 modules was presented with a set of 10 faces, 11 views each. As before, the oddnumbered views were used for training and the even-numbered views for testing. Without the influence of the context layer, the network’s classifica-
360
Suzanna Becker
Figure 5: Thirty clustering units’ normalized activations are plotted against face identity (bottom left axis) and viewing angle (bottom right axis) of patterns. Each graph shows the activations of a single unit over the entire set of training patterns. Units in the same row were trained with a common contextual gating signal (see Figure 4) and have learned to respond to different views of the same face.
Implicit Learning in 3D Object Recognition
361
Figure 6: Ten gating units’ activations are plotted against face identity (bottom left axis) and viewing angle (bottom right axis) of the training patterns. Each graph shows the activations of a single unit over the entire set of training patterns. Each gating unit provided a contextual gating signal to three clustering units (see Figure 4) and learned to respond to a single face, independent of view.
362
Suzanna Becker Table 1: Mean Percentage (and Standard Error) Correctly Classified Faces.
No context, 4 faces No context, 10 faces Context, 4 faces Context, 10 faces
Layer 1 Layer 1 Layer 1 Layer 2 Layer 1 Layer 2
Train
Test
59.2 (2.4) 15.0 (0.0) 88.4 (3.9) 88.8 (4.0) 96.3 (1.2) 91.8 (2.4)
65.0 (3.5) 12.0 (0.0) 74.5 (4.2) 72.7 (4.8) 71.0 (3.0) 70.2 (4.3)
Note: Ten runs, for unsupervised clustering networks trained for 2000 iterations with a learning rate of 0.5, with and without temporal context. Layer 1: clustering units. Layer 2: gating units.
tion performance was very poor. With the addition of contextual modulation, this network still had difficulty classifying all 10 faces correctly and seemed to be somewhat more sensitive to the weights on the gating connections. However, when the weights on the self-pointing connections on the gating units were increased from 1.0 to 3.0, to increase the time constant of temporal averaging, the network performed extremely well. On average, the top-layer units achieved 96% correct classification on the training set and 70% correct on the test set. In further simulations, reported in Becker (1997), the generalization performance of the unsupervised network was shown to be substantially superior to that of supervised backpropagation networks with similar architectures; however, when a temporal smoothness constraint was imposed on the hidden-layer units’ states, even feedforward backpropagation networks performed as well as our unsupervised model. 6 Simulations with Network 2 The network shown in Figure 4 learned a “grandmother cell” representation, where each clustering unit learned to specialize for a single face at a particular viewpoint, and each gating unit therefore responded to a single face over a wide range of viewpoints. Although “face cells” have been identified by many laboratories (Gross, Rocha-Miranda, & Bender, 1971; Perrett, Rolls, & Cann, 1982; Desimone, Albright, Gross, & Bruce, 1984; Yamane, Kaji, & Kawana, 1988; Tanaka, Saito, Fukada, & Moriya, 1991), these cells only rarely exhibit either viewpoint invariance or selectivity for a single individual; the vast majority of face cells are tuned to one of only four views (front, back, left and right) and respond roughly equally to the heads of different individuals (Perrett, Hietanen, Oram, & Benson, 1992). There are several reasons that it is unlikely that the brain uses a grandmother cell representation as a matter of course. For one, it is very expensive with respect to neural machinery. Further, it does not scale well; each time
Implicit Learning in 3D Object Recognition
363
a new face is encountered, new representational units would need to be added. Finally, this type of representation exhibits poor generalization. In our second set of simulations, we sought to explore the interaction between the architecture and the cost function in constraining the representation learned by the network. This time, we used the architecture shown in Figure 7. This network differs from the one used in the first set of simulations in two important ways, chosen to encourage more distributed representations of faces. First, the network has fewer modules than the previous one: only three modules were trained to encode all 10 faces. Now the network must form a more compact encoding of the face stimuli. Second, there is now only one clustering unit per module, and there are multiple gating units per module (four per module in the simulations reported here). Thus, rather than a many-to-one relationship between clustering and gating units in each model, the relationship is one-to-many. The clustering units should therefore be encouraged to develop broader tuning curves and might be expected to cluster faces based on viewpoint (pose) rather than face identity, given the low pixel overlap between successive views of the same face. Further, because there are multiple gating units for each clustering unit, the gating units might be expected to learn a more distributed representation of faces. To accommodate the one-to-many relationship between the clustering and gating units, the cost function was modified so that each clustering unit takes as its gating signal the average of the activations over the gating units in the same module: m l X X 1 log y(α) gij (α) . L= i l α=1 i=1 j=1 n X
(6.1)
As in the first network, we still have a single mixture of gaussian submodels, with each clustering unit corresponding to a submodel. Now, the probability over each submodel, i, given the inputs and contexts, is computed by averaging the activations gij of gating units within the same module. As before, the gating units received time-delayed, temporally blurred inputs from the clustering layer. Unlike in the previous simulations, the gating units also received time-delayed, temporally blurred inputs directly from the input layer. This extra source of context was provided so that gating units in the same module would have some basis for developing differential responses. The clustering units’ connection weights were updated for 2000 iterations with a fixed learning rate of 0.1 while the gating units’ connection weights were initially held fixed. Typical response profiles for the clustering units are shown in Figure 8. As predicted, these units exhibited broad face tuning but relatively narrow pose tuning. The gating units’ connection weights from the input layer were then updated for 2000 further iterations with a fixed learning rate of 0.02. No
364
Suzanna Becker
Figure 7: The architecture used in the second set of simulations reported here. The gating units received normalized, temporally blurred input from clustering units in the same module and neighboring module(s), and direct connections from the input layer. The connections from the clustering units to the gating units had fixed weights of 0.6 for within-module connections, 0.2 for between-module connections to the middle module, and 0.4 for between-module connections to the end modules. The weights on the direct input connections to the gating layer were fixed at zero, while the clustering layer was trained, and were subsequently adapted during a second training phase.
Figure 8: Three clustering units’ normalized activations are plotted against face identity (bottom left axis) and viewing angle (bottom right axis) of patterns. Each graph shows the activations of a single unit over the entire set of training patterns. Each clustering unit received contextual input from three gating units (see Figure 5) and learned to respond to faces from a particular viewpoint, independent of face identity.
Implicit Learning in 3D Object Recognition
365
constraints were placed on these weights, so they could potentially grow larger than the weights from the clustering to the gating layer. Networks with different numbers of gating units per module (but always three or four modules) were experimented with and produced qualitatively similar results. The gating units tended to respond to combinations of one or more faces at similar poses. However, the responses were not convincingly distributed. Rather, different gating units became selective for narrow, relatively nonoverlapping regions of the face-pose space. To encourage the gating units to develop more distributed responses, the time delay and blurring from the direct input connections to the gating layer were removed. Thus, like the clustering units, the gating units could now access only a single time slice of the input at a given moment. As predicted, this decreased the tendency for gating units to group faces of particular individuals over time, resulting in more multimodal response profiles, as in Figure 9. In this case, gating units in the same module (plotted in the same row in Figure 9) tended to have similar pose tuning and multimodal, somewhat overlapping face-tuning profiles. This architecture actually violates the conditional independence assumption about the input and context streams, by using the same signal for both input and context. This would be of greater concern if the clustering and gating layers were adapted simultaneously, in which case they could achieve agreement in trivial ways, such as by attending to only small subsets of their inputs. To address this issue of independence, similar results were obtained in networks in which the clustering and gating layers were randomly connected to the input layer, which provided an approximation to independence.2 To summarize our second set of simulations, we sought to extend our basic findings by exploring several variations in the architecture that were predicted to lead to more distributed representations of faces. In particular, fewer modules were used, and there were multiple gating units per module. As predicted, the clustering units became less tuned to individuals’ faces. Instead, they developed pose tuning and were broadly selective to a wide range of individuals. It was also predicted that the gating units would form distributed codes for faces. However, although their tuning curves were multimodal in face-pose space, they were not strongly overlapping, but instead remained relatively local. This representation would be good for recognizing general features common to many faces, but would not be as appropriate for face classification as compared to that learned by the first architecture.
2 This approximation is still not exact. A better solution would be to connect the clustering and gating layers to physically different parts of the input. For example, the gating units could be connected to the spatial context surrounding the input to the clustering unit(s) in the same module.
366
Suzanna Becker
Figure 9: Twelve gating units’ activations, before normalization, are plotted against face identity (bottom left axis) and viewing angle (bottom right axis) of patterns. Each graph shows the activations of a single unit over the entire set of training patterns. Units in the same row were trained to provide a common contextual gating signal to a single clustering unit (see Figure 5). For the most part, each has learned to respond to multiple faces from a narrow range of views.
7 Discussion The simulation results with our model demonstrate that temporal context can markedly alter the sort of features or classes learned by an unsupervised network. When combined with appropriate architectural constraints, a range of representations can be learned. But does this have anything to say about self-organization in the cortex? In this section, we consider behavioral and physiological lines of evidence in support of our model. Finally, several related computational models are considered. 7.1 Empirical Evidence for the Use of Temporal Context. There is evidence that single cells’ tuning curves exhibit complex temporal dynamics (Ringach et al., 1997; De Angelis, Ohzawa, & Freeman, 1995). But are these effects hard wired, or might temporal context play a role in the learning of receptive fields? Physiological evidence from Miyashita (1988) would support the latter contention. Miyashita repeatedly exposed monkeys to a fixed sequence of 97 randomly generated fractal images during a visual memory task and subsequently recorded from cells in the anterior ventral temporal cortex. Many cells responded to several of the fractal patterns, and the
Implicit Learning in 3D Object Recognition
367
grouping of patterns was based on temporal contiguity rather than geometric similarity. This is rather striking evidence for learning based on temporal associations rather than pattern overlap. Furthermore, recent behavioral evidence suggests that temporal context is important to human learning about novel objects. Seergobin, Joordens, and Becker (unpublished data) exposed experimental participants to sequences of images of faces of the same sort used in the simulations reported here. In one condition, faces were viewed “coherently,” that is, in ordered sequences from left to right or right to left. In another condition, faces were viewed “incoherently,” that is, each face was presented in a scrambled sequence with the views randomly ordered. Participants demonstrated a significant benefit in face matching from the more coherent temporal context during study.3 Given that there may be differences in the way humans process faces as compared to other types of objects (Bruce, 1997), Seergobin et al. extended their results in a further set of experiments using static image sequences of novel, artificially generated bumpy objects resembling asteroids. In this case, a similar advantage for coherent temporal context in implicit learning was shown. 7.2 Justification for a Modular, Hierarchical Architecture. The hierarchical, modular architecture shown in Figure 3 is motivated by several features widely considered to be ubiquitous throughout all regions of the neocortex: a laminar structure (see, e.g., Douglas & Martin, 1990) and a functional organization into “cortical clusters.” As Calvin (1995, p. 269) succinctly puts it, “The bottom layers are like a subcortical ‘out’ box, the middle layer like an ‘in’ box, and the superficial layers somewhat like an ‘interoffice’ box connecting the columns and different cortical areas.” We tentatively suggest a correspondence between the clustering units in our model and layer IV cells, and between the gating units and the deep and superficial layer cells. With respect to functional modularity, in many regions of cortex, spatially nearby columns tend to cluster into functional groupings with similar receptive field properties (see, e.g., Calvin, 1995), including visual area V2 (Levitt, Kiper, & Movshon, 1994) and inferotemporal cortex (Tanaka, Fujita, Kobatake, Cheng, & Ito, 1993). We experimented with two different means of inducing functional modularity in our model. In the first set of simulations, subsets of clustering units shared a common gating unit and learned to predict similar regions of the contextual space. Consequently, 3 One might then wonder whether fully animated video sequences would confer a further benefit on object learning, over and above that of temporally coherent sequences of static images. Interestingly, for the case of animated versus statically studied faces, Bruce and colleagues found no such advantage in two different experiments (Christie & Bruce, 1998; Bruce & Valentine, 1998). Note, however, that dynamic viewing at the time of testing does improve face recognition performance (Christie & Bruce, 1998; Bruce & Valentine, 1998).
368
Suzanna Becker
they became tuned to temporally coherent features: different views of the same individual’s face. In the second set of simulations, subsets of gating units shared a common clustering unit and learned to detect different contextual features that predicted a common region of the input space. In this case, different gating units in the same module became specialized for similar views but different faces. Further, clustering units in nearby modules had partially overlapping contextual inputs. This resulted in a similarity of function across neighboring modules: clustering units in adjacent modules were selective for similar views. It remains to be seen which, if either, of these architectures is a good model of cortical self-organization and modularity. Another possibility is that the functionality of an entire module of clustering and gating units in our model could be computed by a single neuron. The neuron would then require nonlinear interactions among synaptic inputs, so that the context could act in a modulatory fashion, rather than as a primary driving stimulus. A number of models of cortical cell responses have proposed multiplicative interactions between modulatory and primary input sources (Nowlan & Sejnowski, 1993; Mel, 1994; Mundel, Dimitrov, & Cowan, 1997; Pouget & Sejnowski, 1997). 7.3 Face Processing and Shape Recognition in the Cortex. The model in its present implementation is not meant to be a complete account of the way humans learn to recognize faces. Viewpoint-invariant recognition is probably achieved, if at all, in a hierarchical, multistage system. In ongoing work, we are exploring this possibility by training, in series, a sequence of networks like the one shown in Figure 3, with progressively larger receptive fields at each stage. Oram and Perrett (1994) have proposed a roughly hierarchical, multistage scheme for decomposing the ventral visual pathway into a functional processing hierarchy. Of particular relevance to the results reported here is their proposal for the organization of object recognition in the inferotemporal (IT) cortex. A large body of physiological evidence supports the notion that IT cells are responsible for complex shape coding. After Tanaka and colleagues (Tanaka et al., 1991), Oram and Perrett propose that object recognition is accomplished in a distributed network in IT (particularly the anterior inferotemporal area, AIT) as follows: each module or column codes for a particular shape class. A given object activates many modules, corresponding to different complex visual features. Within a module, different cells exhibit slightly different selectivities and can thereby signal more precisely the stimulus features. For example, cells in a given column might all code for a pair of small, round objects aligned horizontally. Within a column, different cells might further specialize for a pair of eyes or a pair of headlights. Responses across many such columns, taken together, could thereby code a great many different objects uniquely. Only under special circumstances would a grandmother cell be devoted to recognizing a unique conjunction of stimulus features.
Implicit Learning in 3D Object Recognition
369
The network shown in Figure 7 learned a representation that is consistent, at least in broad terms, with the scheme for representing objects proposed by Tanaka et al. and Oram and Perrett. Units in the same module learned to code for a particular class of stimuli: faces over some wide range of views. Different gating units in the same module became further specialized to detect particular features of different faces. These units were usually not tuned to one specific face, but each tended to respond to several specific individuals’ faces. A question for future research is whether the model presented here could encode different uncorrelated features, or different classes of objects, across many different modules. 7.4 Related Work. Phillips, Kay, and Smyth (1995; Kay & Phillips, 1997) have proposed a model of cortical self-organization they call coherent Infomax that incorporates contextual modulation. In their model, the outputs from one processing stream modulate the activity in another stream, while the mutual information between the two streams is maximized. They view this algorithm as a compromise between Imax (Becker & Hinton, 1992) and Infomax (Linsker, 1988). A number of other unsupervised learning rules have been proposed based on the assumption of temporally coherent inputs. Becker (1993) and Stone (1996) proposed learning algorithms that maximize the mutual information in a neuron’s output at nearby points in time. Foldi´ ¨ ak (1991) and Weinshall, Edelman, and Bulthoff ¨ (1990; Edelman & Weinshall, 1991) proposed variants of competitive learning that used blurred outputs and time delays, respectively, to associate items over time. Several investigators (Seergobin, 1996; Wallis & Rolls, 1997; Stewart Bartlett, & Sejnowski, 1998) have shown that Foldi´ ¨ ak’s model, when applied to faces, develops units with broad pose tuning. Temporal smoothing has also been shown to broaden pose tuning to faces in feedforward backpropagation networks (Becker, 1997) and in Hopfield-style attractor networks (Stewart Bartlett & Sejnowski, 1997). O’Reilly and Johnson (1994) used feedback inhibition and excitation to achieve temporal smoothing and pose invariance in a multilayer model that is perhaps most similar to the one proposed here. Their network used excitatory feedback from the top-layer units to pools of middle-layer units, so that position invariance was achieved to progressively greater degrees in higher layers. O’Reilly and Johnson’s model could be viewed as a more biologically constrained approximation to the more formal learning model proposed here. Hidden Markov models provide another way to implement the model proposed here (Geoff Hinton, personal communication). However, current techniques for fitting HMMs are intractable if state dependencies span arbitrarily long time intervals. Saul and Jordan (1996) have proposed an elegant generalization of HMMs they call Boltzmann chains for modeling discrete time series. In one special case, they show that the learning is tractable for coupled parallel chains, that is, parallel discrete time series of correlated features, coupled by common hidden variables. This case would correspond
370
Suzanna Becker
exactly to the one assumed here (see Figure 2c), if the temporal dependencies were restricted to adjacent points in time. One limitation of the model proposed here is that it does not provide a complete account of the role of feedback between cortical layers. Although top-down feedback could be viewed as just another source of context, and thereby incorporated into the present model, the solution might not be globally optimal in a multistage system. The work of Hinton and colleagues on the Helmholtz machine (Hinton & Zemel, 1994; Dayan, Hinton, Neal, & Zemel, 1995) and Rao and Ballard’s extended Kalman filter model (Rao & Ballard, 1997) provide two different solutions to this problem. 8 Conclusions A “contextual input” stream was implemented in the simplest possible way in the simulations reported here, using fixed delay lines and recurrent feedback. The model we have proposed provides for a very general way of incorporating arbitrary contextual information and could equally well integrate other sources of input. A wide range of perceptual and cognitive abilities could be modeled by a network that can learn features of its primary input in particular contexts. These include multisensor fusion, feature segregation in object recognition using top-down cues, and semantic disambiguation in natural language understanding. Finally, our model may be able to account for the interaction between multiple memory systems in the brain. For example, it is widely believed that memories are stored rapidly in the hippocampus and related brain structures, and more gradually stored in the parahippocampal and neocortical areas (McClelland, McNaughton, & O’Reilly, 1995). The manner in which information is represented in the hippocampal system is undoubtedly very different from that of the cortex. A major question is how the two systems interact. The model proposed here may be able to explain how interactions between disparate information sources such as the hippocampal and cortical codes are integrated into a unified representation in the cortex. The output of the hippocampus, a rapidly formed novel code, could be treated simply as another source of context, to be integrated with bottom-up information received by various cortical areas. Acknowledgments The ideas in this article arose in the context of many discussions with Ron Racine and Larry Roberts about cortical circuitry and plasticity. Thanks to Geoff Hinton for contributing several valuable insights about the model and to Gary Cottrell, Peter Dayan, Darragh Smyth, and four anonymous reviewers for invaluable comments on earlier drafts. The face images were collected by Ken Seergobin. All simulations were carried out using the Xerion neural network simulator developed in Hinton’s lab and additional
Implicit Learning in 3D Object Recognition
371
software written by Lianxiang Wang. Financial support for this work was provided by the McDonnell-Pew Program in Cognitive Neuroscience and the Natural Sciences and Engineering Research Council of Canada. References Becker, S. (1993). Learning to categorize objects using temporal coherence. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems, 5 (pp. 361–368). San Mateo, CA: Morgan Kaufmann. Becker, S. (1997). Learning temporally persistent hierarchical representations. In M. Mozer, M. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems, 9 (pp. 824–830). Cambridge, MA: MIT Press. Becker, S., & Hinton, G. E. (1992). A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355, 161–163. Bridle, J. S. (1990). Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. In D. S. Touretzky (Ed.), Neural information processing systems, 2 (pp. 111–217). San Mateo, CA: Morgan Kaufmann. Bruce, V. (1997). Human face perception and identification. In NATO ASI on Face Recognition. Bruce, V., & Valentine, T. (1998). When a nod’s as good as a wink: The role of dynamic information in facial recognition. In M. Gruneberg, P. E. Morris, & R. N. Sykes (Eds.), Practical aspects of memory: Current research and issues (Vol. 1, pp. 169–174). New York: Wiley. Cacciatore, T. W., & Nowlan, S. J. (1994). Mixtures of controllers for jump linear and nonlinear plants. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 719–726). San Mateo, CA: Morgan Kaufmann. Calvin, W. H. (1995). Cortical columns, modules, and Hebbian cell assemblies. In M. Arbib (Ed.), The handbook of brain theory and neural networks. Cambridge, MA: MIT Press. Christie, F., & Bruce, V. (1998). The role of dynamic information in the recognition of unfamiliar faces. Memory and Cognition, 26, 780–790. Cudeiro, J., & Sillito, A. M. (1996). Spatial frequency tuning of orientationdiscontinuity-sensitive corticofugal feedback to the cat lateral geniculate nucleus. Journal of Physiology, 490, 481–492. Dayan, P., Hinton, G. E., Neal, R., & Zemel, R. S. (1995). The Helmholtz machine. Neural Computation, 7, 1022–1037. De Angelis, G. C., Ohzawa, I., & Freeman, R. D. (1995). Receptive-field dynamics in the central visual pathways. Trends in Neurosciences, 18, 451–458. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Proceedings of the Royal Statistical Society, B-39, 1–38. Desimone, R., Albright, T. D., Gross, G., & Bruce, C. (1984). Stimulus-selective properties of inferior temporal neurons in the macaque. Journal of Neuroscience, 4(8), 2051–2062.
372
Suzanna Becker
Dong, D. W., & Atick, J. J. (1995). Temporal decorrelation: A theory of lagged and nonlagged responses in the lateral geniculate nucleus. Network: Computation in Neural Systems, 6, 159–178. Douglas, R., & Martin, K. (1990). Neocortex. In G. M. Shepherd (Ed.), The synaptic organization of the brain (pp. 389–438). New York: Oxford University Press. Edelman, S., & Weinshall, D. (1991). A self-organized multiple-view representation of 3D objects. Biological Cybernetics, 64(3), 209–219. Fodli´ ¨ ak, P. (1991). Learning invariance from transformation sequences. Neural Computation, 3(2), 194–200. Gilbert, C. D., Das, A., Ito, M., Kapadia, M., & Westheimer, G. (1996). Spatial integration and cortical dynamics. Proceedings of the National Academy of Sciences, 93, 615–622. Gross, C. G., Rocha-Miranda, C. E., & Bender, D. B. (1971). Visual properties of neurons in inferotemporal cortex of the macaque. Journal of Physiology, 35, 96–111. Hess, D. J., Foss, D. J., & Carroll, P. (1995). Effects of global and local context on lexical processing during language comprehension. Journal of Experimental Psychology: General, 124(1), 62–82. Hinton, G. E., & Zemel, R. S. (1994). Autoencoders, minimum description length, and Helmholtz free energy. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 3–10). San Mateo, CA: Morgan Kaufmann. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation, 3(1), 79–87. Kay, J., & Phillips, W. A. (1997). Activation functions, computational goals, and learning rules for local processors with contextual guidance. Neural Computation, 9(4), 895–910. Levitt, J. B., Kiper, D. C., & Movshon, J. A. (1994). Receptive fields and functional architecture of macaque V2. Journal of Neurophysiology, 71(6), 2517–2541. Linsker, R. (1988, March). Self-organization in a perceptual network. IEEE Computer, 21, 105–117. MacDonald, J., & McGurk, H. (1978). Visual influences on speech perception processes. Perception and Psychophysics, 24(3), 253–257. McClelland, J. L., McNaughton, B. L., & O’Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3), 419–457. McClelland, J. L., & Rumelhart, D. E. (1981). An interactive activation model of context effects in letter perception, part I: An account of basic findings. Psychological Review, 88, 375–407. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748. Mel, B. W. (1994). Information processing in dendritic trees. Neural Computation, 6(6), 1031–1085. Miyashita, Y. (1988). Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature, 335, 817–820. Mundel, T., Dimitrov, A., & Cowan, J. D. (1997). Visual cortex circuitry and
Implicit Learning in 3D Object Recognition
373
orientation tuning. In M. Mozer, J. Jordan, & T. Petsche, (Eds.), Advances in neural information processing systems, 9. Cambridge, MA: MIT Press. Neely, J. (1991). Semantic priming effects in visual word recognition: A selective review of current findings and theories. In D. Besner & G. W. Humphreys (Eds.), Basic processes in reading: Visual word recognition (pp. 264–336). Hillsdale, NJ: Erlbaum. Nowlan, S. J. (1990). Maximum likelihood competitive learning. In D. S. Touretzky (Ed.), Neural information processing systems, 2 (pp. 574–582). San Mateo, CA: Morgan Kaufmann. Nowlan, S. J., & Sejnowski, T. J. (1993). Filter selection model for generating visual motion signals. In S. Hanson, J. D. Cowan, & L. Giles (Eds.), Advances in neural information processing systems, 5 (pp. 369–376). San Mateo, CA: Morgan Kaufmann. Oram, M., & Perrett, D. (1994). Modeling visual recognition from neurobiological constraints. Neural Networks, 7(6/7), 945–972. O’Reilly, R. C., & Johnson, M. H. (1994). Objection recognition and sensitive periods: A computational analysis of visual imprinting. Neural Computation, 6(3), 357–389. Perrett, D. I., Hietanen, J. K., Oram, M. W., & Benson, P. J. (1992). Organization and functions of cells responsive to faces in the temporal cortex. Philosophical Transactions of the Royal Society of London, B, 335, 23–30. Perrett, D. I., Rolls, E. T., & Caan, W. (1982). Visual neurones responsive to faces in the monkey temporal cortex. Experimental Brain Research, 47, 329–342. Phillips, W. A., Kay, J., & Smyth, D. (1995). The discovery of structure by multistream networks of local processors with contextual guidance. Network, 6, 225–246. Pouget, A., & Sejnowski, T. J. (1997). Spatial transformations in the parietal cortex using basis functions. Journal of Cognitive Neuroscience, 9(2), 222–237. Rao, R. P. N., & Ballard, D. H. (1997). Dynamic model of visual recognition predicts neural response properties in the visual cortex. Neural Computation, 9(4), 721–764. Ringach, D. L., Hawken, M. J., & Shapley, R. (1997). Dynamics of orientation tuning in macaque primary visual cortex. Nature, 387, 281–284. Saul, L. K., & Jordan, M. I. (1996). Exploiting tractable substructures in intractable networks. In D. Touretzky, M. Mozer, & M. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 486–492). Cambridge, MA: MIT Press. Seergobin, K. (1996). Unsupervised learning: The impact of temporal and spatial coherence on the formation of visual representations. Unpublished master’s thesis, McMaster University. Sillito, A. M., Grieve, K. L., Jones, H. E., Cudeiro, J., & Davis, J. (1995). Visual cortical mechanisms detecting focal orientation discontinuities. Nature, 378, 492–496. Stewart Bartlett, M., & Sejnowski, T. J. (1997). Viewpoint invariant face recognition using independent component analysis and attractor networks. In M. Mozer, M. Jordan, & T. Petsche (Eds.), Neural information processing systems, 9 (pp. 817–823). Cambridge, MA: MIT Press.
374
Suzanna Becker
Stewart Bartlett, M., & Sejnowski, T. J. (1998). Learning viewpoint invariant face representations from visual experience in an attractor network. Network, 9, 399–417. Stone, J. (1996). Learning perceptually salient visual parameters using spatiotemporal smoothness constraints. Neural Computation, 8, 1463–1492. Tanaka, K., Fujita, I., Kobatake, E., Cheng, K., & Ito, M. (1993). Serial processing of visual object-features in the posterior and anterior parts of the inferotemporal cortex. In T. Ono, L. R. Squire, M. E. Raichle, D. I. Perrett, & M. Fukuda (Eds.), Brain mechanisms of perception and memory, from neuron to behavior (pp. 34–46). New York: Oxford University Press. Tanaka, K., Saito, H., Fukada, Y., & Moriya, M. (1991). Coding visual images of objects in the inferotemporal cortex of the macaque monkey. Journal of Neurophysiology, 66(1), 170–189. Wallis, G., & Rolls, E. T. (1997). Invariant face and object recognition in the visual system. Progress in Neurobiology, 51(2), 167–194. Weinshall, D., Edelman, S., & Bulthoff, ¨ H. H. (1990). A self-organizing multipleview representation of 3D objects. In D. S. Touretzky (Ed.), Advances in neural information processing systems, 2 (pp. 274–282). San Mateo, CA: Morgan Kaufmann. Yamane, S., Kaji, S., & Kawano, K. (1988). What facial features activate face neurons in inferotemporal cortex of the monkey. Experimental Brain Research, 73, 209–214. Received June 24, 1997; accepted April 2, 1998.
NOTE
Communicated by Klaus Obermayer
Relation Between Retinotopical and Orientation Maps in Visual Cortex Udo Ernst Max-Planck Institute for Fluid Dynamics, D-37018 G¨ottingen, Germany
Klaus Pawelzik Institute for Theoretical Physics, Universit¨at Bremen, D-28359 Bremen, Germany
Misha Tsodyks Department of Neurobiology, Weizmann Institute of Science, Rehovot 76100, Israel
Terrence J. Sejnowski Computational Neurobiological Laboratory, Salk Institute, La Jolla, CA 92037, U. S. A. and Department of Biology, University of California, San Diego, La Jolla, CA 92093, U.S.A.
A recent study of cat visual cortex reported abrupt changes in the positions of the receptive fields of adjacent neurons whose preferred orientations strongly differed (Das & Gilbert, 1997). Using a simple cortical model, we show that this covariation of discontinuities in maps of orientation preference and local distortions in maps of visual space reflects collective effects of the lateral cortical feedback.
Theoretical analysis of the role of lateral interactions in the cooperative behavior of neuron ensembles resulted in two main conclusions: 1. Lateral interactions may create a continuum of localized stable states (Ben-Yishai, Bar-Or, & Sompolinsky, 1995). 2. Inhomogeneities in lateral connections break the continuity of these states, leading to clustering at particular locations in the neural net (Tsodyks & Sejnowski, 1995). The visual cortex clustering would imply that receptive fields of nearby neurons exhibit discontinuities in their features. We tested this hypothesis by simulating a network of interacting neocortical neurons with the architecture of primary visual cortex. In such a network, the receptive field properties of a neuron result from both the pattern of external inputs from lateral geniculate nucleus (LGN) and the pattern of lateral connections (Ernst, Pawelzik, Wolf, & Geisel, 1997). Neural Computation 11, 375–379 (1999)
c 1999 Massachusetts Institute of Technology °
376
U. Ernst, K. Pawelzik, M. Tsodyks, and T. Sejnowski
The network is formed by nx ×ny columns consisting of excitatory (index e) and inhibitory (index i) neuronal populations, each receiving afferent input Iaf f from the LGN, lateral excitatory input Ielat , and lateral inhibitory input Iilat . The population dynamics for column j reads, τe ·
dAe (j, t) af f lat = −Ae (j, t) + ge (Iee (j, t) + Iielat (j, t) + Ie (j, t)), dt
(1)
τi ·
dAi (j, t) af f = −Ai (j, t) + gi (Ieilat (j, t) + Iiilat (j, t) + Ii (j, t)). dt
(2)
ge and gi are piecewise linear gain functions (threshold linear neurons) with firing thresholds te , ti and slopes se , si , and τi , τe are time constants. We assume that the connections W from one subpopulation to excitatory and inhibitory subpopulations do not differ significantly except for the total interaction strengths w, and therefore define the synaptic input as:
lat (j, t) = wee · Iee
N X
We (j, k)Ae (k, t)
(3)
We (j, k)Ae (k, t)
(4)
Wi (j, k)Ai (k, t)
(5)
Wi (j, k)Ai (k, t).
(6)
j=0
Ieilat (j, t) = wei ·
N X j=0
Iielat (j, t) = wie ·
N X j=0
Iiilat (j, t) = wii ·
N X j=0
We assume that the strength of lateral connections depends on the proximity of the columns and relative angle between their preferred orientations: Ã
! Ã ! −| Er(j) − Er(k) |2 −| 8(j) − 8(k) |2 · exp exp (7) We (j, k) = 2 · σer 2 2 · σe8 2 Ã ! Ã ! −| Er(j) − Er(k) |2 −| 8(j) − 8(k) |2 0 exp . (8) Wi (j, k) = Wi · exp 2 · σir 2 2 · σi8 2 We0
The constants W 0 normalize the coupling matrices W such that the average coupling strength to other neurons is one. The LGN inputs Iaf f were assumed to originate from retinotopic locations uniformly distributed over
Relation Between Retinotopical and Orientation Maps in Visual Cortex
377
the visual field and to include an orientation bias given by a map of preferred orientations obtained by optical imaging (Bonhoeffer & Grinvald, 1991). The visual field has the same dimensions nx , ny such that one unit square corresponds to one column in the cortex.
2 (k) − 8(j) | | 8 stim (9) M(k) · (1 − ²) + ² · exp − Iaf f (j, t) = 82 2 · σ k=1 rf | Er(k) − Er(j) |2 (10) · exp − 2 · σrfr 2 N X
af f
af f
Ie (j, t) = we · Iaf f (j, t)
(11)
af f Ii (j, t)
(12)
=
af f wi
·I
af f
(j, t).
M(k) denotes a stimulus mask that is 0 if there is no stimulus at position Er(k) in the visual field and 1 otherwise. The model cortex has been stimulated with gratings of radius ρ = 2 in eight different orientations at each position of the visual field. While presenting these localized oriented stimuli, the network was allowed to converge to a solution. Receptive fields were obtained as the set of locations where presentation of stimuli lead to activation of a neuron (see Figure 1). Orientation maps consist of regions where preferred orientation smoothly changes with the cortical location of neurons, separated by a set of discontinuities of preferred orientation called pinwheels and fractures. We assumed (see equation 8) that in accordance with anatomical evidence, lateral connections between a pair of neurons depend on their proximity in the cortex and the relative angle between their preferred orientations (Gilbert & Wiesel, 1985; Malach, Amir, Harel, & Grinvald, 1993). This implies that a pair of neurons located on opposite sides of a discontinuity have a weaker connection than the equally separated pair in the smooth region. Correspondingly, the localized stable states of the network will tend to cluster in the smooth regions, where interactions are stronger. Therefore, a stimulus moving across the visual field will cause a jump of activity across the fracture, leading to a corresponding jump in the locations of the receptive field centers (see Figure 1). Our results demonstrate that lateral connections could play a crucial role in determining the retinotopic map in the visual cortex by causing a mismatch between the pattern of LGN inputs and the resulting receptive field properties. Subsequent development of connections between LGN and cortex could eliminate this mismatch (e.g., using Hebbian plasticity mechanisms).
378
U. Ernst, K. Pawelzik, M. Tsodyks, and T. Sejnowski
Figure 1: (A) Optical map of orientation preference with recording sites arranged on a line that crosses a fracture. A fragment of a map obtained in Bonhoeffer and Grinvald (1991) was used. Orientations are color coded according to the color bars below. (B) Receptive fields of neurons at the recording sites in A, shown with the color corresponding to their optimal orientations. (C) Shifts in the receptive fields positions, normalized by the size of the receptive fields, for nearby neurons versus change in their preferred orientations. The parameters for this simulation were nx = 45, ny = 15, τe = τi = 1.0, se = 1.5, si = 3.0, te = 0.3, ti = 0.6, wee = 1.3, wei = 1.0, wii = 0.2, wie = 1.5, σer = 1.5, σe8 = 30o , σir = 3.5, σi8 = 700o , ² = 0.75, σrfr = 0.5, and σrf8 = 30o .
Relation Between Retinotopical and Orientation Maps in Visual Cortex
379
References Ben-Yishai, R., Bar-Or, R. L., & Sompolinsky, H. (1995). Theory of orientational tuning in visual cortex. PNAS, 92, 3844–3848. Bonhoeffer, T., & Grinvald, A. (1991). Iso-orientation domains in cat visual cortex are arranged in pinwheel-like patterns. Nature, 353, 429–431. Das, A., & Gilbert, C. D. (1997). Distortions of visuotopic map match orientation singularities in primary visual cortex. Nature, 387, 594–598. Ernst, U., Pawelzik, K., Wolf, F., & Geisel, T. (1997). In M. C. Mozer, M. I. Jordan, & T. Petsche (Eds.), Orientation contrast sensitivity from long-range interactions in visual cortex. Advances in neural information processing systems, 9 (pp. 90–96). Cambridge, MA: MIT Press. Gilbert, C. D., & Wiesel, T. N. (1985). Intrinsic connectivity and receptive field properties in visual cortex. Vision Res., 25, 365–374. Malach, R., Amir, Y., Harel, M., & Grinvald, A. (1993). Relationship between intrinsic connections and functional architecture revealed by optical imaging and in vivo targeted biocytin injections in primate striate cortex. PNAS, 90, 10469–10473. Tsodyks, M., & Sejnowski, T. (1995). Associative memory and hippocampal place cells. Int. J. Neural Sys., 6, 81–86. Received February 23, 1998; accepted June 17, 1998.
LETTER
Communicated by Richard Zemel
A Parallel Noise-Robust Algorithm to Recover Depth Information from Radial Flow Fields F. Worg ¨ otter ¨ A. Cozzi Department of Neurophysiology, Ruhr-Universit¨at, 44780 Bochum, Germany
V. Gerdes ¨ Informatik III, Universit¨at Bonn, Bonn, Germany Institut fur
A parallel algorithm operating on the units (“neurons”) of an artificial retina is proposed to recover depth information in a visual scene from radial flow fields induced by ego motion along a given axis. The system consists of up to 600 radii with fewer than 65 radially arranged neurons on each radius. Neurons are connected only to their nearest neighbors, and they are excited as soon as a sufficiently strong gray-level change occurs. The time difference of two subsequently activated neurons is then used by the last-excited neuron to compute the depth information. All algorithmic calculations remain strictly local, and information is exchanged only between adjacent active neurons (except for the final read-out). This, in principle, permits parallel implementation. Furthermore, it is demonstrated that the calculation of the object coordinates requires only a single multiplication with a constant, which is dependent on only the retinal position of the active neuron. The initial restriction to local operations makes the algorithm very noise sensitive. In order to solve this problem, a prediction mechanism is introduced. After an object coordinate has been determined, the active neuron computes the time when the next neuronal excitation should take place. This estimated time is transferred to the respective next neuron, which will wait for this excitation only within a certain time window. If the excitation fails to arrive within this window, the previously computed object coordinate is regarded as noisy and discarded. We will show that this predictive mechanism relies also on only a (second) single multiplication with another neuron-dependent constant. Thus, computational complexity remains low, and noisy depth coordinates are efficiently eliminated. Thus, the algorithm is very fast and operates in real time on 128×128 images even in a serial implementation on a relatively slow computer. The algorithm is tested on scenes of growing complexity, and a detailed error analysis is provided showing that the depth error remains very low in most cases. A comparison to standard flow-field analysis shows that our algorithm outperforms the
Neural Computation 11, 381–416 (1999)
c 1999 Massachusetts Institute of Technology °
382
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
older method by far. The analysis of the algorithm also shows that it is generally applicable despite its restrictions, because it is fast and accurate enough such that a complete depth percept can be composed from radial flow field segments. Finally, we suggest how to generalize the algorithm, waiving the restriction of radial flow. 1 Introduction During the projection of the three-dimensional environment onto the twodimensional receptor surfaces of the eyes, depth information is lost. Several ways exist for recovering depth from these projection images. Many biological and technical systems rely on the analysis of stereo image pairs. In these systems, depth information is retrieved from the analysis of the local image differences between the left and the right image (called disparities), which result from the lateral displacement of the two eyes or cameras (e.g., correlation-based methods: Marr & Poggio, 1976; phase-based methods: Sanger, 1988; Fleet, Jepson, & Jenkin, 1991; for a review of the older work, see Poggio & Poggio, 1984; a recent review is by Qian, 1997). If the viewer or the objects are moving, the motion pattern can be analyzed instead in order to obtain depth information (Ullman, 1979; Prazdny, 1980; LonguetHiggins & Prazdny, 1980; Lucas & Kanade, 1981; Fennema & Thompson, 1979; Heeger, 1988; Fleet & Jepson, 1990). Ego motion or object motion generates a so-called flow field on the receptor surfaces (see, e.g., Horn & Schunck, 1981; Koenderink, 1986; Barron, Beauchemin, & Fleet, 1994a; Barron, Fleet, & Beauchemin, 1994b). The projection of the displaced objects thereby consists of curves of various shapes. In the most general case (object plus ego motion) the curved flow-field patterns cannot be resolved for depth analysis without additional assumptions (rigidity and smoothness constraints; Poggio, Torre, & Koch, 1985; Hildreth & Koch, 1987; Yuille & Ullman, 1987). However, even if simplifying assumptions are made, the problem of structure from motion remains rather complex. The goal of this study is to devise a neuronal algorithm that allows the analysis of radial (diverging) flow fields by the parallel operation of its individual photoreceptive sites (its “neurons”). We will show first that depth information is obtained by a single scalar multiplication with a neurondependent constant at each active neuron. Thus, the algorithm is very simple and so fast that it operates in real time even in our serial computer simulations. The structure of our network is such that all computations remain local, and neurons need “to talk” only to their nearest neighbors, which permits parallelization. The locality of all calculations, however, makes the algorithm initially very noise sensitive. Therefore we will show, second, that the algorithm can be extended by a local predictive mechanism that relies on the propagation of the excitation pattern one step into the network. Prediction of the future excitation pattern requires only one more scalar multiplication at each active neuron. Thus, the computational complexity
Parallel Noise-Robust Algorithm
383
remains low. As a result we will show that this local predictive mechanism almost completely eliminates noise and other errors in the analysis. The restriction to radial flow finds its motivation in the behavior of animals. In different species, varying strategies are observed in order to reduce the optic flow as much as possible to a few or, if possible, a single component. For the housefly (Musca domestica), Wagner (1986, p. 546) stated: “Thus, the flight behavior and the coordination of head and body movements may be interpreted as an active reduction of the image flow to its translational components.” Ideally this would mean that only forward motion exists and that optic flow is reduced to its radial component. In the fly, this actually leads to the tendency to fly along straight lines, making rather sharp turns when changing direction (Wagner, 1986). Flow-field reduction is pushed to an extreme in some birds while they are walking. The intriguing head bobbing of pigeons serves the purpose of eliminating all optic flow while the bird moves its body forward “under” the motionless head (Davies & Green, 1988; Erichsen, Hodos, Evinger, Bessette, & Phillips, 1989). Similarly it has been observed that pigeons and other birds keep their head stable during different flight maneuvers, such that the head pursues a smooth-motion trajectory while the body can make rather jerky movements (Green, Davies, & Thorpe, 1992; Davies & Green, 1990; Erichsen et al., 1989; Wallman & Letelier, 1993). In particular, when very high accuracy is required during landing, the compensatory head movements become very pronounced and accurate such that relatively undisturbed radial flow is obtained (analysis of high-speed video data of free-flying pigeons by J. Ostheim, personal communication). Given the complexity of insect or bird flight, the reduction of the optic flow must remain incomplete; the strategy to reduce the computational complexity of flow-field analysis, however, seems to be pursued widely, even in mammals (e.g., component specificity of MST cells, Duffy & Wurtz, 1991, 1995; Graziano, Anderson, & Snowden, 1994; see also Lappe, Bremmer, Pekel, Thiele, & Hoffmann, 1996; Wang, 1996, for theoretical approaches on medial superior temporal area cells). Under the assumption that flow-field restriction is a biologically justified strategy, the central goal of our study is to arrive at an efficient and noise-robust algorithm that can operate in parallel on a restricted flow field, thereby making use of a task-dedicated artificial neural net architecture. Although the initial motivation comes from biology, it is obvious that the algorithmic transfer of the underlying concept into a more technical domain immediately imposes restrictions with respect to the biological realism of the network. We will describe the algorithm and present results from the analysis of artificial and real image sequences, which demonstrate that depth information is retrieved with very high accuracy. A rather technical appendix provides a detailed error analysis that demonstrates that the algorithm is generally applicable. (This appendix is mainly of relevance for those who wish to implement this algorithm. It may be skipped otherwise.)
384
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
2 Description of the Algorithm The core part of the algorithm consists of a parallel1 operating network of neurons, called the “retina” (see Figure 1), with which we are mainly concerned. In order to describe the algorithm, we assume a moving robot driven by a stepper motor and a visual system consisting of one camera with its camera plane orthogonal to the axis of motion of the robot. Two restrictions are introduced: 1. The robot is assumed to move only along the optical axis of the camera. 2. The environment is regarded as stationary (i.e., no moving objects). The first restriction leads to a purely radial flow field on the camera plane, and this condition seems fatally strong, limiting the algorithm to a special case that exists only during short intervals of robot motion. In particular, during a curve, the focus of expansion is no longer aligned with the motion trajectory, rendering the algorithm useless. Initially the restriction to radial flow was biologically motivated. However, it will almost always be sufficient to approximate the complete “depth percept” by such linear motion segments provided the robot makes rather sharp turns, (similar to the flying pattern of a fly) during which it is “blind.” The high camera frame rates and the speed of the algorithm ensure that a novel depth percept builds up rather fast after a turn, such that periods of “robot blindness” remain short. The second restriction can also be partly waived, as explained in the discussion. 2.1 The Retina. The retina consists of radially arranged neurons that are connected only to their nearest neighbors in both directions on the same radius. In a radial flow field, the virtual speed of the projected objects on the retina increases with the distance from the optical axis, and the retina location of the projected image for the geometrical camera arrangement shown in Figure 3 is inversely proportional to the distance of the object from the nodal point (conventional hyperbolic projection geometry). The goal now is to design an arrangement that takes care of this projection geometry and allows for a uniform sampling of the scene along the locations on the radii during radial flow. We can restrict ourselves to a single radius and define the neuronal density by the hyperbola D(rn ) =
1 rn − rn−1
n ∈ [1, . . . , N],
(2.1)
1 “Parallel” means that this structure can in principle be implemented and operate in parallel. All computational results shown in this study, however, are based on regular workstations, such that all computations are performed serially.
Parallel Noise-Robust Algorithm
385
retina
ϕ
Figure 1: Layout of the retina. Neurons are arranged according to equation 2.4.
where rn is the location of neuron n on a retinal radius and N + 1 the total neuron number on every radius. As a consequence of the hyperbolic projection geometry, we find that the neuronal density D at a given retinal location x should be directly proportional to 1/x. Since we deal with a discrete neuronal placement problem, this reads: D(rn ) ∼
1 . n
(2.2)
This requirement is fulfilled for the following definition of rn : rn =
1 2 1 kn + kn + r0 , 2 2
(2.3)
because in this case we get D(rn ) =
1 1 . = rn − rn−1 kn
Let ρ be the radius of the retina. Then we place one neuron in the center
386
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
of the retina (i.e., r0 = 0) and one on its border (rN = ρ) and get: rn =
ρ · n(n + 1) = h · n(n + 1). N(N + 1)
(2.4)
A VLSI design that basically emulates this structure already exists, but without the required connections between the neurons (Pardo, 1994). 2.2 Flow Diagram of the Algorithm. The individual neurons are designed to perform only very simple operations: reading, comparing, and storing gray-level values; reading the stepper motor count of the robot, computing (two) scalar multiplications; raising or lowering a counter; and transferring the output as specified below. Thus, they contain a “memory” and a simple processing capability. For now, we assume an ideal situation, which consists of noise-free grayscale images. We say that a neuron is excited as soon as the luminance at this neuron changes significantly. In order to explain the algorithm, let us further restrict the situation to a robot that moves in an environment with only a single black-dot object somewhere in the distance. Before the robot starts to move, all neurons will be reset and their memory deleted. At time-step t1 the black dot will excite neuron 1 (see Figure 2A). Since its memory does not contain any information, the neuron will transfer only the gray-level value (“black”) to the next outer neuron (neuron 2). After some time, the projection of the black dot will have traveled to neuron 2 (see Figure 2B). This neuron compares the newly read gray-level value with the one stored in its memory and finds that they are similar within a reasonable range. It will then read the stepper motor count (1Z = 8; see Figure 2B) and compute the cylinder coordinates of the object (R1 , Z1 )ϕ . In addition it will assign a label—say L = α—to this object (see Figure 2B). From the coordinates and the known motion pattern of the robot, neuron 2 can then also compute the stepper motor count, which will be expected at the moment when neuron 3 will be excited (1Zp = 6; see Figure 2B). Neuron 2 will transfer the predicted value (1Zp ), the gray-scale value (“black”), and the label (α) to neuron 3. The coordinates [(R1 , Z1 )ϕ ] of the object, as well as the label (α), will be read by the common read-out to generate the depth map. As soon as neuron 3 is excited (see Figure 2C), it compares the gray-level values; after having found a match, it also compares the predicted and the actual stepper motor count. If they match within a certain tolerance (e.g., 1Z = 1Zp ± 1), the object is regarded as confirmed and the confirmed counter (C; see Figure 2C) is increased. In this way, object positions become more reliable, the detection error is reduced, and false object positions are soon rejected. The object coordinates will be recomputed [(R2 , Z2 )ϕ ], and the object position with label α in the depth map will be updated. In the case that the reconfirm failed (e.g., | 1Z − 1Zp |> 1) the object is considered new, and a new label is assigned to it. In this case, the old object with label
Parallel Noise-Robust Algorithm
387
α is regarded as unreliable and removed from the depth map. If an object has already been confirmed several times, it will not be directly eliminated from the depth map, but the confirmed variable will be lowered gradually until it reaches zero (“slow death”).2 2.3 Equations. Figure 3 shows the geometrical situation for which the equations are defined. Most equations are defined in cylinder coordinates [(R, Z)ϕ ], and only at the end will we give the final result in Euclidean coordinates (X, Y, Z). We will first describe how to obtain the object coordinates (Rn , Zn ) from the excitations of the neurons and then compute the prediction value 1Zp for the next expected excitation occurring at (Rn+1 , Zn+1 ). → In addition, at first we will use vector notation (− s ), which does not impose any restrictions on the geometry, and only later include the already described retinal neuron arrangement. To get the object position, we have to solve the following equation by eliminating k and l: −→ → → s− k·− sn + 1Z = l · − n−1 .
(2.5)
Since we assume that the robot motion contains only a Z-component, we get: ¶ µ µ ¶ ¶ µ 0 sn−1,r sn,r + =l· . (2.6) k· 1Z ϕ sn,z ϕ sn−1,z ϕ Note that the angular component ϕ of the cylinder coordinates is defined by the angle of the neuron chain on the retina. Thus, for each neuron chain, it is constant. For the radial and the Z-component, this reads: k · sn,r = l · sn−1,r ,
and
k · sn,z + 1Z = l · sn−1,z .
(2.7)
From this we get: k=
1Z . s − sn,z n−1,z sn−1,r sn,r
The actual object position can now be computed by: µ ¶ ¶ µ −−→ sn,r R =k· = 1Z · (Pn )ϕ . sn,z ϕ Z ϕ
(2.8)
(2.9)
2 In the current implementation of the algorithm, all information exchange remains local and thus restricted to subsequent neurons. In this case, slow death leads only to the prolonged persistence of highly confirmed objects. In a more elaborate version, information transfer could be implemented over more than two neurons, such that Zp is computed for them. This could better compensate for single misses.
388
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Retina
A
neuron 1 neuron 2 neuron 3 neuron 4 neuron 1
neuron 2
Steps
analysis
Memory col. ∆z ∆zP L C Bank
analysis
col. ∆z ∆zP L C
0 0 0 0
Depth Map
B
Retina
C
neuron 1 neuron 2 neuron 3
Retina neuron 1 neuron 2 neuron 3
neuron 4 neuron 2
neuron 4 neuron 3
neuron 3
neuron 4
Steps
analysis
col. ∆z ∆zP L C 8 0 α 0
Steps
analysis
col. ∆z ∆zP L C 0 6 α 0
analysis
col. ∆z ∆zP L C 5 6 α 0
Depth Map
α (R1,Z1)ϕ
analysis
col. ∆z ∆zP L C 0 8 α 1
Depth Map
α (R2,Z2)ϕ
Figure 2: Flow diagram of the algorithm.
So far the equations do not impose any restrictions on the neuron positioning and also leave other geometrical constants open. However, it makes → sense to assume that the Z-components of the vectors − s are identical to the focal length (i.e., sn−1,z = sn,z = f ). Furthermore from equation 2.4, we get: rn n+1 sn,r . = = sn−1,r rn−1 n−1
(2.10)
Parallel Noise-Robust Algorithm
389
object position(Rn-1,Zn-1)
ϕ
∆Z object position(Rn,Zn)
Sn
ϕ
object position(Rn+1,Zn+1) ϕ
Zn
Sn-1
Sn+1
lens
Z=0
f (focal length)
R
retina
Sn-1,r S n,r Sn+1,r
Figure 3: Geometry of the projection of a black-dot image onto one radius of the retina. This geometry defines the equations used to compute the object coordinates.
Both assumptions now lead to (for the definition of h see equation 2.4): ¶ µ 1 −−→ h · n(n + 1) (Pn )ϕ = ¡ n+1 ¢ f f n−1 − 1 ϕ Ã Ã ! ! 3 =
h·n(n−1)(n+1) 2f n−1 2
= ϕ
h·(n −1) 2f n−1 2
.
(2.11)
ϕ
The general form reads in Euclidean coordinates: X sn,r · cos ϕ → Y = k · sn,r · sin ϕ = 1Z · − Pn . sn,z Z
(2.12)
Again imposing the geometrical restrictions, we get: − → Pn =
h·(n3 −1) · cos ϕ 2f h·(n3 −1) · sin ϕ 2f n−1 2
.
(2.13)
390
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Thus, the object position is obtained from a single multiplication for each − → component of the stepper motor count 1Z with Pn . It should be noted −−→ − → that Pn [or (Pn )ϕ in equation 2.11) is constant but different for each neuron. Thus, the multiplication is effectively reduced to a scaling operation with a different scalar factor at each neuron, which is the central feature on this algorithm, making it exceedingly simple. In the second step, the prediction value 1Zp will be computed using the radial component R from the first computation. In the general case we get: Zn sn,z = sn,r R
and
sn+1,z Zn+1 . = sn+1,r R
Since 1Zp = Zn − Zn+1 we get: µ ¶ sn,z sn+1,z − . 1Zp = R sn,r sn+1,r With the same geometrical restrictions as before, this is: ¶ µ 2f . 1Zp = R · h · n(n + 1)(n + 2)
(2.14)
(2.15)
(2.16)
The radial component R of the cylinder coordinates has been stored from the calculation of the object position. Therefore, this equation amounts to only a simple scaling operation because everything except R is constant. 2.4 Results. We will first show the results obtained with artificial images of increasing realism and then how the algorithm works in a real environment. In the appendix, we present a detailed analysis of the inherent error sources of the algorithm. All results were obtained on a SUN SPARC 10 workstation. 2.4.1 Results on Artificial Images. In a more realistic environment, objects can no longer be described as single dots. Therefore, for the following scenes, we used the color transition that occurs at an edge as the excitation criteria for the neurons. Consequently, all depth maps are defined only at the edges of the objects. We share this characteristic with all depth analysis algorithms that do not introduce additional regularization schemes. Figure 4 shows the results from the analysis of an artificial environment without (A) and with noise (C), which consists of three flat objects of different geometry (triangle, vertical bar, square) located at different distances (4.0 m, 5.5 m, 7.0 m, B) in front of a background 10.5 m away. Quantitative diagrams to supplement the results of Figure 4 are shown in Figure 5. Parameters for this test are given in Table 1.3 The parameter ²Position indicates the maximal position error (here, 3 In this section, we will focus on the description of the basic findings; therefore, we refer readers to the appendix for an explanation of some of the non-self-explanatory
Parallel Noise-Robust Algorithm
391
Table 1: Parameters for the Simulation Shown in Figure 4. Parameter
Value
Sequence Resolution Step ²Position
800 images 160 × 150 pixel 5 mm/image 10 mm
Parameter Neuron chains Neurons per chain N Retina radius ρ ²Displacement
Value 600 ≤ 50 ≤ 105 pixel 0.05 pixel
1.0 cm) allowed between two subsequent depth estimates arising from two adjacent neurons. If this parameter is exceeded, the second depth estimate is rejected, and the point is not included in the depth map. Figure 4 shows the changing retinal projection on the left side (column D). The other panels demonstrate how the depth map (columns E, H, I) and the confirmation map (columns F, G) for this scene evolve over the 800 steps of simulated robot movement, equivalent to 4 m of traveled distance. The confirmation maps show the gray-scale-coded value of the confirmed counter (light gray=0, darker gray=1, etc.), whereas the depth maps show all coordinates that had a confirmed value as indicated in the figure. The left side was computed for the noise-free scene; for the right side, 25% of random noise was added to each individual frame.4 After about 50 cm, the first data points become confirmed more than once, and after 1 m, the outline of all obstacles is clearly visible in the case of no noise. With noise, only a few data points are confirmed twice, but after 1.5 m (not shown) enough data points are reliable to perform for example obstacle avoidance. The bottom part of the figure (J–Q) shows side-view maps of the obtained depth estimates for different confirmed values after the complete run. The horizontal lines pointing left from the start of the robot motion indicate the distance traveled by the robot. The diagrams show that the depth estimates are very accurate. Pixels overlay each other such that the total number of depth estimates cannot be deduced from the side-view maps (but see the histograms in Figure 5). In such an artificial situation, even a confirmed value of zero leads to good results if no noise is present. Increasing the confirmed value leads to more rejections of data points and a reduced density of the depth map. With noise, a confirmed value of 2 is a good compromise between the accuracy of the depth estimates and the density of the map.
parameters (e.g., ²Displacement ) in Table 1. 4 The maximally allowed noise amplitude was 25% of the maximal gray-scale difference between the darkest and the brightest pixel in all frames. For every pixel, a random number was drawn (flat distribution) between −12.5% and +12.5%, and this value was added to the pixel value, clipping at 0 and 255 if necessary.
392
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Scene
Depth Image
A
noisy Scene
C
B
Retina ProjectionDepth Map
E
F
G
H
I
J
Confirmed = 0
K
Confirmed = 1
L
Confirmed = 2
backgd. bar square triangle
M
Side View Map
N Start of Robot Motion
Start of Robot Motion
400cm
300cm
200cm
100cm
50cm
D
Depth Map Depth Map Confirmed = 2 Confirmed = 3
Confirmed Maps
O P
Q
Confirmed = 3
Side View Map
Figure 4: Different stages of the retrieval of depth information from an artificial noise-free (left) and noisy scene (right). Side-view maps are given at the bottom.
Parallel Noise-Robust Algorithm
without noise
800
Depth Estimates
393
A
with noise 500
confirmed=0 1
600
2
confirmed=0 1
3
300 400
400 cm
2
400 cm 3
200
200
100
0
0 0
Cumulative Depth Estimates
B
400
1800
200
400 600 800 1000 Depth [cm]
0
1600
C
confirmed = 0
1400
D
400 600 800 1000 Depth [cm]
confirmed = 1
1200
1
1000
800
2
600 200 0 0 cm
200
100
200 300 Depth [cm]
2
400
3
3 400
0 0 cm
100 200 Depth [cm]
300
400
Figure 5: (A, B) Depth histograms for the scene shown in Figure 4. Note four curves overlay each other in A and B, corresponding to the values of confirmed = 0, 1, 2, 3. In the insets, the four curves can be discerned. Each peak corresponds to one object. (C, D) Cumulative diagram of the number of depth estimates obtained from the triangle in Figure 4 along the robot motion trajectory. Different diagrams for confirmed = 0, 1, 2, 3 are shown in A–D.
To get a better estimate of the quality of the results, we have plotted the detected depth values for the different confirmed values as histograms in Figure 5. The very narrow histograms confirm the high accuracy of the depth map, as already suggested by the side-view maps. Opposite to those, however, the histograms quantify the total number of depth estimates, which decreases with higher confirmed values. To be able to discern this effect, magnifications of the peaks from the “triangle” located at distance 4.0 m are shown in the insets. A second aspect of interest in this context is how fast a reliable outline of an object is obtained. Parts C and D of Figure 5 demonstrate for different values of the confirmed variable how the depth estimates accumulate along the robot’s path toward an object. Even in the noisy scene (D), more than 300 depth estimates are obtained for a confirmed value of 2 until the robot is only 1 m away from the object. This amount of estimates should suffice for most applications.
394
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Table 2: Parameters for the Simulation Shown in Figure 6. Parameter Sequence Resolution Step
Value 600 images 128 ×120 pixel 0.5 cm/image
Parameter Neuron chains Neurons per chain N Retina radius ρ
Value 400 ≤ 64 ≤ 80 pixel
Parameter Confirmed ²Position ²Displacement
Value 1 0.5 cm 0.25 pixel
2.4.2 Results in a Seminatural Environment. In the next step we used a scene generated by a ray tracer that resembles the operating environment of an office robot, simulating a hallway with a few obstacles and several light sources (see Figure 6). While this scene (A) is more realistic than the benchmarking scene used before, it still contains no real error sources (like jitters from the robot motion). Parameters for this simulation are given in Table 2. Figure 6 shows the retinal projection of the first frame (B), where each neuron stores that particular gray level of the image at the respective location with which it is confronted. In part C, the ground-truth depth map is given. The other parts show two snapshots: one after 300 frames and the other at the end of the simulation (frame 600) of the depth map (global depth map for confirmed ≥ 1, right). In addition, we show the current depth map, which reflects the depth values computed from those neurons that are excited within a small time window of ±20 frames around the current frame. Sideview maps shown beneath the depth maps clearly demonstrate that the density of the depth map increases during the run and also show that the depth estimates are rather accurate. As expected, depth errors increase with distance (compare, e.g., the ball and background). Note as before that pixels overlay each other, apparently reducing the density of the side-view map. Some of the pixels lie on the floor and reflect detected shadows. Only a few more are floating in the air, representing erroneous depth estimates, which, however, are still located rather close to the real objects. 2.5 Results for a Real Scene. Figures 7 and 8 show the results we obtained by analyzing a real scene that was recorded in 540 frames over 72 seconds using an NTSC zoom CCD-camera (1/2 inch CCD chip) with autofocus. The viewing angle of the camera was 55.2 × 44.1 degrees, and the focal length was approximately 8 mm (slightly changing during the run because of the autofocus). We used a DataCube as frame grabber with an initial resolution of 512 × 480 pixels and a frame rate of 7.5 Hz. The images were then subsampled by a factor of two, leading to a final resolution of 256 × 240, and then immediately transferred to a SUN SPARC 10 computer for analysis. The camera was mounted on a small vehicle. No special procedure was adopted to adjust the camera axis. Adjustment to the motion trajectory was performed only by hand while viewing the image on a regular monitor.
Parallel Noise-Robust Algorithm
A
395
C
B
Depth Image
Retina Projection
Image #1 (start)
D
E
Image #300 (1.5 m)
F
global Depth Map
current Depth
5m 4
G
3 2 1 0 0
H
1
3
5
7
11 m
J
I
current Depth
Image #600 (3.0 m)
9
global Depth Map
5m
K
4 3 2 1 0 0
1
3
5
7
Figure 6: Results from the Ray-Traced Hallway Scene.
9
11 m
396
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Scene
Depth Map
Confirmed Map
A
F
G
H
I
J
K
B
C
D
E
Top View
Side View
L
M
Scissors m&m
Pliers Elephant
Box edge Box edge
Axle support
0
30
60
90
120 cm
0
30
60
90
120 cm
Figure 7: Results for a real scene. (A–E) Frames 100, 200, 300, 400, and 540. (F, H, J) Confirmation maps for frames 100, 300, and 540. (G, I, K) Depth maps for C ≥ 1 for frames 100, 300, and 540. Different gray shades encode the relative depth of the objects. (L, M) Side-view and top-view map showing all data points in the depth map of frame 540.
Parallel Noise-Robust Algorithm
397
Y X
Z
Figure 8: Aerial view onto the data clusters obtained from the real scene. Different shades show the different objects. The black outliers along the z-axis belong to the right rail, which was otherwise thresholded.
The vehicle was pulled across a table guided by two lateral rails. Pulling was achieved by means of a thin thread, which was continuously wound onto the extended axle of an electrical motor (visible in Figure 7D, left). The total traveled distance was 1.2 m at a velocity of 16.6 mm/s, which amounts to 2.22 mm per camera frame. The scene contained a pliers with their center of gravity at a distance of 0.35 m from the starting point, an elephant on a post (0.55 m), the M&M mascot (0.90 m), a white box that hides the motor, the axle of the motor and the axle support (all at about 1.10 m), and a scissors on the wall (1.20 m). The white box, the aluminum rails, and the (barely visible) thread were blanked out before analysis by thresholding all low-contrast objects. Apart from this thresholding, no other preprocessing of the image data was performed, and the algorithm operated at the unprocessed noisy
398
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Table 3: Parameters for the Analysis of the Real Scene Shown in Figure 7. Parameter Sequence Resolution Step
Value 540 images 256 × 240 pixel 2.22 mm/frame
Parameter Neuron chains Neurons per chain N Radius ρ
Value 600 ≤ 64 ≤ 150 pixel
Parameter Confirmed ²Position ²Displacement
Value 1 1.2 mm 0.10 pixel
gray values as they were recorded. Note that this scene—like all other realworld scenes—is contaminated by reflections and shadows, which could in principle influence the final results. Table 3 lists the parameters of the algorithm used to analyze the scene. Parts F, H, and J show the confirmation map for frames 100, 300, and 540, respectively. The confirmation maps show the gray-scale-coded value of the confirmed counter (light gray = 0, darker gray = 1, etc.). Thus, the confirmation map contains all data points so far encountered up to that particular frame. Most of these data points are encountered only once, which leads to a value of the confirmed counter of C = 0 (lightest shading). Some of them occur more often (C ≥ 1). Panels G, I, and K represent the accumulated depth maps for the same frames showing only those data points confirmed at least once (C ≥ 1). The outlines of the different objects are clearly visible and in good focus. The different gray shading indicates the distance of the objects from the starting point in absolute coordinates. After 100 frames (compare A), the closest objects, still at a safe distance, can be discerned, which would allow for steering maneuvers if desired. After the complete run, even finer details become visible, like the “M” of the M&M man. Intriguingly, a small part of the back of the elephant is left out. Probably this edge fell exactly between two adjacent radii of the retina and therefore remained invisible. Due to the slightly changing focal length as the consequence of the autofocus, an increasing radial spread of the data points is observed in the confirmation maps (F, H, J). The changing focal length, together with the hyperbolic projection geometry, leads to an enhanced radial displacement of the data points the closer the vehicle gets to a certain object. The actual depth maps (G, I, K) nicely demonstrate the efficacy of the confirmation mechanism, which is part of the algorithm. All (with the exception of very few) of these wrongly detected data points are eliminated because they are observed only once and never confirmed. This also applies to other erroneous data points (e.g., those induced by wandering reflections). The top- and side-view maps give an estimate of the accuracy of the depth values. The different objects are clearly discernible, and even subparts like the two handles of the pliers can be seen. Only the axle support and the scissors close to the wall are confused in the top-view map. The side view, however, shows that these elements are also clearly separated. We determined the center-of-gravity Z-coordinates from histograms similar to
Parallel Noise-Robust Algorithm
399
those shown in Figure 5 (not shown here) for the four major objects in the scene as: pliers, 0.341 m; elephant, 0.558 m; M&M man, 0.892 m; and scissors, 1.191 m. None of these values deviates more then 10 mm from the true position. For the closest object (the pliers), the average relative error is maximal but still less than 2.6%. Top- and side-view maps also demonstrate that the depth-extend (thickness) of the individual objects is correctly retrieved. These results show that the algorithm is applicable under real-world conditions, and the accuracy of the results should almost always suffice. The noise reduction due to the confirmation mechanisms is one major component that ensures this accuracy and robustness. The scene (A–E) and the motion parameters were arranged such that every parameter could be upscaled by a factor of 10 in order to represent the situation encountered (e.g., by big robot in an office or industrial environment). Due to the simplicity of the algorithm, data analysis could still be performed in real time at the given frame rate of 7.5 Hz using a rather slow SUN SPARC 10 workstation. Furthermore, it should be noted that no image preprocessing was performed. We would expect that the quality of the results could be further improved, for example, by applying edge-enhancement algorithms prior to depth analysis. With a more powerful computer, we would also estimate that the frame rate could be at least tree times higher. Given that the first 10 to 100 reliable depth estimates occur within 50 frames, the traveled distance at the higher frame rate would be only 37 mm. Thus, “robot blindness” would be restricted to a very short distance after a turn, even when using a regular serially operating processor. Any parallel implementation would be even faster. Figure 9 shows how the direct analysis of the flow field (Lucas & Kanade, 1981; Barron et al., 1994b) performs on the same real-world example. The top-view map is shown as in Figure 7. Parameters of the algorithm were adjusted such that about the same density of depth estimates was achieved. Only vague outlines of the objects are discernible (circled) but without prior knowledge of the scene (e.g., through an image segmentation algorithm; Opara & Worg ¨ otter, ¨ 1998), no matching between the objects and their depth coordinates is possible. In addition, the accuracy of the depth estimates is very low. The reason for the poor performance is the noise and the small systematic distortion due to the zooming in the images. Direct flow-field analysis is not very robust against these effects as compared to our method, which includes the confirmation mechanism.
3 Discussion 3.1 Advantages and Limitations. The goal of this study was to design a fast parallel-implementable module that performs depth analysis in real
400
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Regular Flow-F ield Analysis (T op View)
Scissors Elephant m&m
Pliers 0
Box edge
120 cm
Figure 9: Results for the same real scene as in Figure 7, obtained by direct flowfield analysis.
time. We were able to show that: 1. All calculations remain local, and data transfer exists only between neighbors (with the exception of the final read-out). Thus, the algorithm can be implemented easily in parallel. 2. The computational complexity in our algorithm is reduced to two scaling operations. 3. The confirmation mechanism reduced noise and other error sources tremendously. 4. The error under different testing conditions remained below ≈ 2% for reasonable parameter settings and remained mostly much smaller (see the appendix). The simplicity of our equations results from the neuronal architecture in combination with the restriction to radial flow. Other algorithms are also substantially reduced in their complexity when considering only such restricted flow fields (see below), but the simplicity of needing only scalar multiplications can be obtained only when considering such a radial, parallel neuronal architecture (compare the “time-to-crash” detector; Ancona & Pog-
Parallel Noise-Robust Algorithm
401
gio, 1993). The virtue of a (possible) parallel implementation together with the simplicity of the approach is still not sufficient to render our approach useful, because local operations are exceedingly sensitive to noise. The additionally introduced novel confirmation mechanism solves this noise problem. This mechanism is also able to eliminate even significant systematic errors, like those introduced by optical distortions (e.g., due to using an autofocus zoom lens; see Figure 7). This argument, together with the outcome of the error analysis, gives a positive answer to the question of whether the algorithm would tolerate error sources in general like mismeasured step counts of the robot motion due to, say, a rugged terrain. The scaling properties of the curves shown in Figure 11 indicate a quite high tolerance of such error sources.5 All this shows that the algorithm is indeed functioning very well under the constraint of a radial flow field. Thus, the crucial question that needs to be answered is if it would be applicable for different motion trajectories that also contain turns. The answer to this question lies in a combined speed and accuracy estimate. The analysis of the systematic errors and of the aliasing behavior (see the appendix) showed that even in a serial implementation on a regular computer, systematic errors and aliasing problems are almost always negligible. Due to the simplicity of the algorithm, we can assume that a slightly faster machine will allow for frame rates of above 30 Hz. This now answers the critical question above: Even if the trajectories remain restricted to motion along only the camera axis, such high frame rates in a serial or parallel implementation allow for the analysis of short motion segments that within a short time window will produce a rather distinct map of the environment. The robot is allowed to turn rather abruptly, which would lead to only a brief reset of the algorithm, and the distance traveled before the novel map emerges after the reset remains very small. Thus, even without a parallel implementation, it should be possible to generate a sufficiently accurate complete depth map by piecing together linear motion segments provided that the robot changes its direction less often than once every 2 seconds or so,6 rendering more than 60 consecutive frames. The comparison of the performance of our restricted algorithmic version with standard flow-field analysis (Lucas & Kanade, 1981; Barron et al., 1994b) provides additional support to the sensibility of tailoring an algorithm to the restricted situation of radial flow fields. Still, the question arises as to whether there is a way to generalize our algorithm to a less restricted situation. This is discussed in the last section. 5 Indeed we observed a quite visible jerkiness of the camera motion when recording the visual scene due to slippage and a somewhat nonradial rotation of the motor axis. This error was also nicely eliminated by the confirmation mechanism, and the residual error remained so low that we found the final results to be rather accurate. 6 A turn every 2 seconds still seems rather unrealistic. Almost all navigating robots in industrial environments turn much less frequently.
402
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Complications can occur as the consequence of object motion that is not immediately detected by our algorithm. Slowly moving objects with a rather homogeneous structure will nevertheless be “seen,” but their recognized shape is smaller than in reality. It is clear that this algorithm was not designed for such situations, which are also hard to resolve for most other algorithms. The algorithm will also fail if the terrain is too rugged. We have already noted that a certain robustness against jitter exists, but due to the limited detection range of the individual neurons, data points will be missed if the camera jitter becomes too strong. 3.2 Comparing the Algorithms. The problem of recovering the 3D structure of a scene from the optical flow has often been discussed in the literature (Longuet-Higgins & Prazdny, 1980; Prazdny, 1980; Heeger & Jepson, 1990; Little & Verri, 1989; Nelson & Aloimonos, 1989). Still, most of the literature deals with the general problem of estimating the self-motion, and the structure of the scene can be estimated from the optical flow. In the special case of pure forward motion with known constant speed that we take in account, the differences between the approaches reported in the literature disappear, and the depth recovery equations become extremely simple. Starting from the perspective projection equation expressed in polar coordinates ρ = f R/Z, the depth of the point is given by the simple relation ˙ Z = −(ρ Z)/(v ρ ), where ρ is the radial coordinate of the projection, vρ is the radial component of the optical flow, and Z˙ is the forward speed. Although these algorithms also become very simple, a reduction to single scalar multiplications can be achieved only by our parallel architecture. In addition, there is a severe problem: Using this equation to recover the structure of the scene relies heavily on the precision of the measured optical flow. Small errors in the flow vectors are amplified by the factor ˙ ρ , in particular around the center of the image, where the flow vectors Z/v are very small (see Figure 9). This problem is unavoidable unless smoothness assumptions about the scene are made, for example, allowing neighboring measurements to merge or a sequence of estimations is integrated using data fusion techniques, like the Kalman filter (see Matthies, Kanade, & Szeliski, 1989). Such algorithmic extensions are far more complex than our simple confirmation mechanism, which solves the noise problem in a satisfactory way. There may also be other ways to account explicitly for the restricted situation of only radial flow, for example, by modifying the “classical” flowfield equations in order to make them numerically more stable around the focus of expansion, but we did not investigate this further. 3.3 Confidence Measurement by Long-Range Couplings. The last section also shows that regardless of which flow-field algorithm is used, it is of utmost importance to include a confidence measure in order to judge the accuracy of the depth estimates (Barron et al., 1994a). In our approach, confidence in the data points is gained by means of the confirmation mechanism.
Parallel Noise-Robust Algorithm
403
The propagation of information along the radii by this predictive mechanism is equivalent to a long-range information exchange in a parallel network. In our case, this can be interpreted as if the detector range (receptive field) of each neuron was enlarged. Thus, currently only the confirmation mechanism ensures the necessary noise reduction, and the improvement of the algorithm is truly dramatic (see Figures 4 and 7).
3.4 Generalizing the Algorithm. A more generalized version of the algorithm could be obtained by allowing for a more extended neuronal coupling, which exceeds the nearest-neighbor interactions currently implemented in the algorithm. Single misses at any neuron would become insignificant in this way (see note 2). An additional extension, which is also interesting from a biological viewpoint, would be to introduce more complexity in the receptive fields of the detectors. Here a natural choice is to use center-surround receptive fields with a significant spatial overlap along and across the radii. In this case, a more elaborate version of the algorithm would be required, having to account for the now-existing lateral inhibition. This leads us to the possibility of more generalized architectures. The central problem behind any generalization is the attempt to reach certain invariances, such as against scaling or rotation, which are common in flow fields. Indeed, there is a biologically motivated way to achieve a higher degree of invariance: Retinal coordinates are projected onto the visual cortex employing (roughly) a complex logarithmic transform (Schwartz, 1977). By means of this, rotational and scaling invariance is obtained because rotations or scaling operations translate into horizontal (medio-lateral) or vertical (anterior-posterior) shifts on the cortical grid, respectively. A combination of both leads to an oblique shift (Schwartz, 1980). In the context of our algorithm, one could now think of a rectangular grid with horizontal, vertical, and (several angles of) oblique connections replacing the design of our retina here. Then, given the results of the current study, it seems likely that a relatively simple set of local equations could be found that operate on such a grid after the input images have been transformed by the complex logarithm. The advantages of a complex logarithmic mapping in the context of classical (nonparallel) flow-field analysis have been demonstrated already (Tistarelli & Sandini, 1993). However, a parallel version does not exist yet. The ultimate version of a parallel algorithm for flow-field analysis would probably make use of such a wire mesh connection pattern and employ spatiotemporal receptive fields (e.g., spatiotemporal Gabor filters) in order to implement one of the well-consolidated phase-based or energy-based flow-field algorithms (Heeger, 1988; Fleet & Jepson, 1990). For this reason we think that the current study is the first step toward a class of parallel algorithms that could become of greater relevance in image analysis as soon as more sophisticated ways for producing parallel VLSI chips exist.
404
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
3.5 A Possible Hardware Implementation. The central advantage that makes our algorithm so simple is that all computations at a given neuron remain restricted to scaling operations. This feature is not dependent on the actual distribution of neurons on the retina. Such scaling operations could in principle be performed even in analog hardware by amplifiers adjusted to the right gain. Memory transfer operations are also rather limited because transfer occurs only in one direction (radially) between pairs of neurons. In a parallel system, two related problems remain: (1) how to set the individual gain values for each neuron and (2) how to retrieve the depth map from the output of the different neurons. Douglas and Mahowald (1995; see also Mahowald & Douglas, 1991) have suggested a hardware for a multiplexing system that allows loading (and retrieving) values into (from) individual neurons in a parallel network. Such a system, or a similar one, could be used for this purpose. Loading would have to be performed only once, and for the retrieval it would have to operate at manageable frequencies of below 50 kHz even for very large networks at high frame rates (e.g., 100 Hz). The layout of the retina should allow for a relatively easy hardware implementation (Pardo, 1994) as compared to other more elaborate parallel flow field algorithms (Bulthoff, ¨ Little, & Poggio, 1989). This could be achieved by a regular grid layout of the photoreceptive sites and an additional address decoder like the so-called Phytagoras processor (GEC Plessey, Semiconductors, PDSP16330/A/B), which converts 16-bit Cartesian coordinates into polar coordinates such that a radial arrangement and also the subsequent computations can be electronically simulated. A related approach, simulating the compound eye of flies, has already been undertaken by Franceschini and colleagues (Franceschini, Pichon & Blanes, 1992; Franceschini, 1996). Appendix: Error Analysis A.1 Systematic Errors. In the following section, we analyze two types of systematic errors that are inherent in the design of the algorithm. We distinguish between the depth error and the aliasing problem. A.1.1 Depth Errors from Mismeasurements and Neuron Placement Inaccuracies. By depth error, we mean any error of the computed Z-component as compared to the true Z-component of an object. Figure 10 shows the actual situation most commonly encountered when an edge (vertical line) excites two neurons subsequently. Due to the radial flow, the edge travels along the ray indicated by the dashed line. The finite resolution of the pixel grid (usually integer resolution), however, restricts the placement of the neurons to the pixel centers, as drawn in the figure. Therefore, the left neuron will be excited too early by the edge and the right neuron too late.7 The actual
7 In other words, this means that as opposed to the optimal situation, both neurons are not excited by exactly the same location on the edge.
Parallel Noise-Robust Algorithm
405
Pixelgrid
α
α
∆ ∆
ε
Figure 10: Geometry underlying the error estimation.
measurement error that results from this effect is given by 1r² . The radial distance between the neurons used to compute the depth estimate is 1r. Thus, the actually traveled distance on the ray is underestimated by 1r² and the true depth estimate would be obtained with 1rtrue = 1r + 1r² . This error depends on the sum of the distances between the ray and the neurons (d) and on the angle between ray and edge (α). It is immediately clear that the 1r² is zero for α = 90 degrees, whereas it becomes infinitely large for α = 0 degrees. Under the assumption that: sn−1,z = sn,z = f , we get for the depth estimate Z: Z=
1Z
sn,r −sn−1,r sn−1,r
= 1Z
sn−1,r . sn,r − sn−1,r
(A.1)
Let sn,r = sn−1,r + 1r. Then, Z = 1Z
rn−1 sn−1,r = 1Z . 1r 1r
(A.2)
The second form of this equation relates to the labeling of the variables used in Figure 12A and will become of relevance later. In addition to the error introduced by the neuron placement (1r² ), we have to take into account that the measured value of 1Z is probably erroneous, for example, due to inaccuracies following a wrong count of the stepper motor between two subsequently excited neurons. Thus, we should assume that the actually g = 1Z + 1Z² , where 1Z is the true value measured value is given by 1Z and 1Z² the measuring error.
406
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Thus, the erroneously estimated depth is given by e = 1Z g rn−1 = (1Z + 1Z² ) rn−1 . Z 1r 1r
(A.3)
On the other hand, the correct estimate would be: Z = 1Z
rn−1 . 1r + 1r²
(A.4)
The absolute error is then: e − Z = (1Z + 1Z² ) Z² = Z
rn−1 rn−1 − 1Z . 1r 1r + 1r²
(A.5)
The relative error is ²=
1r² Z² = Z 1r
µ ¶ 1Z² 1Z² 1+ + . 1Z 1Z
(A.6)
The last equation shows that the relative error critically depends on the ² “relative placement error” 1r 1r , which could in principle reach infinity. The examples from above (see Figures 4, 6, and 7), however, show that this is practically never the case. Nonetheless, in the course of this study, we observed that 1r² can have a tremendously destructive impact on the results as soon as d (see Figure 10) is too large. In a hardware implementation, the neuron grid can be made fine enough in order to reduce d sufficiently, such that this problem is negligible. In our computer implementation, however, we had to find a work-around. Therefore, we resorted to slightly modify the exact geometrical spacing of the neurons on the retina given by equation 2.2 and shifted them a small amount away from the computed locations in order to ensure that the displacement d of two adjacent neurons never exceeds the predefined threshold of ²Displacement (see Tables 1 and 2). If the limit given by ²Displacement could not be achieved by neuron shifting, the neuron was eliminated from the retina. In this way, the initial calculation of the lookup − → table for the neuron-dependent constants Pn became more complicated, but otherwise the accuracy of the results shown in Figures 4 and 6 would have deteriorated. Figure 11 shows how the parameters ²Position and ²Displacement affect the average depth error and the average density of the depth map computed for two objects in the scene shown in Figure 4. For this diagram, the measurements of the triangle at 4.0 m and the vertical bar at 5.5 m in Figure 4 were evaluated for the run with and without noise. In general, the vertical bar (dotted lines) is less susceptible to error than the triangle (solid lines) because of the orientation of its edges. The orientation of an object edge relative to the radii of the retina thereby determines the error susceptibility. If an edge is parallel (orthogonal) to a radius, the error will be high
Parallel Noise-Robust Algorithm
407
Total Number of Depth Estimates
Error per Depth Estimate [%]
without noise 3 2.5
with noise 0.40
A
8
0.40 0.10 0.05
B
0.20
2
6
1.5
0.40 0.20 0.20 0.10 0.10 0.05 0.05
1 0.5 0 0.1 0.2
1
2 34
3500 2500
10
εPosition [cm]
3000
0.20 0.20
1
2 34
0.05
1
2 34
0.10 0.05
εPosition [cm]
εPosition [cm] 0.40
D
0.40 0.20
2000
0.10 0.05
500 0 0.1 0.2
0.10
0 0.1 0.2
0.40
1500
0.40 0.20
2
4000
0.40
C
4
0.10 0.20 0.05
1000 0 0.1 0.2
0.10 0.05
1
2 34
εPosition [cm]
Figure 11: Depth error and density of the depth maps plotted for four different settings of ²Displacement = 0.05, 0.1, 0.2, 0.4 pixel (marked on the curves) against ²Position , which has been varied between 0.1 and 16.0 cm. Results from the triangle (solid lines) and the vertical bar (dotted lines) from Figure 4 are shown with and without noise after the complete simulated robot run. To make them comparable, error values are normalized with respect to the number of total depth estimates obtained. With noise, the algorithm becomes unstable for ²Position < 0.3; thus, these values have been excluded.
(low). Without noise (see Figure 11A) the error increases in small steps with increasing ²Displacement but remains almost the same for different ²Position values. As soon as noise is introduced (B) the situation reverts, and ²Position is the more sensitive parameter. For large values of ²Position , the curves saturate at the maximal number of obtainable depth estimates in the case of no noise (C). Such a saturation is not observed if noise is present; instead, more and more wrong pixels are included in the depth map if ²Position is increased (D). In summary, these diagrams show that if very little noise is expected from the robot’s camera system, large values of ²Position should be used, while ²Displacement is uncritical. On the other hand, if the noise level is high, one should increase the value for ²Displacement in order to get more depth estimates but keep the value of ²Position low in order to reduce the error. In our simulations, we found that a reasonable range that limits the number of totally misplaced points is given by 0.1 < ²Displacement < 0.3 and 0.5 < ²Position < 2.0 for interframe distances of 1 cm. Both parameters scale with the step size.
408
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Figure 12: (A) Geometrical definitions used for the calculation of the systematic errors and the aliasing. (B, C) Systematic errors. Equation A.10 is plotted using (B) the interneuronal distance and (C) the retina location as parameter. Scale bars indicate the values of the fixed parameters.
It should be emphasized that the error introduced by the neuronal placement is a typical grid-aliasing problem and becomes irrelevant by means of the described anti-aliasing procedure or as soon as the grid is fine enough (e.g., in a hardware implementation). For this reason, we will restrict all further analysis to the unavoidable “relative measuring error” 1Z² /1Z introduced by false measurements. Setting 1r² = 0, the equation for the relative error (see equation A.6) is reduced to: ²=
1Z² . 1Z
(A.7)
We assume a geometry as shown in Figure 12A, which is essentially identical to the one used for deriving the basic equations, with the exception of relabeling a few variables for convenience. At the stepper motor counter, the minimal nonzero measuring error is one; that is, to get the minimal relative error, we set 1Z² = 1. For any error bigger than one, Figures 12B and 12C would have to be scaled.8 Then the 8 All following diagrams are metrically scaled. To get this, we have defined a motion constant of 1 mm/Step of the stepper motor, which will not be explicitly mentioned in the following equations.
Parallel Noise-Robust Algorithm
409
relative error is simply: ²min =
1 . 1Z
(A.8)
This is intuitively clear. Since we assume a constant (minimal) measuring error of 1, this mismeasurement will contribute a lot to the relative error if the total measured interval 1Z is small. The question arises, How will the parameters of the retina design will affect the relative error? Using equation A.2, we get: ²min =
rn−1 (rn−1 + 1r) , 1r · f · R
(A.9)
where R is the radial component of the cylinder coordinates of the object in Figure 12A. We set R = 1 m, which means we estimate the error for all objects at that particular lateral distance and get: ²min =
r2n−1 + rn−1 1r 1r · f
1 = f
Ã
r2n−1 1r
! + rn−1
(A.10)
Figures 12B and 12C show the behavior of equation A.10 for a retina with radius ρ = 0.025 m and a focal length of f = 0.025 m. For the sake of completeness, the curves in Figures 12B and 12C extend into meaningless regions (e.g., interneuronal distance of 0.015 m at a total radius of only 0.025 m, etc.). These cases were included to show the shape of the total curves better. The figure and the corresponding equation show that the relative error is strongly affected by the retina position rn−1 , and Z-coordinates computed by neurons in the far periphery of the retina are highly sensitive to measuring errors (see Figure 12B). These curves have been obtained for a minimal measuring error of one step, and the curves need to be multiplicatively upscaled if the error would be larger. From the curves, it can be seen that even for the worst case, the (minimal) relative error is very low. Thus, the system survives significant error upscaling, which also explains the rather high accuracy of the results obtained from the simulations. In addition, the relative error is inversely proportional to the focal length f (held constant in the figure) and to the distance between two neurons 1r. The sensitivity to this parameter (see Figure 12C), however, is much smaller than that to the retinal position (B). At first glance, this inverse relation is quite intriguing, because it means that if the distance between the two adjacent neurons is lowered, then the error increases. In other words, if the neuronal density is increased by raising N, then 1r gets smaller but the error at a given retinal location9 9
For this error estimation we had to fix the retinal location at rn−1 . Thus, it is not
410
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
also increases. The reason for this counterintuitive observation lies in the fact that for decreasing 1r, the measured interval 1Z also decreases, and this increases the relative error. Average neuronal distances of 1 or 2 mm, however, still lead to rather small errors such that total neuron numbers between 50 and 100, as used in the simulations, are very well applicable. A.1.2 Aliasing Problem. As an additional effect, which is of practical relevance, one needs to consider the time between two camera frames. If this time is too long, excitations will skip one or more neurons as soon as these are too densely packed. This problem is of central relevance for a system that implements the algorithm with conventional hardware (i.e., as a serial program on a computer), because the computational effort will limit the number of frames per second significantly. The following discussion is specifically dedicated to such a computer implementation. Thus, this section is rather irrelevant for a parallel processing system with high frame rate where no such aliasing occurs over a huge parameter range. We will compute the shape of the maximal region in the environment that the robot will “see” without aliasing. We will consider the limit case where two subsequent excitations of an edge will fall exactly onto two subsequent neurons, rn−1 and rn . Considering the same geometry as before (see Figure 12A), we have: Z1 R
=
f rn−1
and
Z2 R
=
f rn .
(A.11)
From this and equation 2.4, we get: Z2 =
rn−1 n−1 Z1 . Z1 = rn n+1
(A.12)
Let v be the velocity and µ the camera frame rate. Then: Z1 = Z2 +
v µ
(A.13)
and Z2 =
µ ¶ n−1 v Z2 + . n+1 µ
(A.14)
Solving this for the neuron number, we get: n=
2µ Z2 + 1. v
(A.15)
possible to introduce equation 2.4 here, because in this equation the neuronal positions shift with changing N or ρ.
Parallel Noise-Robust Algorithm
411
If this equation holds for an object distance Z2 , then n is the number of the outermost neuron rn for which no aliasing occurs at a given frame rate and velocity. The neuron number n and the neuron location rn are directly related by equation 2.4. Due to the simple geometry, it is also possible to project the neuron location back into the environment along the ray that connects neuron rn with the nodal point. From equation 2.4, we get: n1,2
1 =− ± 2
r
1 rn + . 4 h
(A.16)
This equation enters in equation A.15. In addition, we substitute rn using equation A.11 and after some arithmetic get: h R= f
Ã
! 4µ2 Z32 6µZ22 + 2Z2 . + v2 v
(A.17)
Given an object with depth Z2 , equation A.17 describes the radial distance R from the optical axis, which is maximally allowed such that no aliasing occurs. In other words, as soon as this object has a radial distance larger than R, its detection is subject to aliasing. The actual depth of the object is the most crucial parameter and enters with a power of three. Objects very nearby are therefore almost always detected with aliasing. From the controllable parameters, frame rate, velocity, and total neuron number (contained in h) are most sensitive; focal length and retina radius (also contained in h) contribute less strongly. Figure 13A shows the regions for which aliasing occurs at different velocities. To obtain these curves, we have assumed a total number of N = 50 neurons, a frame rate of µ = 5 images per second, a retina radius of ρ = 0.025 m, and a focal length of f = 0.025 m. The solid lines reflect a reasonable working range lying inside the “bowl” enclosed by the two solid lines. For this curve, a radial displacement of maximally |R| ≈ 0.8 m is allowed for objects that are 1 m away (Z2 = 1 m) from the robot (crossing points with horizontal line). In the following (see Figures 13B and 13C), we keep ρ = 0.025 m and f = 0.025 m. If we assume that a detection range of R = ±0.5 m is desired for objects at a distance of Z2 = 1 m, then we can solve equation A.17 for the velocity and plot v as a function of the total neuron number (see Figure 13B). The plotted equation reads: v=
N2
p 2µ 6µ + 2 2N2 + 2N + 1 . +N−4 N +N−4
(A.18)
Thus, the diagram shows for different frame rates the maximally allowed
412
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes Object Depth 2Z [m] Velocity =1.0, 0.5,0.2 5 0.1 m/s
A
4
3
2
-100
-1000
-1
-10
-0.1
-0.01
-0.001
-1e-05
-0.0001
-1e-06
1e-06
1e-05
0.0001
0.01
0.001
1
0.1
10
100
1000
1
0.01
0
50
100
150
Total Neuron Number N
200
3
2
01
02
150
0.0
µ=1
100
0.0
1 0.1
µ=5 µ=2
09
µ=10
0.0
10
5
6.7
0.0
100
50
0.1
µ=100 frames/s
0.4
1000
1.
10000
C
25
6
R=100 m
75
B 100000
Total Neuron Number N
Max. Allowed elocity V [m/s]
Object Excentricity R [m]
200
0.1 1 Travelled Distance perrame F L [m]
Figure 13: Aliasing behavior. For A–C we set f = ρ = 0.025 m. (A) The regions in which no aliasing occurs (in between the curves) for different robot velocities and with N = 50, µ = 5. (B) The maximally allowed velocity for which no aliasing occurs at (R, Z) = (± 12 , 1) [m] for different total neuron numbers using the frame rate as parameter. (C) A contour plot showing the allowed radial range R at Z = 1 m for different neuron numbers and different traveled distances per frame.
velocity at a given neuron number for which no aliasing occurs for an object with coordinates R = ±0.5 m, Z = 1 m. If such a system would be designed serially with conventional hardware (a program on a workstation) the computational effort will limit the frame rate, and it is reasonable to assume frame rates of about 10 Hz. In this case, velocities between 0.5 m/s and 1.5 m/s will be obtained for neuron numbers between 25 and 60. If the system would be designed by parallel processing hardware, much higher frame rates could be achieved. The top curve shows that robot velocity is not a limiting factor even for frame rates of 100 Hz for any neuron number below 200. Such a frame rate should be easily obtainable in a parallel processing system. Finally we note that v/µ is identical to the distance L the robot travels
Parallel Noise-Robust Algorithm
413
between two images. Again we set Z2 = 1 m and rewrite equation A.17 as: R=
2 N2 + N
µ
¶ 2 3 + 1 . + L2 L
(A.19)
This allows us to generate a contour plot (see Figure 13C) that shows the iso-radii that limit the range where no aliasing occurs for objects 1 m away as a function of both, N and L. The contour lines between R = 0.46 m and R = 1.75 m reflect a reasonable working range, and the total number of neurons N per radius can be chosen according to the desired speed and frame rate of the system. It should be remembered that the analysis of the aliasing behavior is based on the limit case assumption—the assumption that an edge will—in the limit case— excite exactly two subsequent neurons in two subsequent frames. If the neurons have highly nonoverlapping excitable regions (“receptive fields”), then the projection of a thin edge could also fall between two neurons, exciting neither of them (dubbed an in-between miss). This could lead to a deterioration of the performance even for parameter settings that would be tolerated under the limit case aliasing condition. However, it can be expected that the problem of in-between misses is of minor relevance in any realistic situation, because edges usually are not infinitely thin. If the color on this edge surface is relatively similar over small distances, then neuron rn will detect a different part of the edge as compared to neuron rn−1 , but at least it will not experience an in-between miss. This will lead to a small error in the depth estimate similar to the one introduced by the neuron placement problem discussed above. This error in the Z-component, however, is negligible (see Figures 4 and 6). Acknowledgments We thank R. Opara and B. Porr for their advice at several stages of this project. F. W. acknowledges the support from several grants of the Deutsche Forschungsgemeinschaft WO 388, as well as from the European Community ESPRIT Program (CORMORANT). Furthermore we thank the robotics group at the Computer Science Department of the University of Bonn for letting us use their facilities. Thanks are due to O. Gunt ¨ urk ¨ un, ¨ J. Ostheim, and H. Wagner for providing detailed information and helping us interpret their animal behavior studies. This research was not sponsored by M&Ms, but we enjoyed a fair number of them during programming. References Ancona, N., & Poggio, T. (1993). Optical flow from 1D correlation: Application to a simple time-to-crash detector (A.I. Memo No. 1375). Cambridge, MA: Artificial Intelligence Laboratory, Massachusetts Institute of Technology. Avail-
414
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
able online at: http://www.ai.mit.edu/publications/bibliography/BIBonline.html. Barron, J. L., Beauchemin, S., & Fleet, D. (1994a). On optical flow. Presented at the Sixth International Conference on Artificial Intelligence and Information-Control Systems of Robots, Bratislava, Slovakia, September 12–16 (pp. 3–14). The C source code is available on the FTP server at ftp://csd.uwo.ca/pub/vision. Barron, J. L., Fleet, D., & Beauchemin, S. (1994b). Performance of optical flow techniques. Int. J. Comp. Vis., 12, 43–77. The C source code is available on the FTP server at ftp://csd.uwo.ca/pub/vision. Bulthoff, ¨ H., Little, J., & Poggio, T. (1989). A parallel algorithm for real-time computation of optical flow. Nature, 337, 549–553. Davies, M. N. O., & Green, P. R. (1988). Head-bobbing during walking, running and flying: Relative motion perception in the pigeon. J. Exp. Biol., 138, 71–91. Davies, M. N. O., & Green, P. R. (1990). Optic flow-field variables trigger landing in hawk but not in pigeons. Naturwissenschaften, 77, 142–144. Douglas, R., & Mahowald, M. (1995). Silicon neurons. In M. Arbib (Ed.), The handbook of brain theory and neural networks (pp. 871–875). Cambridge, MA: Bradford Books, MIT Press. Duffy, C. J., & Wurtz, R. H. (1991). Sensitivity of MST neurons to optic flow stimuli. I. A continuum of response selectivity to large-field stimuli. J. Neurophysiol., 65, 1329–1345. Duffy, C. J., & Wurtz, R. H. (1995). Response of monkey MST neurons to optic flow stimuli with shifted centers of motion. J. Neurosci., 7, 5192–5208. Erichsen, J. T., Hodos, W., Evinger, C., Bessette, B. B., & Phillips, S. J. (1989). Head orientation in pigeons: Postural, locomotor and visual determinants. Brain Behav. Evol., 33, 268–278. Fennema, C., & Thompson, W. (1979). Velocity determination in scenes containing several moving objects. Comp. Graph. Image Process., 9, 301–315. Fleet, D., Jepson, A., & Jenkin, M. (1991). Phase-based disparity measurement. Comp. Vision, Graphic and Image Proc., 53, 198–210. Fleet, D., & Jepson, A. (1990). Computation of component image velocity from local phase information. Int. J. Comp. Vis., 5, 77–104. Franceschini, N. (1996). Engineering applications of small brains. FED Journal, 7 (Suppl. 2), 38–52. Franceschini, N., Pichon, J. M., & Blanes, C. (1992). From insect vision to robot vision. Phil. Trans. R. Soc. Lond. B, 337, 283–294. Graziano, M. S., Andersen, R. A., & Snowden, R. J. (1994). Tuning of MST neurons to spiral motions. J. Neurosci., 14, 54–67. Green, P. R., Davies, M. N. O., & Thorpe, P. H. (1992). Head orientation in pigeon during landing flight. Vision Res., 32, 2229–2234. Heeger, D. (1988). Optical flow using spatiotemporal filters. Int. J. Comp. Vis., 1, 279–302. Heeger, D. J., & Jepson, A. D. (1990). Visual perception of three-dimensional motion. Neural Comp., 2, 129–137. Hildreth, E. C., & Koch, C. (1987). The analysis of visual motion: From computational theory to neuronal mechanisms. Annu. Rev. Neurosci., 10, 477–533.
Parallel Noise-Robust Algorithm
415
Horn, B. K. P., & Schunck, B. (1981). Determining optical flow. Artif. Intell., 17, 185–203. Koenderink, J. J. (1986). Optic flow. Vision Res., 26, 161–180. Lappe, M., Bremmer, F., Pekel, M., Thiele, A., & Hoffmann, K. P. (1996). Optic flow processing in monkey STS: A theoretical and experimental approach. J. Neurosci., 16, 6265–6285. Little, J. J., & Verri, A. (1989). Analysis of differential and matching methods for optical flow. IEEE Proc. of Visual Motion Workshop (pp. 173–179). Longuet-Higgins, H. C., & Prazdny, K. (1980). The interpretation of a moving retinal image. Proc. Roy. Soc. Lond., B-208, 385–397. Lucas, B. D., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In DARPA Image Understanding Workshop (pp. 121–130). Mahowald, M., & Douglas, R. (1991). A silicon neuron. Nature, 354, 515–518. Matthies, L., Kanade, T., & Szeliski, R. (1989). Kalman filter–based algorithms for estimating depth from image sequences. Int. J. Comp. Vis., 3, 209–236. Marr, D., & Poggio, T. (1976). Cooperative computation of stereo disparity. Science, 194, 283–287. Nelson, R. C., & Aloimonos, J. (1989). Using flow field divergence for obstacle avoidance in visual navigation. IEEE Trans. PAMI, 11, 1102–1106. Opara, R., & Worg ¨ otter, ¨ F. (1998). A fast and robust cluster update algorithm for image segmentation in spin-lattice models without annealing—Visual latencies revisited. Neural Comp., 10, 1547–1566. Poggio, G. F., & Poggio, T. (1984). The analysis of stereopsis. Annu. Rev. Neurosci., 7, 379. Poggio, T. A., Torre, V., & Koch, C. (1985). Computational vision and regularization theory. Nature, 317, 314–319. Prazdny, K. (1980). Egomotion and relative depth map from optical flow. Biol. Cybern., 36, 87–102. Pardo, F. (1994). Development of a retinal image sensor based on CMOS technology (Tech. Rep.). Genova, Italy: Laboratory for Integrated Advanced Robotic, University of Genova. Available online at: http://afrodite. lira.dist.unige.it:81/LIRA/expsetup/ccd.html and http://afrodite.lira.dist. unige.it:81/LIRA/expsetup/retina.html. Qian, N. (1997). Binocular disparity and the perception of depth. Neuron, 18, 359–368. Sanger, T. D. (1988). Stereo disparity computation using Gabor filters. Biol. Cybern., 59, 405–418. Schwartz, E. L. (1977). Spatial mapping in primate sensory projection: Analytic structure and relevance to perception. Biol. Cybern., 25, 181–194. Schwartz, E. L. (1980). Computational anatomy and functional architecture of striate cortex: A spatial mapping approach to perceptual coding. Vision Res., 20, 645–669. Tistarelli, M., & Sandini, G. (1993). On the advantages of polar and log-polar mapping for direct estimation of time-to-impact from optical flow. IEEE Trans. PAMI, 15, 401–410.
416
F. Worg ¨ otter, ¨ A. Cozzi, and V. Gerdes
Ullman, S. (1979). The interpretation of structure from motion. Proc. R. Soc. London Ser. B, 203, 405–426. Wagner, H. (1986). Flight performance and visual control of flight of the freeflying housefly (Musca domestica l.). I. Organization of the flight motor. Phil. Trans. R. Soc. Lond., B-312, 527–551. Wallman, J., & Letelier, J.-C. (1993). Eye movements, head movements and gaze stabilization in birds. In H. P. Zeigler & H. J. Bischof (Eds.), Vision, brain and behavior in birds. Cambridge, MA: MIT Press. Wang, R. Y. (1996). A network model for the optic flow Computation of the MST neurons. Neural Networks, 9, 411–426. Yuille, A. L., & Ullman, S. (1987). Rigidity and smoothness of motion (A. I. Memo No. 989). Cambridge, MA: Artificial Intelligence Laboratory, Massachusetts Institute of Technology. Available online at: http://www.ai.mit.edu/publications/bibliography/BIB-online.html. Received April 28, 1997; accepted May 14, 1998.
LETTER
Communicated by Jean-Fran¸cois Cardoso
Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources Te-Won Lee Howard Hughes Medical Institute, Computational Neurobiology Laboratory, Salk Institute, La Jolla, CA 92037, U.S.A., and ¨ Electronik, Technische Universit¨at Berlin, Berlin, Germany Institut fur
Mark Girolami Department of Computing and Information Systems, University of Paisley, PA1 2BE, Scotland
Terrence J. Sejnowski Howard Hughes Medical Institute, Computational Neurobiology Laboratory, Salk Institute, La Jolla, CA 92037, U.S.A., and Department of Biology, University of California, San Diego, La Jolla, CA 92093, U.S.A.
An extension of the infomax algorithm of Bell and Sejnowski (1995) is presented that is able blindly to separate mixed signals with sub- and supergaussian source distributions. This was achieved by using a simple type of learning rule first derived by Girolami (1997) by choosing negentropy as a projection pursuit index. Parameterized probability distributions that have sub- and supergaussian regimes were used to derive a general learning rule that preserves the simple architecture proposed by Bell and Sejnowski (1995), is optimized using the natural gradient by Amari (1998), and uses the stability analysis of Cardoso and Laheld (1996) to switch between sub- and supergaussian regimes. We demonstrate that the extended infomax algorithm is able to separate 20 sources with a variety of source distributions easily. Applied to high-dimensional data from electroencephalographic recordings, it is effective at separating artifacts such as eye blinks and line noise from weaker electrical signals that arise from sources in the brain. 1 Introduction Recently, blind source separation by independent component analysis (ICA) has received attention because of its potential signal processing applications, such as speech enhancement systems, telecommunications, and medical signal processing. The goal of ICA is to recover independent sources given only sensor observations that are unknown linear mixtures of the unobserved independent source signals. In contrast to correlation-based transformations Neural Computation 11, 417–441 (1999)
c 1999 Massachusetts Institute of Technology °
418
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
such as principal component analysis (PCA), ICA reduces the statistical dependencies of the signals, attempting to make the signals as independent as possible. The blind source separation problem has been studied by many researchers in neural networks and statistical signal processing (Jutten & H´erault, 1991; Comon, 1994; Cichocki, Unbehauen, & Rummert, 1994; Bell & Sejnowski, 1995; Cardoso & Laheld, 1996; Amari, Cichocki, & Yang, 1996; Pearlmutter & Parra, 1996; Deco & Obradovic, 1996; Oja, 1997; Karhunen, Oja, Wang, Vigario, & Joutsensalo, 1997; Girolami & Fyfe, 1997a). See the introduction of Nadal and Parga (1997) for a historical review of ICA, and Karhunen (1996) for a review of different neural-based blind source separation algorithms. More general ICA reviews are in Cardoso (1998), Lee (1998), & Lee, Girolami, Bell, & Sejnowski (1999). Bell and Sejnowski (1995) have developed an unsupervised learning algorithm based on entropy maximization in a single-layer feedforward neural network. The algorithm is effective in separating sources that have supergaussian distributions: sharply peaked probability density functions (p.d.f.s) with heavy tails. As illustrated in section 4 of Bell and Sejnowski (1995), the algorithm fails to separate sources that have negative kurtosis (e.g., uniform distribution). Pearlmutter and Parra (1996) have developed a contextual ICA algorithm within the maximum likelihood estimation (MLE) framework that is able to separate a more general range of source distributions. Motivated by computational simplicity, we use an information-theoretic algorithm that preserves the simple architecture in Bell and Sejnowski (1995) and allows an extension to the separation of mixtures of supergaussian and subgaussian sources. Girolami (1997) derived this type of learning rule from the viewpoint of negentropy maximization1 for exploratory projection pursuit (EPP) and ICA. These algorithms can be used on-line as well as off-line. Off-line algorithms that can also separate mixtures of supergaussian and subgaussian sources were proposed by Cardoso and Soloumiac (1993), Comon (1994), and Pham and Garrat (1997). The extended infomax algorithm preserves the simple architecture in Bell and Sejnowski (1995) and the learning rule converges rapidly with the “natural” gradient proposed by Amari et al. (1996) and Amari (1998) or the “relative” gradient proposed by Cardoso and Laheld (1996). In computer simulations, we show that this algorithm can successfully separate 20 mixtures of the following sources: 10 soundtracks2 , 6 speech and sound signals used in Bell and Sejnowski (1995), 3 uniformly distributed subgaussian noise signals, and 1 noise source with a gaussian distribution. To test the extended infomax algorithm on more challenging real-world data, we
1
Relative entropy is the general term for negentropy. Negentropy maximization refers to maximizing the sum of marginal negentropies. 2 Obtained from Pearlmutter online at http://sweat.cs.unm.edu/∼bap/demos.html.
An Extended Infomax Algorithm
419
performed experiments with electroencephalogram (EEG) recordings and show that it can clearly separate electrical artifacts from brain activity. This technique shows great promise for analyzing EEG recordings (Makeig, Jung, Bell, Ghahremani, & Sejnowski, 1997; Jung et al., 1998) and functional magnetic resonance imaging (fMRI) data (McKeown et al., 1998). In section 2, the problem is stated and a simple but general learning rule that can separate sub- and supergaussian sources is presented. This rule is applied to simulations and real data in section 3. Section 4 contains a brief discussion of other algorithms and architectures, potential applications to real-world problems, limitations, and further research problems. 2 The Extended Infomax Algorithm Assume that there is an M-dimensional zero-mean vector s(t) = [s1 (t), . . . , sM (t)]T , such that the components si (t) are mutually independent. The vector s(t) corresponds to M independent scalar-valued source signals si (t). We can write the multivariate p.d.f. of the vector as the product of marginal independent distributions:
p(s) =
M Y
pi (si ).
(2.1)
i=1
A data vector x(t) = [x1 (t), . . . , xN (t)]T is observed at each time point t, such that x(t) = As(t),
(2.2)
where A is a full-rank N × M scalar matrix. Because the components of the observed vectors are no longer independent, the multivariate p.d.f. will not satisfy the p.d.f. product equality. In this article, we shall consider the case where the number of sources is equal to the number of sensors N = M. If the components of s(t) are such that at most one source is normally distributed, then it is possible to extract the sources s(t) from the received mixtures x(t) (Comon, 1994). The mutual information of the observed vector is given by the Kullback-Leibler (KL) divergence of the multivariate density from the product of the marginal (univariate) densities: Z I(x1 , x2 , . . . , xN ) =
+∞ Z +∞
−∞
−∞
× log
Z ···
+∞
−∞
p(x1 , x2 , . . . , xN )
p(x1 , x2 , . . . , xN ) dx1 dx2 , . . . , dxN . QN i=1 pi (xi )
(2.3)
420
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
For simplicity, we write: Z p(x) dx. I(x) = p(x) log QN i=1 pi (xi )
(2.4)
The mutual information will always be positive and will equal zero only when the components are independent (Cover & Thomas, 1991). The goal of ICA is to find a linear mapping W such that the unmixed signals u, u(t) = Wx(t) = WAs(t),
(2.5)
are statistically independent. The sources are recovered up to scaling and permutation. There are many ways for learning W. Comon (1994) minimizes the degree of dependence among outputs using contrast functions approximated by the Edgeworth expansion of the KL divergence. The higher-order statistics are approximated by cumulants up to fourth order. Other methods related to minimizing mutual information can be derived from the infomax approach. Nadal and Parga (1994) showed that in the low-noise case, the maximum of the mutual information between the input and output of a neural processor implied that the output distribution was factorial. Roth and Baram (1996) and Bell and Sejnowski (1995) independently derived stochastic gradient learning rules for this maximization and applied them, respectively, to forecasting, time-series analysis, and the blind separation of sources. A similar adaptive method for source separation has been proposed by Cardoso and Laheld (1996). 2.1 A Simple But General Learning Rule. The learning algorithm can be derived using the maximum likelihood formulation. The MLE approach to blind source separation was first proposed by Gaeta and Lacoume (1990) and Pham and Garrat (1997) and was pursued more recently by Pearlmutter and Parra (1996) and Cardoso (1997). The p.d.f. of the observations x can be expressed as (Amari & Cardoso, 1997): p(x) = | det(W)|p(u)
(2.6)
Q where p(u) = N i=1 pi (ui ) is the hypothesized distribution of p(s). The loglikelihood of equation 2.6 is L(u, W) = log | det(W)| +
N X
log pi (ui ).
(2.7)
i=1
Maximizing the log-likelihood with respect to W gives a learning algorithm for W (Bell & Sejnowski, 1995): i h (2.8) 1W ∝ (WT )−1 − ϕ(u)xT ,
An Extended Infomax Algorithm
421
where ϕ(u) = −
∂p(u) ∂u
p(u)
" = −
∂p(u1 ) ∂u1
p(u1 )
,...,−
∂p(uN ) ∂uN
#T
p(uN )
.
(2.9)
An efficient way to maximize the log-likelihood is to follow the “natural” gradient (Amari, 1998), 1W ∝
h i ∂L(u, W) T W W = I − ϕ(u)uT W, ∂W
(2.10)
as proposed by Amari et al. (1996), or the relative gradient, proposed by Cardoso and Laheld (1996). Here WT W rescales the gradient, simplifies the learning rule in equation 2.8, and speeds convergence considerably. It has been shown that the general learning algorithm in equation 2.10 can be derived from several theoretical viewpoints such as MLE (Pearlmutter & Parra, 1996), infomax (Bell & Sejnowski, 1995), and negentropy maximization (Girolami & Fyfe, 1997b). Lee, Girolami, Bell, & Sejnowski, in press, review these techniques and show their relation to each other. The parametric density estimate pi (ui ) plays an essential role in the success of the learning rule in equation 2.10. Local convergence is ensured if pi (ui ) is the derivative of the log densities of the sources (Pham & Garrat, 1997). If we choose gi (u) to be a logistic function (gi (ui ) = tanh(ui )) so that ϕ(u) = 2 tanh(u), the learning rule reduces to that in Bell and Sejnowski (1995) with the natural gradient: i h 1W ∝ I − 2 tanh(u)uT W.
(2.11)
Theoretical considerations as well as empirical observations3 have shown that this algorithm is limited to separating sources with supergaussian distributions. The sigmoid function used in Bell and Sejnowski (1995) provides a priori knowledge about the source distribution, that is, the supergaussian shape of the sources. However, they also discuss a “flexible” sig(a sigmoid function with parameters p, r so that g(ui ) = Rmoid function g(ui )p (1 − g(ui ))r ) can be used to match the source distribution. The idea of modeling a parametric nonlinearity has been further investigated and generalized by Pearlmutter and Parra (1996) in their contextual ICA (cICA) algorithm. They model the p.d.f. in a parametric form by taking into account the temporal information and choosing pi (ui ) as a weighted sum of several logistic density functions with variable means and scales. Moulines, Cardoso, and Cassiat (1997) and Xu, Cheung, Yang, and Amari (1997) model the underlying p.d.f. with mixtures of gaussians and show that they can 3
As detailed in section 4 of Bell and Sejnowski (1995).
422
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
separate sub- and supergaussian sources. These parametric modeling approaches are in general computationally expensive. In addition, our empirical results on EEG and event-related potentials (ERP) using contextual ICA indicate that cICA can fail to find independent components. Our conjecture is that this is due to the limited number of recorded time points (e.g., 600 data points for ERPs) from which a reliable density estimate is difficult. 2.2 Deriving a Learning Rule to Separate Sub- and Supergaussian Sources. The purpose of the extended infomax algorithm is to provide a simple learning rule with a fixed nonlinearity that can separate sources with a variety of distributions. One way of generalizing the learning rule to sources with either sub- or supergaussian distributions is to approximate the estimated p.d.f. with an Edgeworth expansion or Gram-Charlier expansion (Stuart & Ord, 1987), as proposed by Girolami and Fyfe (1997b). Girolami (1997) used a parametric density estimate to derive the same learning rule without making any approximations, as we show below. A symmetric strictly subgaussian density can be modeled using a symmetrical form of the Pearson mixture model (Pearson, 1894) as follows (Girolami, 1997, 1998): p(u) =
´ 1³ N(µ, σ 2 ) + N(−µ, σ 2 ) , 2
(2.12)
where N(µ, σ 2 ) is the normal density with mean µ and variance σ 2 . Figure 1 shows the form of the density p(u) for σ 2 = 1 with varying µ = [0 · · · 2]. For µ = 0 p(u) is a gaussian model whereas for µi = 1.5, for example, the p(u) is clearly bimodal. The kurtosis k4 (normalized fourth-order cumulant) of p(u) is κ=
−2µ4 c4 = , (µ2 + σ 2 )2 c22
(2.13)
where ci is the ith-order cumulant (Girolami, 1997). Depending on the values of µ and σ 2 , the kurtosis lies between −2 and 0. So equation 2.12 defines a strictly subgaussian symmetric density when µ > 0. Defining a = σµ2 and applying equation 2.12, we may write for ϕ(u) ¶ µ u exp(au) − exp(−au) = 2 −a . ϕ(u) = − p(u) σ exp(au) + exp(−au) ∂p(u) ∂u
(2.14)
Using the definition of the hyperbolic tangent, we can write ϕ(u) =
³µ ´ µ u − 2 tanh u . 2 σ σ σ2
(2.15)
An Extended Infomax Algorithm
423
Setting µ = 1 and σ 2 = 1, equation 2.15 reduces to ϕ(u) = u − tanh(u).
(2.16)
The learning rule for strictly subgaussian sources is now (equations 2.10 and 2.16) i h (2.17) 1W ∝ I + tanh(u)uT − uuT W. In the case of unimodal supergaussian sources, we adopt the following density model p(u) ∝ pG (u)sech2 (u),
(2.18)
where pG (u) = N(0, 1) is a zero-mean gaussian density with unit variance. Figure 2 shows the density model for p(u). The nonlinearity ϕ(u) is now ϕ(u) = −
∂p(u) ∂u
p(u)
= u + tanh(u).
(2.19)
The learning rule for supergaussian sources is (equations 2.10 and 2.19): i h 1W ∝ I − tanh(u)uT − uuT W.
(2.20)
The difference between the supergaussian learning rule in equation 2.20 and the subgaussian learning rule equation 2.17 is the sign before the tanh function: ¤ ½ £ I − tanh(u)uT − uuT ¤ W : supergaussian £ (2.21) 1W ∝ I + tanh(u)uT − uuT W : subgaussian The learning rules differ in the sign before the tanh function and can be determined using a switching criterion. Girolami (1997) employs the sign of kurtosis of the unmixed sources as a switching criterion. However, because there is no general definition for sub- and supergaussian sources, we chose a switching criterion, based on stability criteria, presented in the next subsection. 2.3 Switching Between Nonlinearities. The switching between the suband supergaussian learning rule is i h 1W ∝ I − K tanh(u)uT − uuT ½ ki = 1 : supergaussian ×W ki = −1 : subgaussian
(2.22)
424
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
Figure 1: Estimated subgaussian density models for the extended infomax learning rule with σ 2 = 1 and µi = {0 · · · 2}. The density becomes clearly bimodal when µi > 1.
Figure 2: Density model for the supergaussian distribution. The supergaussian model has a heavier tail than the normal density.
An Extended Infomax Algorithm
425
where ki are elements of the N-dimensional diagonal matrix K. The switching parameter ki can be derived from the generic stability analysis of separating solutions as employed by Cardoso and Laheld (1996)4 , Pham and Garrat (1997), and Amari et al. (1997). In the stability analysis, the mean field is approximated by a first-order perturbation in the parameters of the separating matrix. The linear approximation near the stationary point is the gradient of the mean field at the stationary point. The real part of the eigenvalues of the derivative of the mean field must be negative so that the parameters are on average pulled back to the stationary point. A sufficient condition guaranteeing asymptotic stability can be derived (Cardoso, 1998, in press) so that κi > 0 1 ≤ i ≤ N,
(2.23)
where κi is κi = E{ϕi0 (ui )}E{u2i } − E{ϕi (ui )ui }
(2.24)
ϕi (ui ) = ui + ki tanh(ui ).
(2.25)
and
Substituting equation 2.25 in equation 2.24 gives κi = E{ki sech2 (ui ) + 1}E{u2i } − E{[ki tanh(ui ) + ui ]ui } ´ ³ = ki E{sech2 (ui )}E{u2i } − E{[tanh(ui )]ui } .
(2.26) (2.27)
To ensure κi > 0 the sign of ki must be the same as the sign of E{sech2 (ui )}E{u2i } − E{[tanh(ui )]ui }. Therefore we can use the learning rule in equation 2.22, where the ki ’s are ³ ´ ki = sign E{sech2 (ui )}E{u2i } − E{[tanh(ui )]ui } .
(2.28)
2.4 The Hyperbolic-Cauchy Density Model. We present another parametric density model that may be used for the separation of sub- and supergaussian sources. We define the parametric mixture density as p(u) ∝ sech2 (u + b) + sech2 (u − b).
(2.29)
Figure 3 shows the parametric density as a function of b. For b = 0, the parametric density is proportional to the hyperbolic-Cauchy distribution 4
See equations 40 and 41 in their paper.
426
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
and is therefore suited for separating supergaussian sources. For b = 2 the parametric density estimator has a bimodal5 distribution with negative kurtosis and is therefore suitable for separating subgaussian sources: ϕ(u) = −
∂ log p(u) ∂u
= −2 tanh(u) + 2 tanh(u + b) + 2 tanh(u − b).
(2.30)
The learning algorithm for sub- and supergaussian sources is now (equations 2.30 and 2.10) i h 1W ∝ I+2 tanh(u)uT −2 tanh(u + b)uT −2 tanh(u − b)uT W. (2.31) When b = 0 (where 0 is an N-dimension vector with elements 0), then the learning rule reduces to i h 1W ∝ I − 2 tanh(u)uT W,
(2.32)
which is exactly the learning rule in Bell and Sejnowski (1995) with the natural gradient extension. For b > 1, the parametric density is bimodal (as shown in Figure 3), and the learning rule is suitable for separating signals with subgaussian distributions. Here again we may use the sign of the general stability criteria in equation 2.23 and κi in equation 2.24 to determine bi so that we can switch between bi = 0 and, for example, bi = 2. In Figure 4 we compare the range of kurtosis values of the parametric mixture density models in equations 2.12 and 2.29. The kurtosis value is shown as a function of the shaping parameter µ for the symmetric Pearson density model and b for the hyperbolic-Cauchy mixture density model. The kurtosis for the Pearson model is strictly negative except for µ = 0 when the kurtosis is zero. Because the kurtosis for the hyperbolic-Cauchy model ranges from positive to negative, it may be used to separate signals with both sub- and supergaussian densities. 3 Simulations and Experimental Results Extensive simulations and experiments were performed on recorded data to verify the performance of the extended infomax algorithm equation 2.21. First, we show that the algorithm is able to separate a large number of sources with a wide variety of sub- and supergaussian distributions. We compared the performance of the extended infomax learning rule in equa5 Symmetric bimodal densities considered in this article are subgaussian; however, this is not always the case.
An Extended Infomax Algorithm
427
Figure 3: p(u) as a function of b. For b = 0 the density estimate is suited to separate supergaussian sources. If, for example, b = 2 the density estimate is bimodal and therefore suited to separate subgaussian sources.
tion 2.10 to the original infomax learning rule equation 2.11. Second, we performed a set of experiments on EEG data, which are high dimensional and include various noise sources. 3.1 Ten Mixed Sound Sources. We obtained 10 mixed sound sources that were separated by contextual ICA as described in Pearlmutter and Parra (1996). No prewhitening is required since the transformation W is not restricted to a rotation, in contrast to nonlinear PCA (Karhunen et al., 1997). All 55,000 data points were passed 20 times through the learning rule using a block size (batch) of 300. This corresponds to 3666 iterations (weight updates). The learning rate was fixed at 0.0005. Figure 5 shows the error measure during learning. Both learning rules converged. The small variations of the extended infomax algorithm (upper curve) were due to the adaptation process of K. The matrix K was initialized to the identity matrix, and during the learning process the elements of K converge to −1 or 1 to extract sub- or supergaussian sources, respectively. In this simulation example, sources 7, 8, and 9 are close to gaussian, and slight variations of their density estimation change the sign. Annealing of the learning rate reduced the variation. All the music signals had supergaussian distribution and therefore were separable by the original infomax algorithm. The sources are already well separated after one pass through the data (about 10 sec on a SPARC 10 workstation using MATLAB) as shown in Table 1.
428
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
Figure 4: The kurtosis value is shown as a function of the shaping parameter µ and b (µ for the Pearson density model and b for the hyperbolic-Cauchy density model). Both models approach k4 = −2 as the shaping parameter increases. The kurtosis for the Pearson model is strictly negative except for µ = 0. The kurtosis for the hyperbolic-Cauchy model ranges from positive to negative so that we may use this single parametric model to separate signals with sub- and supergaussian densities.
Table 1: Performance Matrix P (Equation 3.2) for 10 Mixed Sound Sources after One Pass through the Data. −0.09 11.18 0.15 0.39 0.04 0.11 0.45 0.31 −0.54 −0.08
−0.38 −0.01 0.078 0.61 0.76 12.89 0.16 0.14 −0.81 −0.26
0.14 0.14 −0.08 −0.70 14.89 −0.54 −0.02 0.23 0.62 0.15
−0.10 0.05 −0.02 −0.07 0.03 −0.23 6.53 0.03 0.84 −0.10
−0.06 −0.08 10.19 0.14 0.03 −0.43 0.24 −0.14 −0.18 0.49
0.93 0.02 −0.02 0.32 −0.17 −0.21 0.98 −17.25 0.47 0.01
−0.36 0.07 0.15 −0.08 0.18 −0.12 −0.39 −0.39 −0.04 −10.25
−0.54 0.21 0.05 0.85 −0.31 0.05 −0.97 −0.25 10.48 0.59
0.17 −0.12 0.07 7.64 −0.19 0.07 0.06 0.19 −0.92 0.33
14.79 −0.68 0.17 −0.16 0.04 0.18 −0.08 0.39 0.12 −0.94
Note: After one pass through the data P are already close to the identity matrix after rescaling and reordering. An italicized entry signifies the largest component for the corresponding channel.
An Extended Infomax Algorithm
429
Figure 5: Error measure E in equation 3.2 for the separation of 10 sound sources. The upper curve is the performance for extended infomax, and the lower curve shows the performance for the original infomax. The separation quality is shown in Table 1.
For all experiments and simulations, a momentum term helped to accelerate the convergence of the algorithm: 1W(n + 1) = (1 − α)1W(n) + αW(n),
(3.1)
where α takes into account the history of W and α can be increased with an increasing number of weight updates (as n → ∞, α → 1). The performance during the learning process we monitored by the error measure proposed by Amari et al. (1996), ! Ã N N N N X X X X |pij | |pij | − 1 + −1 , (3.2) E= maxk |pik | maxk |pkj | i=1 i=1 j=1 j=1 where pij are elements of the performance matrix P = WA. P is close to a permutation of the scaled identity matrix when the sources are separated. Figure 5 shows the error measure during the learning process. To compare the speed of the extended infomax algorithm with another closely related one, we separated the 10 mixed sound sources using the extended exploratory projection pursuit network with inhibitory lateral connections (Girolami & Fyfe, 1997a). The single feedforward neural network
430
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
converged several times faster than this architecture using the same learning rate and a block size of 1. Larger block sizes can be used in the feedforward network but not the feedback networks, which increase the convergence speed considerably due to a more reliable estimate of the switching matrix K. 3.2 Twenty Mixed Sound Sources. We separated the following 20 sources: 10 soundtracks obtained from Pearlmutter, 6 speech and sound signals used in Bell and Sejnowski (1995), 3 uniformly distributed subgaussian noise signals, and 1 noise source with a gaussian distribution. The densities of the mixtures were close to the gaussian distributions. The following parameters were used: learning rate fixed at 0.0005, block size of 100 data points, and 150 passes through the data (41,250 iterations). Table 2 compares the separation quality between the infomax algorithm and the extended infomax algorithm. Figure 6 shows the performance of the matrix P after the rows were manually reordered and normalized to unity. P is close to the identity matrix, and its off-diagonal elements indicate the amount of error. In this simulation, we employ k4 as a measure of the recovery of the sources. The original infomax algorithm separated most of the positive kurtotic sources. However, it failed to extract several sources, including two supergaussian sources (music 7 and 8) with low kurtosis (0.78 and 0.46, respectively). In contrast, Figure 7 shows that the performance matrix P for the extended infomax algorithm is close to the identity matrix. In a listening test, there was a clear separation of all sources from their mixtures. Note that although the sources ranged from Laplacian distributions (p(s) ∝ exp(−|s|), e.g., speech), and gaussian noise to uniformly distributed noise, they were all separated using one nonlinearity. The simulation results suggest that the supergaussian and subgaussian density estimates in equations 2.12 and 2.18 are sufficient to separate the true sources. The learning algorithms in equations 2.21 and 2.31 performed almost identically. 3.3 EEG Recordings. EEG recordings of brain electrical activity from the human scalp, artifacts such as line noise, eye movements, blinks, and cardiac signals (EKG) pose serious problems in analyzing and interpreting the recordings. Regression methods have been used to remove eye movement partially from the EEG data (Berg & Scherg, 1991); other artifacts such as electrode noise, cardiac signals, and muscle noise are even more difficult to remove. Recently, Makeig, Bell, Jung, & Sejnowski (1996) applied ICA to the analysis of EEG data using the original infomax algorithm. They showed that some artifactual components can be isolated from overlapping EEG signals, including alpha and theta bursts. We analyzed EEG data that were collected to develop a method of objectively monitoring the alertness of operators listening for auditory signals
Music 1 Music 2 Music 3 Music 4 Music 5 Music 6 Music 7 Music 8 Music 9 Music 10 Speech 1 Speech 2 Music 11 Speech 3 Music 12 Speech 4 Uniform noise 1 Uniform noise 2 Uniform noise 3 Gaussian noise
Source Type 2.4733 1.5135 2.4176 1.076 1.0317 1.8626 0.7867 0.4639 0.5714 2.6358 6.6645 3.3355 1.1082 7.2846 2.8308 10.8838 −1.1959 −1.2031 −1.1966 −0.0148
Original Kurtosis 2.4754 1.5129 2.4206 1.0720 1.0347 1.8653 0.8029 0.2753 0.5874 2.6327 6.6652 3.3389 1.1072 7.2828 2.8198 10.8738 −0.2172 −0.2080 −0.2016 −0.0964
Recovered Kurtosis (infomax) 2.4759 1.5052 2.4044 1.0840 1.0488 1.8467 0.7871 0.4591 0.5733 2.6343 6.6663 3.3324 1.1053 7.2875 2.8217 10.8128 −1.1955 −1.2013 −1.1955 −0.0399
Recovered Kurtosis (extended infomax) 43.4 55.2 44.1 31.7 43.6 48.1 32.7 29.4 36.4 46.4 54.3 50.5 48.1 50.5 52.6 57.1 61.4 67.7 63.6 24.9
Signal-to-Noise Ratio (SNR) (extended infomax)
Note: The source signals range from highly kurtotic speech signals, gaussian noise (kurtosis is zero) to noise sources with uniform distribution (negative kurtosis). The sources that failed to separate clearly are italicized. In addition, the SNR is computed for extended infomax.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Source Number
Table 2: Kurtosis of the Original Signal Sources and Recovered Signals.
An Extended Infomax Algorithm 431
432
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
Figure 6: Performance matrix P for the separation of 20 sources using the original infomax algorithm after normalizing and reordering. Most supergaussian sources were recovered. However, the three subgaussian sources (17, 18, 19), the gaussian source (20), and two supergaussian sources (7, 8) remain mixed and aliased in other sources. In total, 14 sources were extracted, and 6 channels remained mixed. See Table 2.
Figure 7: Performance matrix P for the separation of 20 sources using the extended infomax algorithm after normalizing and reordering. P is approximately the identity matrix that indicates nearly perfect separation.
An Extended Infomax Algorithm
433
(Makeig & Inlow, 1993). During a half-hour session, the subject was asked to push a button whenever he or she detected an auditory target stimulus. EEG was collected from 14 electrodes located at sites of the International 1020 System (Makeig et al., 1997) at a sampling rate of 312.5 Hz. The extended infomax algorithm was applied to the 14 channels of 10 seconds of data with the following parameters: learning rate fixed at 0.0005, 100 passes with block size of 100 (3125 weight updates). The power spectrum was computed for each channel, and the power in a band around 60 Hz was used to compute the relative power for each channel and each separated component. Figure 8 shows the time course of 14 channels of EEG and Figure 9 the independent components found by the extended infomax algorithm. Several observations on the ICA components in Figure 9 and its power spectrum are of interest: • Alpha bursts (about 11 Hz) were detected in components 1 and 5. Alpha band activity (8–12 Hz) occurs most often when the eyes are closed and the subject is relaxed. Most subjects have more than one alpha rhythm, with somewhat different frequencies and scalp patterns. • Theta bursts (about 7 Hz) were detected in components 4, 6, and 9. Theta-band rhythms (4–8 Hz) may occur during drowsiness and transient losses of awareness or microsleeps (Makeig & Inlow, 1993), but frontal theta bursts may occur during intense concentration. • An eye blink was isolated in component 2 at 8 sec. • Line noise of 60 Hz was concentrated in component 3 (see the bottom of Figure 10). Figure 10 (top) shows power near 60 Hz distributed in all EEG channels but predominantly in components 4, 13, and 14. Figure 10 (middle) shows that the original infomax cannot concentrate the line noise into one component. In contrast, extended infomax (figure 10, bottom panel) concentrates it mainly in one subgaussian component, channel 3. Figure 11 shows another EEG data set with 23 channels, including 2 EOG (electrooculogram) channels. The eye blinks near 5 sec and 7 sec contaminated all of the channels. Figure 12 shows the ICA components without normalizing the components with respect to their contribution to the raw data. ICA component 1 in Figure 12 contained the pure eye blink signal. Small periodic muscle spiking at the temporal sites (T3 and T4) was extracted into ICA component 14. Experiments with several different EEG data sets confirmed that the separation of artifactual signals was highly reliable. In particular, severe line noise signals could always be decomposed into one or two components with subgaussian distributions. Jung et al. (1998) show further that eye movement also can be extracted.
434
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
Figure 8: A 10 sec portion of the EEG time series with prominent alpha rhythms (8–21 Hz). The location of the recording electrode from the scalp is indicated on the left of each trace. The electrooculogram (EOG) recording is taken from the temples.
4 Discussion 4.1 Applications to Real-World Problems. The results reported here for the separation of eye movement artifacts from EEG recordings have immediate application to medical and research data. Independently, Vigario, Hyvaerinen, and Oja (1996) reported similar findings for EEG recordings using a fixed-point algorithm for ICA (Hyvaerinen & Oja, 1997). It would be useful to compare this and other ICA algorithms on the same data sets to assess their merits. Compared to traditional techniques in EEG analysis, extended infomax requires less supervision and is easy to apply (see Makeig et al., 1997; Jung et al., 1998). In addition to the very encouraging results on EEG data given here, McKeown et al. (1998) have demonstrated another successful use of the extended infomax algorithm on fMRI recordings. They investigated task-related human brain activity in fMRI data. In this application, they considered both spatial and temporal ICA and found that the extended infomax algorithm extracted subgaussian temporal components that could not be extracted with the original infomax algorithm.
An Extended Infomax Algorithm
435
Figure 9: The 14 ICA components extracted from the EEG data in Figure 8. Components 3, 4, 7, 8, and 10 have subgaussian distributions, and the others have supergaussian distributions. There is an eye movement artifact at 8 seconds. Line noise is concentrated in component 3. The prominent rhythms in components 1, 4, 5, 6, and 9 have different time courses and scalp distributions.
4.2 Limitations and Future Research. The extended infomax learning algorithm makes several assumptions that limit its effectiveness. First, the algorithm requires the number of sensors to be the same as or greater than the number of sources (N ≥ M). The case when there are more sources than sensors, N < M, is of theoretical and practical interest. Given only one or two sensors that observe more than two sources, can we still recover all sources? Preliminary results by Lewicki and Sejnowski (1998) suggest that an overcomplete representation of the data to some extent can extract the independent components using a priori knowledge of the source distribution. This has been applied by Lee, Lewicki, Girolami, and Sejnowski (in press b) to separate three sources from two sensors. Second, researchers have recently tackled the problem of nonlinear mixing phenomena. Yang, Amari, and Cichocki (1997), Taleb and Jutten (1997), and Lee, Koehler, and Orglmeister (1997) propose extensions when linear mixing is combined with certain nonlinear mixing models. Other approaches use self-organizing feature maps to identify nonlinear features in the data (Lin & Cowan, 1997; Pajunen & Karhunen, 1997). Hochreiter and
436
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
Figure 10: (Top) Ratio of power near 60 Hz over 14 components for EEG data in Figure 8. (Middle) Ratio of power near 60 Hz for the 14 infomax ICA components. (Bottom) Ratio of power near 60 Hz for the 14 extended infomax ICA components in Figure 9. Note the difference in scale by a factor of 10 between the original infomax and the extended infomax.
Schmidhuber (1999) have proposed low-complexity coding and decoding approaches for nonlinear ICA. Third, sources may not be stationary; sources may appear and disappear and move (as when a speaker moves in a room). In these cases, the weight matrix W may change completely from one time point to the next. This is a challenging problem for all existing ICA algorithms. A method to model the context switching (nonstationary mixing matrix) in an unsupervised way is proposed in Lee, Lewicki, and Sejnowski (1999). Fourth, sensor noise may influence separation and should be included in the model (Nadal & Parga 1994; Moulines et al., 1997; Attias & Schreiner, 1999). Much more work needs to be done to determine the effect of noise on performance. In addition to these limitations, there are other issues that deserve further research. In particular, it remains an open question to what extent the learning rule is robust to parametric mismatch given a limited number of data points. Despite these limitations, the extended infomax ICA algorithm presented here should have many applications where both subgaussian and super-
An Extended Infomax Algorithm
437
Figure 11: EEG data set with 23 channels including 2 EOG channels. Note that at around 4–5 sec and 6–7 sec, artifacts from severe eye blinks contaminate the data set.
gaussian sources need to be separated without additional prior knowledge of their statistical properties. 5 Conclusions The extended infomax ICA algorithm proposed here is a promising generalization that satisfies a general stability criterion for mixed subgaussian and supergaussian sources (Cardoso & Laheld, 1996). Based on the learning algorithm first derived by Girolami (1997) and the natural gradient, the extended infomax algorithm has shown excellent performance on several large real data sets derived from electrical and blood flow measurements of functional activity in the brain. Compared to the originally proposed infomax algorithm (Bell and Sejnowski, 1995), the extended infomax algorithm separates a wider range of source signals while maintaining its simplicity. Acknowledgments T. W. L. was supported by the German Academic Exchange Program. M. G. was supported by a grant from NCR Financial Systems (Ltd.), Knowledge Laboratory, Advanced Technology Development Division, Dundee, Scotland. T. J. S. was supported by the Howard Hughes Medical Institute. We
438
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
Figure 12: Extended infomax ICA components derived from the EEG recordings in Figure 11. The eye blinks are clearly concentrated in component 1. Component 14 contains the steady-state signal.
are much indebted to Jean-Fran¸cois Cardoso for insights and helpful comments on the stability criteria and Tony Bell for general comments and discussions. We are grateful to Tzyy-Ping Jung and Scott Makeig for EEG data, as well as useful discussions and comments, and to Olivier Coenen for helpful comments. We thank the reviewers for fruitful comments. References Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10, 251–276. Amari, S., & Cardoso, J.-F. (1997). Blind source separation—Semiparametric statistical approach. IEEE Trans. on Signal Processing, 45(11), 2692–2700. Amari, S., Chen, T.-P., & Cichocki, A. (1997). Stability analysis of adaptive blind source separation. Neural Networks, 10(8), 1345–1352. Amari, S., Cichocki, A., & Yang, H. (1996). A new learning algorithm for blind signal separation. In D. Touretzky, M. Mozer, & M. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 757–763). Cambridge, MA: MIT Press. Attias, H. (1999). Blind separation of noisy mixtures: An EM algorithm for factor analysis. Neural Computation, 11, to appear in 11-4.
An Extended Infomax Algorithm
439
Bell, A. J., & Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7, 1129– 1159. Berg, P., & Scherg, M. (1991). Dipole models of eye movements and blinks. Electroencephalog. Clin. Neurophysiolog., 79, 36–44. Cardoso, J.-F. (1998). Blind signal separation: Statistical principles. Proceedings of IEEE, 86(10), 2009–2025. Cardoso, J.-F. (in press). Unsupervised adaptive filtering. In S. Haykin (Ed.), Entropic contrasts for source separation. Englewood Cliffs, NJ: Prentice Hall. Cardoso, J.-F. (1997). Infomax and maximum likelihood for blind source separation. IEEE Signal Processing Letters, 4(4), 112–114. Cardoso, J.-F., & Laheld, B. (1996). Equivariant adaptive source separation. IEEE Trans. on S.P., 45(2), 434–444. Cardoso, J., & Soloumiac, A. (1993). Blind beamforming for non-gaussian signals. IEE Proceedings-F, 140(46), 362–370. Cichocki, A., Unbehauen, R., & Rummert, E. (1994). Robust learning algorithm for blind separation of signals. Electronics Letters, 30(17), 1386–1387. Comon, P. (1994). Independent component analysis—A new concept? Signal Processing, 36(3), 287–314. Cover, T., & Thomas, J. (Eds.). (1991). Elements of information theory. New York: Wiley. Deco, G., & Obradovic, D. (1996). An information-theoretic approach to neural computing. Berlin: Springer-Verlag. Gaeta, M., & Lacoume, J.-L. (1990). Source separation without prior knowledge: The maximum likelihood solution. Proc. EUSIPO (pp. 621–624). Girolami, M. (1997). Self-organizing artificial neural networks for signal separation. Unpublished Ph.D. dissertation, Paisley University, Scotland. Girolami, M. (1998). An alternative perspective on adaptive independent component analysis algorithms. Neural Computation, 10, 2103–2114. Girolami, M., & Fyfe, C. (1997a). Extraction of independent signal sources using a deflationary exploratory projection pursuit network with lateral inhibition. I.E.E Proceedings on Vision, Image and Signal Processing Journal, 14(5), 299–306. Girolami, M., & Fyfe, C. (1997b). Generalised independent component analysis through unsupervised learning with emergent bussgang properties. In Proc. ICNN (pp. 1788–1891). Houston, TX. Hochreiter, S., & Schmidhuber, J. (1999). Feature extraction through LOCOCODE. Neural Computation, 11, to appear in 11-3. Hyvaerinen, A., & Oja, E. (1997). A fast fixed-point algorithm for independent component analysis. Neural Computation, 9, 1483–1492. Jung, T.-P., Humphries, C., Lee, T.-W., Makeig, S., McKeown, M., Iragui, V., & Sejnowski, T. J. (1998). Extended ICA removes artifacts from electroencephalographic recordings. In M. Kearns, M. Jordan, & S. Solla (Eds.), Advances in neural information processing systems, 10 (pp. 894–900). Cambridge, MA: MIT Press. Jutten, C., & H´erault, J. (1991). Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture. Signal Processing, 24, 1–10. Karhunen, J. (1996). Neural approaches to independent component analysis
440
Te-Won Lee, Mark Girolami, and Terrence J. Sejnowski
and source separation. In Proc. 4th European Symposium on Artificial Neural Networks (pp. 249–266). Bruges, Belgium. Karhunen, J., Oja, E., Wang, L., Vigario, R., & Joutsensalo, J. (1997). A class of neural networks for independent component analysis. IEEE Trans. on Neural Networks, 8, 487–504. Lee, T.-W. (1998). Independent component analysis: Theory and applications. Dordrecht: Kluwer Academic Publishers, ISBN: 0-7923-8261-7. Lee, T.-W., Girolami, M., Bell, A. J., & Sejnowski, T. J. (1999). A unifying framework for independent component analysis. Computers and Mathematics with Applications, in press. Lee, T.-W., Koehler, B., & Orglmeister, R. (1997). Blind separation of nonlinear mixing models. In IEEE NNSP (pp. 406–415). Florida. Lee, T.-W., Lewicki, M. S., Girolami, M., & Sejnowski, T. J. (in press). Blind source separation of more sources than mixtures using overcomplete representations. IEEE Signal Processing Letters. Lee, T.-W., Lewicki, M. S., and Sejnowski, T. J. (1999). Unsupervised classification with non-gaussian mixture models using ICA. In Advances in neural information processing systems, 11, Cambridge, MA: MIT Press. Lewicki, M., & Sejnowski, T. J. (1998). Learning nonlinear overcomplete representations for efficient coding. In M. Kearns, M. Jordan, & S. Solla (Eds.), Advances in neural information processing systems, 10 (pp. 815–821). Cambridge, MA: MIT Press. Lin, J., & Cowan, J. (1997). Faithful representation of separable input distributions. Neural Computation, 9, 6:1305–1320. Makeig, S., Bell, A. J., Jung, T., & Sejnowski, T. J. (1996). Independent component analysis of electroencephalographic data. In D. Touretzky, M. Moser, & M. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 145–151). Cambridge, MA: MIT Press. Makeig, S., & Inlow, M. (1993). Changes in the EEG spectrum predict fluctuations in error rate in an auditory vigilance task. Society for Psychophysiology, 28, S39. Makeig, S., Jung, T., Bell, A. J., Ghahremani, D., & Sejnowski, T. J. (1997). Blind separation of event-related brain response into spatial independent components. Proceedings of the National Academy of Sciences, 94, 10979–10984. McKeown, M., Makeig, S., Brown, G., Jung, T.-P., Kindermann, S., Lee, T.-W., & Sejnowski, T. J. (1998). Spatially independent activity patterns in functional magnetic resonance imaging data during the Stroop color-naming task. Proceedings of the National Academy of Sciences, 95, 803–810. Moulines, E., Cardoso, J.-F., & Cassiat, E. (1997). Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models. In Proc. ICASSP’97 (Vol. 5, pp. 3617–3620). Munich. Nadal, J.-P., & Parga, N. (1994). Non linear neurons in the low noise limit: A factorial code maximizes information transfer. Network, 5, 565–581. Nadal, J.-P., & Parga, N. (1997). Redundancy reduction and independent component analysis: Conditions on cumulants and adaptive approaches. Neural Computation, 9, 1421–1456. Oja, E. (1997). The nonlinear PCA learning rule in independent component analysis. Neurocomputing, 17, 25–45.
An Extended Infomax Algorithm
441
Pajunen, P., & Karhunen, J. (1997). A maximum likelihood approach to nonlinear blind source separation. In ICANN (pp. 541–546). Lausanne. Pearlmutter, B., & Parra, L. (1996). A context-sensitive generalization of ICA. In International Conference on Neural Information Processing (pp. 151–157). Pearson, K. (1894). Contributions to the mathematical study of evolution. Phil. Trans. Roy. Soc. A, 185(71). Pham, D.-T., & Garrat, P. (1997). Blind separation of mixture of independent sources through a quasi–maximum likelihood approach. IEEE Trans. on Signal Proc., 45(7), 1712–1725. Roth, Z., & Baram, Y. (1996). Multidimensional density shaping by sigmoids. IEEE Trans. on Neural Networks, 7(5), 1291–1298. Stuart, A., & Ord, J. (1987). Kendall’s advanced theory of statistic, 1, Distribution theory. New York: Wiley. Taleb, A., & Jutten, C. (1997). Nonlinear source separation: The post-nonlinear mixtures. In ESANN (pp. 279–284). Vigario, R., Hyvaerinen, A., & Oja, E. (1996). ICA fixed-point algorithm in extraction of artifacts from EEG. In IEEE Nordic Signal Processing Symposium (pp. 383–386). Espoo, Finland. Xu, L., Cheung, C., Yang, H., & Amari, S. (1997). Maximum equalization by entropy maximization and mixture of cumulative distribution functions. In Proc. of ICNN’97 (pp. 1821–1826). Houston. Yang, H., Amari, S., & Cichocki, A. (1997). Information back-propagation for blind separation of sources from non-linear mixtures. In Proc. of ICNN (pp. 2141–2146). Houston. Received August 1, 1997; accepted May 11, 1998.
LETTER
Communicated by Todd Leen
Mixtures of Probabilistic Principal Component Analyzers Michael E. Tipping Christopher M. Bishop Microsoft Research, St. George House, Cambridge CB2 3NH, U.K.
Principal component analysis (PCA) is one of the most popular techniques for processing, compressing, and visualizing data, although its effectiveness is limited by its global linearity. While nonlinear variants of PCA have been proposed, an alternative paradigm is to capture data complexity by a combination of local linear PCA projections. However, conventional PCA does not correspond to a probability density, and so there is no unique way to combine PCA models. Therefore, previous attempts to formulate mixture models for PCA have been ad hoc to some extent. In this article, PCA is formulated within a maximum likelihood framework, based on a specific form of gaussian latent variable model. This leads to a well-defined mixture model for probabilistic principal component analyzers, whose parameters can be determined using an expectationmaximization algorithm. We discuss the advantages of this model in the context of clustering, density modeling, and local dimensionality reduction, and we demonstrate its application to image compression and handwritten digit recognition. 1 Introduction Principal component analysis (PCA) (Jolliffe, 1986) has proved to be an exceedingly popular technique for dimensionality reduction and is discussed at length in most texts on multivariate analysis. Its many application areas include data compression, image analysis, visualization, pattern recognition, regression, and time-series prediction. The most common definition of PCA, due to Hotelling (1933), is that for a set of observed d-dimensional data vectors {tn }, n ∈ {1 . . . N}, the q principal axes wj , j ∈ {1, . . . , q}, are those orthonormal axes onto which the retained variance under projection is maximal. It can be shown that the vectors wj are given by the q dominant eigenvectors (those withP the largest associated eigenvalues) of the sample covariance matrix S = n (tn − ¯t)(tn − ¯t)T /N such that Swj = λj wj and where ¯t is the sample mean. The vector xn = WT (tn − ¯t), where W = (w1 , w2 , . . . , wq ), is thus a q-dimensional reduced representation of the observed vector tn . A complementary property of PCA, and that most closely related to the original discussions of Pearson (1901), is that the projection onto the Neural Computation 11, 443–482 (1999)
c 1999 Massachusetts Institute of Technology °
444
Michael E. Tipping and Christopher M. Bishop
P principal subspace minimizes the squared reconstruction error ktn − tˆn k2 . The optimal linear reconstruction of tn is given by ˆtn = Wxn + ¯t, where xn = WT (tn − ¯t), and the orthogonal columns of W span the space of the leading q eigenvectors of S. In this context, the principal component projection is often known as the Karhunen-Lo`eve transform. One limiting disadvantage of these definitions of PCA is the absence of an associated probability density or generative model. Deriving PCA from the perspective of density estimation would offer a number of important advantages, including the following: • The corresponding likelihood would permit comparison with other density-estimation techniques and facilitate statistical testing. • Bayesian inference methods could be applied (e.g., for model comparison) by combining the likelihood with a prior. • In classification, PCA could be used to model class-conditional densities, thereby allowing the posterior probabilities of class membership to be computed. This contrasts with the alternative application of PCA for classification of Oja (1983) and Hinton, Dayan, and Revow (1997). • The value of the probability density function could be used as a measure of the “degree of novelty” of a new data point, an alternative approach to that of Japkowicz, Myers, and Gluck (1995) and Petsche et al. (1996) in autoencoder-based PCA. • The probability model would offer a methodology for obtaining a principal component projection when data values are missing. • The single PCA model could be extended to a mixture of such models. This final advantage is particularly significant. Because PCA defines only a linear projection of the data, the scope of its application is necessarily somewhat limited. This has naturally motivated various developments of nonlinear PCA in an effort to retain a greater proportion of the variance using fewer components. Examples include principal curves (Hastie & Stuetzle, 1989; Tibshirani, 1992), multilayer autoassociative neural networks (Kramer, 1991), the kernel-function approach of Webb (1996), and the generative topographic mapping (GTM) of Bishop, Svens´en, and Williams (1998). An alternative paradigm to such global nonlinear approaches is to model nonlinear structure with a collection, or mixture, of local linear submodels. This philosophy is an attractive one, motivating, for example, the mixture-ofexperts technique for regression (Jordan & Jacobs, 1994). A number of implementations of “mixtures of PCA” have been proposed in the literature, each defining a different algorithm or a variation. The variety of proposed approaches is a consequence of ambiguity in the formulation of the overall model. Current methods for local PCA generally necessitate a two-stage procedure: a partitioning of the data space followed by esti-
Mixtures of Probabilistic Principal Component Analyzers
445
mation of the principal subspace within each partition. Standard Euclidean distance-based clustering may be performed in the partitioning phase, but more appropriately, the reconstruction error may be used as the criterion for cluster assignments. This conveys the advantage that a common cost measure is used in both stages. However, even recently proposed models that adopt this cost measure still define different algorithms (Hinton et al., 1997; Kambhatla & Leen, 1997), while a variety of alternative approaches for combining local PCA models have also been proposed (Broomhead, Indik, Newell, & Rand, 1991; Bregler & Omohundro, 1995; Hinton, Revow, & Dayan, 1995; Dony & Haykin, 1995). None of these algorithms defines a probability density. One difficulty in implementation is that when using “hard” clustering in the partitioning phase (Kambhatla & Leen, 1997), the overall cost function is inevitably nondifferentiable. Hinton et al. (1997) finesse this problem by considering the partition assignments as missing data in an expectationmaximization (EM) framework, and thereby propose a “soft” algorithm where instead of any given data point being assigned exclusively to one principal component analyzer, the responsibility for its generation is shared among all of the analyzers. The authors concede that the absence of a probability model for PCA is a limitation to their approach and propose that the responsibility of the jth analyzer for reconstructing data point tn be P given by rnj = exp (−Ej2 /2σ 2 )/{ j0 exp (−Ej20 /2σ 2 )}, where Ej is the corresponding reconstruction cost. This allows the model to be determined by the maximization of a pseudo-likelihood function, and an explicit two-stage algorithm is unnecessary. Unfortunately, this also requires the introduction of a variance parameter σ 2 whose value is somewhat arbitrary, and again, no probability density is defined. Our key result is to derive a probabilistic model for PCA. From this a mixture of local PCA models follows as a natural extension in which all of the model parameters may be estimated through the maximization of a single likelihood function. Not only does this lead to a clearly defined and unique algorithm, but it also conveys the advantage of a probability density function for the final model, with all the associated benefits as outlined above. In section 2, we describe the concept of latent variable models. We then introduce probabilistic principal component analysis (PPCA) in section 3, showing how the principal subspace of a set of data vectors can be obtained within a maximum likelihood framework. Next, we extend this result to mixture models in section 4, and outline an efficient EM algorithm for estimating all of the model parameters in a mixture of probabilistic principal component analyzers. The partitioning of the data and the estimation of local principal axes are automatically linked. Furthermore, the algorithm implicitly incorporates a soft clustering similar to that implemented by Hinton et al. (1997), in which the parameter σ 2 appears naturally within the model.
446
Michael E. Tipping and Christopher M. Bishop
Indeed, σ 2 has a simple interpretation and is determined by the same EM procedure used to update the other model parameters. The proposed PPCA mixture model has a wide applicability, and we discuss its advantages from two distinct perspectives. First, in section 5, we consider PPCA for dimensionality reduction and data compression in local linear modeling. We demonstrate the operation of the algorithm on a simple toy problem and compare its performance with that of an explicit reconstruction-based nonprobabilistic modeling method on both synthetic and real-world data sets. A second perspective is that of general gaussian mixtures. The PPCA mixture model offers a way to control the number of parameters when estimating covariance structures in high dimensions, while not overconstraining the model flexibility. We demonstrate this property in section 6 and apply the approach to the classification of images of handwritten digits. Proofs of key results and algorithmic details are provided in the appendixes. 2 Latent Variable Models and PCA 2.1 Latent Variable Models. A latent variable model seeks to relate a ddimensional observed data vector t to a corresponding q-dimensional vector of latent variables x: t = y(x; w) + ²,
(2.1)
where y(·; ·) is a function of the latent variables x with parameters w, and ² is an x-independent noise process. Generally, q < d such that the latent variables offer a more parsimonious description of the data. By defining a prior distribution over x, together with the distribution of ², equation 2.1 induces a corresponding distribution in the data space, and the model parameters may then be determined by maximum likelihood techniques. Such a model may also be termed generative, as data vectors t may be generated by sampling from the x and ² distributions and applying equation 2.1. 2.2 Factor Analysis. Perhaps the most common example of a latent variable model is that of statistical factor analysis (Bartholomew, 1987), in which the mapping y(x; w) is a linear function of x: t = Wx + µ + ².
(2.2)
Conventionally, the latent variables are defined to be independent and gaussian with unit variance, so x ∼ N (0, I). The noise model is also gaussian such that ² ∼ N (0, Ψ), with Ψ diagonal, and the (d × q) parameter matrix W contains the factor loadings. The parameter µ permits the data model to have nonzero mean. Given this formulation, the observation vectors are
Mixtures of Probabilistic Principal Component Analyzers
447
also normally distributed t ∼ N (µ, C), where the model covariance is C = Ψ + WWT . (As a result of this parameterization, C is invariant under postmultiplication of W by an orthogonal matrix, equivalent to a rotation of the x coordinate system.) The key motivation for this model is that because of the diagonality of Ψ, the observed variables t are conditionally independent given the latent variables, or factors, x. The intention is that the dependencies between the data variables t are explained by a smaller number of latent variables x, while ² represents variance unique to each observation variable. This is in contrast to conventional PCA, which effectively treats both variance and covariance identically. There is no closed-form analytic solution for W and Ψ, so their values must be determined by iterative procedures. 2.3 Links from Factor Analysis to PCA. In factor analysis, the subspace defined by the columns of W will generally not correspond to the principal subspace of the data. Nevertheless, certain links between the two methods have been noted. For instance, it has been observed that the factor loadings and the principal axes are quite similar in situations where the estimates of the elements of Ψ turn out to be approximately equal (e.g., Rao, 1955). Indeed, this is an implied result of the fact that if Ψ = σ 2 I and an isotropic, rather than diagonal, noise model is assumed, then PCA emerges if the d − q smallest eigenvalues of the sample covariance matrix S are exactly equal. This homoscedastic residuals model is considered by Basilevsky (1994, p. 361), for the case where the model covariance is identical to its data sample counterpart. Given this restriction, the factor loadings W and noise variance σ 2 are identifiable (assuming correct choice of q) and can be determined analytically through eigendecomposition of S, without resort to iteration (Anderson, 1963). This established link with PCA requires that the d − q minor eigenvalues of the sample covariance matrix be equal (or, more trivially, be negligible) and thus implies that the covariance model must be exact. Not only is this assumption rarely justified in practice, but when exploiting PCA for dimensionality reduction, we do not require an exact characterization of the covariance structure in the minor subspace, as this information is effectively discarded. In truth, what is of real interest in the homoscedastic residuals model is the form of the maximum likelihood solution when the model covariance is not identical to its data sample counterpart. Importantly, we show in the following section that PCA still emerges in the case of an approximate model. In fact, this link between factor analysis and PCA had been partially explored in the early factor analysis literature by Lawley (1953) and Anderson and Rubin (1956). Those authors showed that the maximum likelihood solution in the approximate case was related to the eigenvectors of the sample covariance matrix, but did not show that these were the principal eigenvectors but instead made this additional assumption. In the next section (and in appendix A) we extend this earlier
448
Michael E. Tipping and Christopher M. Bishop
work to give a full characterization of the properties ¢ of the model we term ¡ probabilistic PCA. Specifically, with ² ∼ N 0, σ 2 I , the columns of the maximum likelihood estimator WML are shown to span the principal subspace of the data even when C 6= S. 3 Probabilistic PCA ¢ ¡ 3.1 The Probability Model. For the case of isotropic noise ² ∼ N 0, σ 2 I , equation 2.2 implies a probability distribution over t-space for a given x of the form ½ ¾ 1 (3.1) p(t|x) = (2πσ 2 )−d/2 exp − 2 kt − Wx − µk2 . 2σ With a gaussian prior over the latent variables defined by ½ ¾ 1 p(x) = (2π)−q/2 exp − xT x , 2
(3.2)
we obtain the marginal distribution of t in the form Z p(t) =
p(t|x)p(x)dx,
½ ¾ 1 −d/2 −1/2 T −1 |C| exp − (t − µ) C (t − µ) , = (2π) 2
(3.3) (3.4)
where the model covariance is C = σ 2 I + WWT .
(3.5)
Using Bayes’ rule, the posterior distribution of the latent variables x given the observed t may be calculated: p(x|t) = (2π)−q/2 |σ −2 M|1/2 · oT 1n x − M−1 WT (t − µ) (σ −2 M) × exp − 2 o¸ n x − M−1 WT (t − µ) ,
(3.6)
where the posterior covariance matrix is given by σ 2 M−1 = σ 2 (σ 2 I + WT W)−1 . Note that M is q × q while C is d × d.
(3.7)
Mixtures of Probabilistic Principal Component Analyzers
449
The log-likelihood of observing the data under this model is
L=
N X
© ª ln p(tn ) ,
n=1
=−
³ ´o Nn d ln(2π) + ln |C| + tr C−1 S , 2
(3.8)
where S=
N 1 X (tn − µ)(tn − µ)T , N n=1
(3.9)
is the sample covariance matrix of the observed {tn }. 3.2 Properties of the Maximum Likelihood Estimators. The maximum likelihood estimate of the parameter µ is given by the mean of the data:
µML =
N 1 X tn . N n=1
(3.10)
We now consider the maximum likelihood estimators for the parameters W and σ 2 . 3.2.1 The Weight Matrix W. The log-likelihood (see equation 3.8) is maximized when the columns of W span the principal subspace of the data. To show this we consider the derivative of equation 3.8 with respect to W: ∂L = N(C−1 SC−1 W − C−1 W). ∂W
(3.11)
In appendix A it is shown that with C given by equation 3.5, the only nonzero stationary points of equation 3.11 occur for W = Uq (Λq − σ 2 I)1/2 R,
(3.12)
where the q column vectors in the d × q matrix Uq are eigenvectors of S, with corresponding eigenvalues in the q × q diagonal matrix Λq , and R is an arbitrary q×q orthogonal rotation matrix. Furthermore, it is also shown that the stationary point corresponding to the global maximum of the likelihood occurs when Uq comprises the principal eigenvectors of S, and thus Λq contains the corresponding eigenvalues λ1 , . . . , λq , where the eigenvalues of S are indexed in order of decreasing magnitude. All other combinations of eigenvectors represent saddle points of the likelihood surface. Thus, from equation 3.12, the latent variable model defined by equation 2.2 effects a
450
Michael E. Tipping and Christopher M. Bishop
mapping from the latent space into the principal subspace of the observed data. 3.2.2 The Noise Variance σ 2 . It may also be shown that for W = WML , the maximum likelihood estimator for σ 2 is given by 2 = σML
d 1 X λj , d − q j=q+1
(3.13)
2 has a clear where λq+1 , . . . , λd are the smallest eigenvalues of S, and so σML interpretation as the average variance “lost” per discarded dimension.
3.3 Dimensionality Reduction and Optimal Reconstruction. To implement probabilistic PCA, we would generally first compute the usual eigendecomposition of S (we consider an alternative, iterative approach shortly), 2 is found from equation 3.13 followed by W after which σML ML from equation 3.12. This is then sufficient to define the associated density model for PCA, allowing the advantages listed in section 1 to be exploited. In conventional PCA, the reduced-dimensionality transformation of a data point tn is given by xn = UTq (tn −µ) and its reconstruction by ˆtn = Uq xn + µ. This may be similarly achieved within the PPCA formulation. However, we note that in the probabilistic framework, the generative model defined by equation 2.2 represents a mapping from the lower-dimensional latent space to the data space. So in PPCA, the probabilistic analog of the dimensionality reduction process of conventional PCA would be to invert the conditional distribution p(t|x) using Bayes’ rule, in equation 3.6, to give p(x|t). In this case, each data point tn is represented in the latent space not by a single vector, but by the gaussian posterior distribution defined by equation 3.6. As an alternative to the standard PCA projection, then, a convenient summary of this distribution and representation of tn would be the posterior mean hxn i = M−1 WTML (tn − µ), a quantity that also arises naturally in (and is computed in) the EM implementation of PPCA considered in section 3.4. Note also from equation 3.6 that the covariance of the posterior distribution is given by σ 2 M−1 and is therefore constant for all data points. However, perhaps counterintuitively given equation 2.2, WML hxn i + µ is not the optimal linear reconstruction of tn . This may be seen from the fact that for σ 2 > 0, WML hxn i + µ is not an orthogonal projection of tn , as a consequence of the gaussian prior over x causing the posterior mean projection to become skewed toward the origin. If we consider the limit as σ 2 → 0, the projection WML hxn i = WML (WTML WML )−1 WTML (tn − µ) does become orthogonal and is equivalent to conventional PCA, but then the density model is singular and thus undefined. Taking this limit is not necessary, however, since the optimal least-squares linear reconstruction of the data from the posterior mean vectors hxn i may
Mixtures of Probabilistic Principal Component Analyzers
451
be obtained from (see appendix B) ³ ´−1 ˆtn = WML WTML WML Mhxn i + µ,
(3.14)
with identical reconstruction error to conventional PCA. For reasons of probabilistic elegance, therefore, we might choose to exploit the posterior mean vectors hxn i as the reduced-dimensionality representation of the data, although there is no material benefit in so doing. Indeed, we note that in addition to the conventional PCA representation UTq (tn − µ), the vectors xˆ n = WTML (tn − µ) could equally be used without loss of information and reconstructed using ³ ´−1 ˆtn = WML WTML WML xˆ n + µ. 3.4 An EM Algorithm for PPCA. By a simple extension of the EM formulation for parameter estimation in the standard linear factor analysis model (Rubin & Thayer 1982), we can obtain a principal component projection by maximizing the likelihood function (see equation 3.8). We are not suggesting that such an approach necessarily be adopted for probabilistic PCA; normally the principal axes would be estimated in the conventional manner, via eigendecomposition of S, and subsequently incorporated in the probability model using equations 3.12 and 3.13 to realize the advantages outlined in the introduction. However, as discussed in appendix A.5, there may be an advantage in the EM approach for large d since the presented algorithm, although iterative, requires neither computation of the d × d covariance matrix, which is O(Nd2 ), nor its explicit eigendecomposition, which is O(d3 ). We derive the EM algorithm and consider its properties from the computational perspective in appendix A.5. 3.5 Factor Analysis Revisited. The probabilistic PCA algorithm was obtained by introducing a constraint into the noise matrix of the factor analysis latent variable model. This apparently minor modification leads to significant differences in the behavior of the two methods. In particular, we now show that the covariance properties of the PPCA model are identical to those of conventional PCA and are quite different from those of standard factor analysis. Consider a nonsingular linear transformation of the data variables, so that t → At. Using equation 3.10, we see that under such a transformation, the maximum likelihood solution for the mean will be transformed as µML → AµML . From equation 3.9, it then follows that the covariance matrix will transform as S → ASAT . The log-likelihood for the latent variable model, from equation 3.8, is
452
Michael E. Tipping and Christopher M. Bishop
given by
L(W, Ψ) = −
½ N d ln(2π ) + ln |WWT + Ψ| 2 i¾ h + tr (WWT + Ψ)−1 S ,
(3.15)
where Ψ is a general noise covariance matrix. Thus, using equation 3.15, we see that under the transformation t → At, the log-likelihood will transform as
L(W, Ψ) → L(A−1 W, A−1 ΨA−T ) − N ln |A|,
(3.16)
where A−T ≡ (A−1 )T . Thus, if WML and ΨML are maximum likelihood solutions for the original data, then AWML and AΨML AT will be maximum likelihood solutions for the transformed data set. In general, the form of the solution will not be preserved under such a transformation. However, we can consider two special cases. First, suppose Ψ is a diagonal matrix, corresponding to the case of factor analysis. Then Ψ will remain diagonal provided A is also a diagonal matrix. This says that factor analysis is covariant under component-wise rescaling of the data variables: the scale factors simply become absorbed into rescaling of the noise variances, and the rows of W are rescaled by the same factors. Second, consider the case Ψ = σ 2 I, corresponding to PPCA. Then the transformed noise covariance σ 2 AAT will be proportional to the unit matrix only if AT = A−1 — in other words, if A is an orthogonal matrix. Transformation of the data vectors by multiplication with an orthogonal matrix corresponds to a rotation of the coordinate system. This same covariance property is shared by standard nonprobabilistic PCA since a rotation of the coordinates induces a corresponding rotation of the principal axes. Thus we see that factor analysis is covariant under componentwise rescaling, while PPCA and PCA are covariant under rotations, as illustrated in Figure 1. 4 Mixtures of Probabilistic Principal Component Analyzers The association of a probability model with PCA offers the tempting prospect of being able to model complex data structures with a combination of local PCA models through the mechanism of a mixture of probabilistic principal component analysers (Tipping & Bishop, 1997). This formulation would permit all of the model parameters to be determined from maximum likelihood, where both the appropriate partitioning of the data and the determination of the respective principal axes occur automatically as the likelihood is maximized. The log-likelihood of observing the data set for such a mixture
Mixtures of Probabilistic Principal Component Analyzers
453
Figure 1: Factor analysis is covariant under a componentwise rescaling of the data variables (top plots), while PCA and probabilistic PCA are covariant under rotations of the data space coordinates (bottom plots).
model is:
L= =
N X
© ª ln p(tn ) ,
n=1
( M X
N X n=1
ln
(4.1) )
πi p(tn |i) ,
(4.2)
i=1
where p(t|i) is a single PPCA P model and πi is the corresponding mixing proportion, with πi ≥ 0 and πi = 1. Note that a separate mean vector µi is now associated with each of the M mixture components, along with the parameters Wi and σi2 . A related model has recently been exploited for data visualization (Bishop & Tipping, 1998), and a similar approach, based on
454
Michael E. Tipping and Christopher M. Bishop
the standard factor analysis diagonal (Ψ) noise model, has been employed for handwritten digit recognition (Hinton et al. 1997), although it does not implement PCA. The corresponding generative model for the mixture case now requires the random choice of a mixture component according to the proportions πi , followed by sampling from the x and ² distributions and applying equation 2.2 as in the single model case, taking care to use the appropriate parameters µi , Wi , and σi2 . Furthermore, for a given data point t, there is now a posterior distribution associated with each latent space, the mean of which for space i is given by (σi2 I + WTi Wi )−1 WTi (t − µi ). We can develop an iterative EM algorithm for optimization of all of the model parameters πi , µi , Wi , and σi2 . If Rni = p(i|tn ) is the posterior responsibility of mixture i for generating data point tn , given by Rni =
p(tn |i)πi , p(tn )
(4.3)
then in appendix C it is shown that we obtain the following parameter updates: N 1 X Rni , N n=1 PN Rni tn ei = Pn=1 µ . N n=1 Rni
e πi =
(4.4) (4.5)
ei correspond exactly to those of a stanThus the updates for e πi and µ dard gaussian mixture formulation (e.g., see Bishop, 1995). Furthermore, in appendix C, it is also shown that the combination of the E- and M-steps leads to the intuitive result that the axes Wi and the noise variance σi2 are determined from the local responsibility–weighted covariance matrix: Si =
N 1 X ei )(tn − µ ei )T , Rni (tn − µ e πi N n=1
(4.6)
by standard eigendecomposition in exactly the same manner as for a single PPCA model. However, as noted in section 3.4 (and also in appendix A.5), for larger values of data dimensionality d, computational advantages can be obtained if Wi and σi2 are updated iteratively according to an EM schedule. This is discussed for the mixture model in appendix C. Iteration of equations 4.3, 4.4, and 4.5 in sequence followed by computation of Wi and σi2 , from either equation 4.6 using equations 2.12 and 2.13 or using the iterative updates in appendix C, is guaranteed to find a local maximum of the log-likelihood in equation 4.2. At convergence of the algorithm each weight matrix Wi spans the principal subspace of its respective Si .
Mixtures of Probabilistic Principal Component Analyzers
455
In the next section we consider applications of this PPCA mixture model, beginning with data compression and reconstruction tasks. We then consider general density modeling in section 6. 5 Local Linear Dimensionality Reduction In this section we begin by giving an illustration of the application of the PPCA mixture algorithm to a synthetic data set. More realistic examples are then considered, with an emphasis on cases in which a principal component approach is motivated by the objective of deriving a reduced-dimensionality representation of the data, which can be reconstructed with minimum error. We will therefore contrast the clustering mechanism in the PPCA mixture model with that of a hard clustering approach based explicitly on reconstruction error as used in a typical algorithm. 5.1 Illustration for Synthetic Data. For a demonstration of the mixture of PPCA algorithm, we generated a synthetic data set comprising 500 data points sampled uniformly over the surface of a hemisphere, with additive gaussian noise. Figure 2a shows this data. A mixture of 12 probabilistic principal component analyzers was then fitted to the data using the EM algorithm outlined in the previous section, with latent space dimensionality q = 2. Because of the probabilistic formalism, a generative model of the data is defined, and we emphasize this by plotting a second set of 500 data points, obtained by sampling from the fitted generative model. These data points are shown in Figure 2b. Histograms of the distances of all the data points from the hemisphere are also given to indicate more clearly the accuracy of the model in capturing the structure of the underlying generator. 5.2 Clustering Mechanisms. Generating a local PCA model of the form illustrated above is often prompted by the ultimate goal of accurate data reconstruction. Indeed, this has motivated Kambhatla and Leen (1997) and Hinton et al. (1997) to use squared reconstruction error as the clustering criterion in the partitioning phase. Dony and Haykin (1995) adopt a similar approach to image compression, although their model has no set of independent mean parameters µi . Using the reconstruction criterion, a data point is assigned to the component that reconstructs it with lowest error, and the principal axes are then reestimated within each cluster. For the mixture of PPCA model, however, data points are assigned to mixture components (in a soft fashion) according to the responsibility Rni of the mixture component for its generation. Since Rni = p(tn |i)πi /p(tn ) and p(tn ) is constant for all components, Rni ∝ p(tn |i), and we may gain further insight into the clustering by considering the probability density associated with component i at
456
Michael E. Tipping and Christopher M. Bishop
(a)
100
0 −0.2
(b)
−0.1 0 0.1 distance from sphere
0.2
−0.1 0 0.1 distance from sphere
0.2
100
0 −0.2
Figure 2: Modeling noisy data on a hemisphere. (a) On the left, the synthetic data; on the right, a histogram of the Euclidean distances of each data point to the sphere. (b) Data generated from the fitted PPCA mixture model with the synthetic data on the left and the histogram on the right.
data point tn : n o p(tn |i) = (2π)−d/2 |Ci |−1/2 exp −E2ni /2 ,
(5.1)
where E2ni = (tn − µi )T C−1 i (tn − µi ), Ci =
σi2 I
+
Wi WTi .
(5.2) (5.3)
It is helpful to express the matrix Wi in terms of its singular value decomposition (and although we are considering an individual mixture component i, the i subscript will be omitted for notational clarity): W = Uq (Kq − σ 2 I)1/2 R,
(5.4)
Mixtures of Probabilistic Principal Component Analyzers
457
where Uq is a d × q matrix of orthonormal column vectors and R is an arbitrary q×q orthogonal matrix. The singular values are parameterized, without loss of generality, in terms of (Kq −σ 2 I)1/2 , where Kq = diag(k1 , k2 , . . . , kq ) is a q × q diagonal matrix. Then n o−1 (tn − µ). E2n = (tn − µ)T σ 2 I + Uq (Kq − σ 2 I)UTq
(5.5)
The data point tn may also be expressed in terms of the basis of vectors U = (Uq , Ud−q ), where Ud−q comprises (d − q) vectors perpendicular to Uq , which complete an orthonormal set. In this basis, we define zn = UT (tn − µ) and so tn − µ = Uzn , from which equation 5.5 may then be written as n o−1 Uzn , E2n = zTn UT σ 2 I + Uq (Kq − σ 2 I)UTq = zTn D−1 zn ,
(5.6) (5.7)
where D = diag(k1 , k2 , . . . , kq , σ 2 , . . . , σ 2 ) is a d × d diagonal matrix. Thus: zTout zout , σ2 = E2in + E2rec /σ 2 ,
E2n = zTin K−1 q zin +
(5.8) (5.9)
where we have partitioned the elements of z into zin , the projection of tn − µ onto the subspace spanned by W, and zout , the projection onto the corresponding perpendicular subspace. Thus, E2rec is the squared reconstruction error, and E2in may be interpreted as an in-subspace error term. At the maximum likelihood solution, Uq is the matrix of eigenvectors of the local covariance matrix and Kq = Λq . ¡ ¢ As σi2 → 0, Rni ∝ πi exp −E2rec /2 and, for equal prior probabilities, cluster assignments are equivalent to a soft reconstruction-based clustering. However, for σA2 , σB2 > 0, consider a data point that lies in the subspace of a relatively distant component A, which may be reconstructed with zero error yet lies closer to the mean of a second component B. The effect of the noise variance σB2 in equation 5.9 is to moderate the contribution of E2rec for component B. As a result, the data point may be assigned to the nearer component B even though the reconstruction error is considerably greater, given that it is sufficiently distant from the mean of A such that E2in for A is large. It should be expected, then, that mixture of PPCA clustering would result in more localized clusters, but with the final reconstruction error inferior to that of a clustering model based explicitly on a reconstruction criterion. Conversely, it should also be clear that clustering the data according to the proximity to the subspace alone will not necessarily result in localized partitions (as noted by Kambhatla, 1995, who also considers the relationship
458
Michael E. Tipping and Christopher M. Bishop
Figure 3: Comparison of the partitioning of the hemisphere effected by a VQPCA-based model (left) and a PPCA mixture model (right). The illustrated boundaries delineate regions of the hemisphere that are best reconstructed by a particular local PCA model. One such region is shown shaded to emphasize that clustering according to reconstruction error results in a nonlocalized partitioning. In the VQPCA case, the circular effects occur when principal component planes intersect beneath the surface of the hemisphere.
of such an algorithm to a probabilistic model). That this is so is simply illustrated in Figure 3, in which a collection of 12 conventional PCA models have been fitted to the hemisphere data, according to the VQPCA (vectorquantization PCA) algorithm of Kambhatla and Leen (1997), defined as follows: 1. Select initial cluster centers µi at random from points in the data set, and assign all data points to the nearest (in terms of Euclidean distance) cluster center. 2. Set the Wi vectors to the first two principal axes of the covariance matrix of cluster i. 3. Assign data points to the cluster that best reconstructs them, setting each µi to the mean of those data points assigned to cluster i. 4. Repeat from step 2 until the cluster allocations are constant. In Figure 3, data points have been sampled over the hemisphere, without noise, and allocated to the cluster that best reconstructs them. The left plot shows the partitioning associated with the best (i.e., lowest reconstruction error) model obtained from 100 runs of the VQPCA algorithm. The right plot shows a similar partitioning for the best (i.e., greatest likelihood) PPCA mixture model using the same number of components, again from 100 runs. Note that the boundaries illustrated in this latter plot were obtained using
Mixtures of Probabilistic Principal Component Analyzers
459
Table 1: Data Sets Used for Comparison of Clustering Criteria. Data Set
N
d
M
q
Description
Hemisphere
500
3
12
2
Synthetic data used above
Oil
500
12
12
2
Diagnostic measurements pipeline flows
Digit 1
500
64
10
10
8 × 8 gray-scale images of handwritten digit 1
Digit 2
500
64
10
10
8 × 8 gray-scale images of handwritten digit 2
Image
500
64
8
4
8 × 8 gray-scale blocks from a photographic image
EEG
300
30
8
5
Delay vectors from an electroencephalogram time-series signal
from
oil
assignments based on reconstruction error for the final model, in identical fashion to the VQPCA case, and not on probabilistic responsibility. We see that the partitions formed when clustering according to reconstruction error alone can be nonlocal, as exemplified by the shaded component. This phenomenon is rather contrary to the philosophy of local dimensionality reduction and is an indirect consequence of the fact that reconstruction-based local PCA does not model the data in a probabilistic sense. However, we might expect that algorithms such as VQPCA should offer better performance in terms of the reconstruction error of the final solution, having been designed explicitly to optimize that measure. In order to test this, we compared the VQPCA algorithm with the PPCA mixture model on six data sets, detailed in Table 1. Figure 4 summarizes the reconstruction error of the respective models, and in general, VQPCA performs better, as expected. However, we also note two interesting aspects of the results. First, in the case of the oil data, the final reconstruction error of the PPCA model on both training and test sets is counterintuitively superior, despite the fact that the partitioning of the data space was based only partially on reconstruction error. This behavior is, we hypothesize, a result of the particular structure of that data set. The oil data are known to comprise a number of disjoint, but locally smooth, two-dimensional cluster structures (see Bishop & Tipping, 1998, for a visualization). For the oil data set, we observed that many of the models found by the VQPCA algorithm exhibit partitions that are not only often nonconnected (similar to those shown for the hemisphere in Figure 3) but may also span more than one of the disjoint cluster structures. The evidence of Figure 4 suggests that these models represent poor local minima of the reconstruc-
460
Michael E. Tipping and Christopher M. Bishop
Training Set 108
% Reconstruction Error
106 104 102 100 98 96 94 92
HEMISPHERE
OIL
DIGIT_1
DIGIT_2
IMAGE
EEG
IMAGE
EEG
Dataset
Test Set 106
% Reconstruction Error
104 102 100 98 96 94 92 90 88
HEMISPHERE
OIL
DIGIT_1
DIGIT_2
Dataset
Figure 4: Reconstruction errors for reconstruction-based local PCA (VQPCA) and the PPCA mixture. Errors for the latter (∗) have been shown relative to the former (∇), and are averaged over 100 runs with random initial configurations.
tion error cost function. The PPCA mixture algorithm does not find such suboptimal solutions, which would have low likelihood due to the locality implied by the density model. The experiment indicates that by avoiding these poor solutions, the PPCA mixture model is able to find solutions with lower reconstruction error (on average) than VQPCA.
Mixtures of Probabilistic Principal Component Analyzers
461
These observations apply only to the case of the oil data set. For the hemisphere, digit 1, image, and electroencephalogram (EEG) training sets, the data manifolds are less disjoint, and the explicit reconstruction-based algorithm, VQPCA, is superior. For the digit 2 case, the two algorithms appear approximately equivalent. A second aspect of Figure 4 is the suggestion that the PPCA mixture model algorithm may be less sensitive to overfitting. As would be expected, compared with the training set, errors on the test set increase for both algorithms (although, because the errors have been normalized to allow comparisons between data sets, this is not shown in Figure 4). However, with the exception of the case of the digit 2 data set, for the PPCA mixture model this increase is proportionately smaller than for VQPCA. This effect is most dramatic for the image data set, where PPCA is much superior on the test set. For that data set, the test examples were derived from a separate portion of the image (see below), and as such, the test set statistics can be expected to differ more significantly from the respective training set than for the other examples. A likely explanation is that because of the soft clustering of the PPCA mixture model, there is an inherent smoothing effect occurring when estimating the local sets of principal axes. Each set of axes is determined from its corresponding local responsibility–weighted covariance matrix, which in general will be influenced by many data points, not just the subset that would be associated with the cluster in a “hard” implementation. Because of this, the parameters in the Wi matrix in cluster i are also constrained by data points in neighboring clusters (j 6= i) to some extent. This notion is discussed in the context of regression by Jordan and Jacobs (1994) as motivation for their mixture-of-experts model, where the authors note how soft partitioning can reduce variance (in terms of the bias-variance decomposition). Although it is difficult to draw firm conclusions from this limited set of experiments, the evidence of Figure 4 does point to the presence of such an effect. 5.3 Application: Image Compression. As a practical example, we consider an application of the PPCA mixture model to block transform image coding. Figure 5 shows the original image. This 720 × 360 pixel image was segmented into 8 × 8 nonoverlapping blocks, giving a total data set of 4050 64-dimensional vectors. Half of these data, corresponding to the left half of the picture, were used as training data. The right half was reserved for testing; a magnified portion of the test image is also shown in Figure 5. A reconstruction of the entire image based on the first four principal components of a single PCA model determined from the block-transformed left half of the image is shown in Figure 6. Figure 7 shows the reconstruction of the original image when modeled by a mixture of probabilistic principal component analyzers. The model parameters were estimated using only the left half of the image. In this example, 12
462
Michael E. Tipping and Christopher M. Bishop
Figure 5: (Left) The original image. (Right) Detail.
Figure 6: The PCA reconstructed image, at 0.5 bit per pixel. (Left) The original image. (Right) Detail.
components were used, of dimensionality 4; after the model likelihood had been maximized, the image coding was performed in a “hard” fashion, by allocating data to the component with the lowest reconstruction error. The resulting coded image was uniformly quantized, with bits allocated equally to each transform variable, before reconstruction, in order to give a final bit rate of 0.5 bits per pixel (and thus compression of 16 to 1) in both Figures 6 and 7. In the latter case, the cost of encoding the mixture component label was included. For the simple principal subspace reconstruction, the normalized test error was 7.1 × 10−2 ; for the mixture model, it was 5.7 × 10−2 . The VQPCA algorithm gave a test error of 6.2 × 10−2 . 6 Density Modeling A popular approach to semiparametric density estimation is the gaussian mixture model (Titterington, Smith, & Makov, 1985). However, such models suffer from the limitation that if each gaussian component is described by a full covariance matrix, then there are d(d + 1)/2 independent covariance parameters to be estimated for each mixture component. Clearly, as the dimensionality of the data space increases, the number of data points required to specify those parameters reliably will become prohibitive. An alternative
Mixtures of Probabilistic Principal Component Analyzers
463
Figure 7: The mixture of PPCA reconstructed image, using the same bit rate as Figure 6. (Left) The original image. (Right) Detail.
approach is to reduce the number of parameters by placing a constraint on the form of the covariance matrix. (Another would be to introduce priors over the parameters of the full covariance matrix, as implemented by Ormoneit & Tresp, 1996.) Two common constraints are to restrict the covariance to be isotropic or to be diagonal. The isotropic model is highly constrained as it assigns only a single parameter to describe the entire covariance structure in the full d dimensions. The diagonal model is more flexible, with d parameters, but the principal axes of the elliptical gaussians must be aligned with the data axes, and thus each individual mixture component is unable to capture correlations among the variables. A mixture of PPCA models, where the covariance of each gaussian is parameterized by the relation C = σ 2 I + WWT , comprises dq + 1 − q(q − 1)/2 free parameters.1 (Note that the q(q − 1)/2 term takes account of the number of parameters needed to specify the arbitrary rotation R.) It thus permits the number of parameters to be controlled by the choice of q. When q = 0, the model is equivalent to an isotropic gaussian. With q = d − 1, the general covariance gaussian is recovered. 6.1 A Synthetic Example: Noisy Spiral Data. The utility of the PPCA mixture approach may be demonstrated with the following simple example. A 500-point data set was generated along a three-dimensional spiral configuration with added gaussian noise. The data were then modeled by both a mixture of PPCA models and a mixture of diagonal covariance gaussians, using eight mixture components. In the mixture of PPCA case, q = 1 for each component, and so there are four variance parameters per component compared with three for the diagonal model. The results are visualized in Figure 8, which illustrates both side and end projections of the data.
1 An alternative would be a mixture of factor analyzers, implemented by Hinton et al. (1997), although that comprises more parameters.
464
Michael E. Tipping and Christopher M. Bishop
Figure 8: Comparison of an eight-component diagonal variance gaussian mixture model with a mixture of PPCA model. The upper two plots give a view perpendicular to the major axis of the spiral; the lower two plots show the end elevation. The covariance structure of each mixture component is shown by projection of a unit Mahalanobis distance ellipse, and the log-likelihood per data point is given in parentheses above the figures.
The orientation of the ellipses in the diagonal model can be seen not to coincide with the local data structure, which is a result of the axial alignment constraint. A further consequence of the diagonal parameterization is that the means are also implicitly constrained because they tend to lie where the tangent to the spiral is parallel to either axis of the end elevation. This qualitative superiority of the PPCA approach is underlined quantitatively by the log-likelihood per data point given in parentheses in the figure. Such a result would be expected given that the PPCA model has an extra parameter in each mixture component, but similar results are observed if the spiral is embedded in a space of much higher dimensionality where the extra parameter in PPCA is proportionately less relevant. It should be intuitive that the axial alignment constraint of the diagonal model is, in general, particularly inappropriate when modeling a smooth
Mixtures of Probabilistic Principal Component Analyzers
465
Table 2: Log-Likelihood per Data Point Measured on Training and Test Sets for Gaussian Mixture Models with Eight Components and a 100-Point Training Set.
Training Test
Isotropic
Diagonal
Full
PPCA
−3.14 −3.68
−2.74 −3.43
−1.47 −3.09
−1.65 −2.37
and continuous lower dimensional manifold in higher dimensions, regardless of the intrinsic dimensionality. Even with q = 1, the PPCA approach is able to track the spiral manifold successfully. Finally, we demonstrate the importance of the use of an appropriate number of parameters by modeling a three-dimensional spiral data set of 100 data points (the number of data points was reduced to emphasize the overfitting) as above with isotropic, diagonal, and full covariance gaussian mixture models, along with a PPCA mixture model. For each model, the log-likelihood per data point for both the training data set and an unseen test set of 1000 data points is given in Table 2. As would be expected in this case of limited data, the full covariance model exhibits the best likelihood on the training set, but test set performance is worse than for the PPCA mixture. For this simple example, there is only one intermediate PPCA parameterization with q = 1 (q = 0 and q = 2 are equivalent to the isotropic and full covariance cases respectively). In realistic applications, where the dimensionality d will be considerably larger, the PPCA model offers the choice of a range of q, and an appropriate value can be determined using standard techniques for model selection. Finally, note that these advantages are not limited to mixture models, but may equally be exploited for the case of a single gaussian distribution. 6.2 Application: Handwritten Digit Recognition. One potential application for high-dimensionality density models is handwritten digit recognition. Examples of gray-scale pixel images of a given digit will generally lie on a lower-dimensional smooth continuous manifold, the geometry of which is determined by properties of the digit such as rotation, scaling, and thickness of stroke. One approach to the classification of such digits (although not necessarily the best) is to build a model of each digit separately, and classify unseen digits according to the model to which they are most similar. Hinton et al. (1997) gave an excellent discussion of the handwritten digit problem and applied a mixture of PCA approach, using soft reconstruction– based clustering, to the classification of scaled and smoothed 8 × 8 grayscale images taken from the CEDAR U.S. Postal Service database (Hull, 1994). The models were constructed using an 11,000-digit subset of the br
466
Michael E. Tipping and Christopher M. Bishop
Figure 9: Mean vectors µi , illustrated as gray-scale digits, for each of the 10 digit models. The model for a given digit is a mixture of 10 PPCA models, one centered at each of the pixel vectors shown on the corresponding row. Note how different components can capture different styles of digit.
data set (which was further split into training and validation sets), and the bs test set was classified according to which model best reconstructed each digit (in the squared-error sense). We repeated the experiment with the same data using the PPCA mixture approach using the same choice of parameter values (M = 10 and q = 10). To help visualize the final model, the means of each component µi are illustrated in digit form in Figure 9. The digits were again classified, using the same method of classification, and the best model on the validation set misclassified 4.64% of the digits in the test set. Hinton et al. (1997) reported an error of 4.91%, and we would expect the improvement to be a result partly of the localized clustering of the PPCA model, but also the use of individually estimated values of σi2 for each component, rather than a single, arbitrarily chosen, global value. One of the advantages of the PPCA methodology is that the definition of the density model permits the posterior probabilities of class membership
Mixtures of Probabilistic Principal Component Analyzers
467
to be computed for each digit and used for subsequent classification, rather than using reconstruction error as above. Classification according to the largest posterior probability for the M = 10 and q = 10 model resulted in an increase in error, and it was necessary to invest significant effort to optimize the parameters M and q for each model to provide comparable performance. Using this approach, our best classifier on the validation set misclassified 4.61% of the test set. An additional benefit of the use of posterior probabilities is that it is possible to reject a proportion of the test samples about which the classifier is most “unsure” and thus hopefully improve the classification performance. Using this approach to reject 5% of the test examples resulted in a misclassification rate of 2.50%. (The availability of posteriors can be advantageous in other applications, where they may be used in various forms of follow-on processing.) 7 Conclusions Modeling complexity in data by a combination of simple linear models is an attractive paradigm offering both computational and algorithmic advantages along with increased ease of interpretability. In this article, we have exploited the definition of a probabilistic model for PCA in order to combine local PCA models within the framework of a probabilistic mixture in which all the parameters are determined from maximum likelihood using an EM algorithm. In addition to the clearly defined nature of the resulting algorithm, the primary advantage of this approach is the definition of an observation density model. A possible disadvantage of the probabilistic approach to combining local PCA models is that by optimizing a likelihood function, the PPCA mixture model does not directly minimize squared reconstruction error. For applications where this is the salient criterion, algorithms that explicitly minimize reconstruction error should be expected to be superior. Experiments indeed showed this to be generally the case, but two important caveats must be considered before any firm conclusions can be drawn concerning the suitability of a given model. First, and rather surprisingly, for one of the data sets (‘oil’) considered in the article, the final PPCA mixture model was actually superior in the sense of squared reconstruction error, even on the training set. It was demonstrated that algorithms incorporating reconstruction-based clustering do not necessarily generate local clusters, and it was reasoned that for data sets comprising a number of disjoint data structures, this phenomenon may lead to poor local minima. Such minima are not found by the PPCA density model approach. A second consideration is that there was also evidence that the smoothing implied by the soft clustering inherent in the PPCA mixture model helps to reduce overfitting, particularly in the case of the image compression experiment where the statistics of the test data set differed from the training data much more so than for other examples. In that instance, the reconstruction test error for the PPCA model was, on
468
Michael E. Tipping and Christopher M. Bishop
average, more than 10% lower. In terms of a gaussian mixture model, the mixture of probabilistic principal component analyzers enables data to be modeled in high dimensions with relatively few free parameters, while not imposing a generally inappropriate constraint on the covariance structure. The number of free parameters may be controlled through the choice of latent space dimension q, allowing an interpolation in model complexity from isotropic to full covariance structures. The efficacy of this parameterization was demonstrated by performance on a handwritten digit recognition task. Appendix A: Maximum Likelihood PCA A.1 The Stationary Points of the Log-Likelihood. The gradient of the log-likelihood (see equation 3.8) with respect to W may be obtained from standard matrix differentiation results (e.g., see Krzanowski & Marriott, 1994, p. 133): ∂L = N(C−1 SC−1 W − C−1 W). ∂W
(A.1)
At the stationary points SC−1 W = W,
(A.2)
assuming that σ 2 > 0, and thus that C−1 exists. This is a necessary and sufficient condition for the density model to remain nonsingular, and we will restrict ourselves to such cases. It will be seen shortly that σ 2 > 0 if q < rank(S), so this assumption implies no loss of practicality. There are three possible classes of solutions to equation A.2: 1. W = 0. This is shown later to be a minimum of the log-likelihood. 2. C = S, where the covariance model is exact, such as is discussed by Basilevsky (1994, pp. 361–363) and considered in section 2.3. In this unrealistic case of an exact covariance model, where the d − q smallest eigenvalues of S are identical and equal to σ 2 , W is identifiable since σ 2 I + WWT = S, ⇒ W = U(Λ − σ 2 I)1/2 R,
(A.3)
where U is a square matrix whose columns are the eigenvectors of S, with Λ the corresponding diagonal matrix of eigenvalues, and R is an arbitrary orthogonal (i.e., rotation) matrix. 3. SC−1 W = W, with W 6= 0 and C 6= S.
Mixtures of Probabilistic Principal Component Analyzers
469
We are interested in case 3 where C 6= S and the model covariance need not be equal to the sample covariance. First, we express the weight matrix W in terms of its singular value decomposition: W = ULVT ,
(A.4)
where U is a d×q matrix of orthonormal column vectors, L = diag(l1 , l2 , . . . , lq ) is the q × q diagonal matrix of singular values, and V is a q × q orthogonal matrix. Now, C−1 W = (σ 2 I + WWT )−1 W, = W(σ 2 I + WT W)−1 , = UL(σ 2 I + L2 )−1 VT .
(A.5)
Then at the stationary points, SC−1 W = W implies that SUL(σ 2 I + L2 )−1 VT = ULVT , SUL = U(σ 2 I + L2 )L.
⇒
(A.6)
For lj 6= 0, equation A.6 implies that if U = (u1 , u2 , . . . , uq ), then the corresponding column vector uj must be an eigenvector of S, with eigenvalue λj such that σ 2 + lj2 = λj , and so lj = (λj − σ 2 )1/2 .
(A.7)
For lj = 0, uj is arbitrary (and if all lj are zero, then we recover case 1). All potential solutions for W may thus be written as W = Uq (Kq − σ 2 I)1/2 R,
(A.8)
where Uq is a d × q matrix comprising q column eigenvectors of S, and Kq is a q × q diagonal matrix with elements: ½ kj =
λj , the corresponding eigenvalue to uj , or, σ 2,
(A.9)
where the latter case may be seen to be equivalent to lj = 0. Again, R is an arbitrary orthogonal matrix, equivalent to a rotation in the principal subspace. A.2 The Global Maximum of the Likelihood. The matrix Uq may contain any of the eigenvectors of S, so to identify those that maximize the
470
Michael E. Tipping and Christopher M. Bishop
likelihood, the expression for W in equation A.8 is substituted into the loglikelihood function (see equation 3.8) to give ½ q0 d X N 1 X d ln(2π) + L=− ln(λj ) + 2 λj 2 σ j=q0 +1 j=1 ¾ 0 2 0 + (d − q ) ln σ + q ,
(A.10)
where q0 is the number of nonzero lj , {λ1 , . . . , λq0 } are the eigenvalues corresponding to those retained in W, and {λq0 +1 , . . . , λd } are those discarded. Maximizing equation A.10 with respect to σ 2 gives σ2 =
d X 1 λj , d − q0 j=q0 +1
(A.11)
and so
( q0 d X N X 1 L=− ln(λj ) + (d − q0 ) ln λj 2 j=1 d − q0 j=q0 +1 ) + d ln(2π) + d .
(A.12)
Note that equation A.11 implies that σ 2 > 0 if rank(S) > q as stated earlier. We wish to find the maximum of equation A.12 with respect to the choice of eigenvectors/eigenvalues to retain in W, j ∈ {1, . . . , q0 }, and those to discard, j ∈ {q0 + 1, . . . , d}. By exploiting the constancy of the sum of all eigenvalues with respect to this choice, the condition for maximization of the likelihood can be expressed equivalently as minimization of the quantity d d X X 1 1 λj − ln(λj ), (A.13) E = ln 0 0 d − q j=q0 +1 d − q j=q0 +1 which conveniently depends on only the discarded values and is nonnegative (Jensen’s inequality). We consider minimization of E by first assuming that d − q0 discarded eigenvalues have been chosen arbitrarily and, by differentiation, consider how a single such value λk affects the value of E: 1 1 ∂E = Pd − . ∂λk (d − q0 )λk j=q0 +1 λj
(A.14)
From equation A.14, it can be seen that E(λk ) is convex and has a single minimum when λk is equal to the mean of the discarded eigenvalues (including
Mixtures of Probabilistic Principal Component Analyzers
471
itself). The eigenvalue λk can only take discrete values, but if we consider exchanging λk for some retained eigenvalue λj , j ∈ {1, . . . , q0 }, then if λj lies between λk and the current mean discarded eigenvalue, swapping λj and λk must decrease E. If we consider that the eigenvalues of S are ordered, for any combination of discarded eigenvalues that includes a gap occupied by a retained eigenvalue, there will always be a sequence of adjacent eigenvalues with a lower value of E. It follows that to minimize E, the discarded eigenvalues λq0 +1 , . . . , λd must be chosen to be adjacent among the ordered eigenvalues of S. This alone is not sufficient to show that the smallest eigenvalues must be discarded in order to maximize the likelihood. However, a further constraint is available from equation A.7, since lj = (λj − σ 2 )1/2 implies that there can be no real solution to the stationary equations of the log-likelihood if any retained eigenvalue λj < σ 2 . Since, from equation A.11, σ 2 is the average of the discarded eigenvalues, this condition would be violated if the smallest eigenvalue were not discarded. Now, combined with the previous result, this indicates that E must be minimized when λq0 +1 , . . . , λd are the smallest d − q0 eigenvalues and so L is maximized when λ1 , . . . , λq are the principal eigenvalues of S. It should also be noted that the log-likelihood L is maximized, with respect to q0 , when there are fewest terms in the sum in equation A.13 that occurs when q0 = q, and therefore no lj is zero. Furthermore, L is minimized when W = 0, which is equivalent to the case of q0 = 0. A.3 The Nature of Other Stationary Points. If stationary points represented by minor (nonprincipal) eigenvector solutions are stable maxima of the likelihood, then local maximization (via an EM algorithm, for example) is not guaranteed to find the principal eigenvectors. We may show, however, that minor eigenvector solutions are in fact saddle points on the likelihood surface. Consider a stationary point of the log-likelihood, given by equation A.8, b = Uq (Kq − σ 2 I)1/2 R, where Uq may contain q arbitrary eigenvectors of at W S and Kq contains either the corresponding eigenvalue or σ 2 . We examine the nature of this stationary point by considering a small perturbation of b + ²PR, where ² is an arbitrarily small, positive constant the form W = W and P is a d × q matrix of zeroes except for column W, which contains a discarded eigenvector uP not contained in Uq . By considering each potenb we may tial eigenvector uP individually applied to each column W of W, elucidate the nature of the stationary point by evaluating the inner product of the perturbation with the gradient at W (where we treat the parameter matrix W or its derivative as a single column vector). If this inner product is negative for all possible perturbations, then the stationary point will be stable and represent a (local) maximum. b + ²PR, then from So defining G = (∂ L/∂W)/N evaluated at W = W
472
Michael E. Tipping and Christopher M. Bishop
equation A.1, CG = SC−1 W − W, = SW(σ 2 I + WT W)−1 − W, b TW b + ² 2 RT PT PR)−1 − W, = SW(σ 2 I + W
(A.15)
b = 0. Ignoring the term in ² 2 then gives: since PT W b −1 − (W b T W) b + ²PR), b + ²PR)(σ 2 I + W CG = S(W 2 T b −1 b = ²SPR(σ I + W W) − ²PR,
(A.16)
b −W b = 0 at the stationary point. Then substituting b 2I + W b T W) since SW(σ b TW b gives σ 2 I + W b = RT Kq R, and so for W CG = ²SPR(RT K−1 q R) − ²PR, ⇒ G = ²C−1 P(ΛK−1 q − I)R,
(A.17)
where Λ is a d × d matrix of zeros, except for the Wth diagonal element, which contains the eigenvalue corresponding to uP , such that (Λ)WW = λP . Then the sign of the inner product of the gradient G and the perturbation ²PR is given by ´o n h io n ³ T −1 , sign tr GT PR = sign ²tr RT (ΛK−1 q − I)P C PR o n = sign (λP /kW − 1)uTP C−1 uP , ª © = sign λP /kW − 1 ,
(A.18)
since C−1 is positive definite and where kW is the Wth diagonal element value in Kq , and thus in the corresponding position to λP in Λ. When kW = λW , the expression given by equation A.18 is negative (and the maximum a stable b must be a saddle point. one) if λP < λW . For λP > λW , W In the case that kW = σ 2 , the stationary point will generally not be stable since, from equation A.11, σ 2 is the average of d−q0 eigenvalues, and so λP > σ 2 for at least one of those eigenvalues, except when all those eigenvalues are identical. Such a case is considered shortly. From this, by considering all possible perturbations P, it can be seen that the only stable maximum occurs when W comprises the q principal eigenvectors, for which λP < λW , ∀P 6= W. A.4 Equality of Eigenvalues. Equality of any of the q principal eigenvalues does not affect the maximum likelihood estimates. However, in terms of conventional PCA, consideration should be given to the instance when all the d − q minor (discarded) eigenvalue(s) are equal and identical to at
Mixtures of Probabilistic Principal Component Analyzers
473
least one retained eigenvalue. (In practice, particularly in the case of sample covariance matrices, this is unlikely.) To illustrate, consider the example of extracting two components from data with a covariance matrix possessing eigenvalues λ1 , λ2 and λ2 , and λ1 > λ2 . In this case, the second principal axis is not uniquely defined within the minor subspace. The spherical noise distribution defined by σ 2 = λ2 , in addition to explaining the residual variance, can also optimally explain the second principal component. Because λ2 = σ 2 , l2 in equation A.7 is zero, and W effectively comprises only a single vector. The combination of this single vector and the noise distribution still represents the maximum of the likelihood, but no second eigenvector is defined. A.5 An EM Algorithm for PPCA. In the EM approach to PPCA, we consider the latent variables {xn } to be “missing” data. If their values were known, estimation of W would be straightforward from equation 2.2 by applying standard least-squares techniques. However, for a given tn , we do not know the value of xn that generated it, but we do know the joint distribution of the observed and latent variables, p(t, x), and we can calculate the expectation of the corresponding complete-data log-likelihood. In the E-step of the EM algorithm, this expectation, calculated with respect to the posterior distribution of xn given the observed tn , is computed. In the Me and e step, new parameter values W σ 2 are determined that maximize the expected complete-data log-likelihood, and this is guaranteed to increase Q the likelihood of interest, n p(tn ), unless it is already at a local maximum (Dempster, Laird, & Rubin, 1977). The complete-data log-likelihood is given by:
LC =
N X
© ª ln p(tn , xn ) ,
(A.19)
n=1
where, in PPCA, from equations 3.1 and 3.4, ¾ ½ ktn − Wxn − µk2 (2π )−q/2 p(tn , xn ) = (2πσ 2 )−d/2 exp − 2σ 2 ½ ¾ 1 × exp − xTn xn . 2
(A.20)
In the E-step, we take the expectation with respect to the distributions p(xn |tn , W, σ 2 ): hLC i = −
N ½ X d n=1
´ 1 ³ 1 ln σ 2 + tr hxn xTn i + 2 ktn − µk2 2 2 2σ ³ ´¾ 1 1 T T T T − 2 hxn i W (tn − µ)+ 2 tr W Whxn xn i , σ 2σ
(A.21)
474
Michael E. Tipping and Christopher M. Bishop
where we have omitted terms independent of the model parameters and hxn i = M−1 WT (tn − µ),
(A.22)
hxn xTn i = σ 2 M−1 + hxn ihxn iT ,
(A.23)
with M = (σ 2 I + WT W). Note that these statistics are computed using the current (fixed) values of the parameters and that equation A.22 is simply the posterior mean from equation 3.6. Equation A.23 follows from this in conjunction with the posterior covariance of equation 3.7. In the M-step, hLC i is maximized with respect to W and σ 2 by differentiating equation A.21 and setting the derivatives to zero. This gives: e = W
" X
(tn −
n
e σ2 =
#" #−1 X T hxn xn i
µ)hxTn i
(A.24)
n
N ½ 1 X e T (tn − µ) ktn − µk2 − 2hxTn iW Nd n=1 ´¾ ³ e e TW + tr hxn xTn iW
(A.25)
To maximize the likelihood then, the sufficient statistics of the posterior distributions are calculated from the E-step equations A.22 and A.23, followed by the maximizing M-step equations (A.24 and A.25). These four equations are iterated in sequence until the algorithm is judged to have converged. We may gain considerable insight into the operation of equations A.24 and A.25 by substituting for hxn i and hxn xTn i from A.22 and A.23. Taking care not to confuse new and old parameters, some further manipulation leads to both the E-step and M-step’s being combined and rewritten as: e = SW(σ 2 I + M−1 WT SW)−1 , and W ´ 1 ³ eT , e σ 2 = tr S − SWM−1 W d
(A.26) (A.27)
where S is again given by S=
N 1 X (tn − µ)(tn − µ)T . N n=1
(A.28)
Note that the first instance of W in equation A.27 is the old value of the e is the new value calculated from equaweights, while the second instance W tion A.26. Equations A.26, A.27, and A.28 indicate that the data enter into the EM formulation only through its covariance matrix S, as we would expect.
Mixtures of Probabilistic Principal Component Analyzers
475
Although it is algebraically convenient to express the EM algorithm in terms of S, care should be exercised in any implementation. When q ¿ d, it is possible to obtain considerable computational savings by not explicitly evaluating the covariance matrix, computation of which is O(Nd2 ). This is because inspection of equations A.24 and A.25 indicates that complexity is only O(Ndq), and is reflected in equations A.26 and A.27 by the fact that S appears only within the terms SW and tr (S), which may be computed with O(Ndq) P and O(Nd) ©complexity, ªrespectively. That is, SW should be computed as n (tn − µ) (tn − µ)T W , as that form is more efficient than ©P ª T W, which is equivalent to finding S explicitly. Hown (tn − µ)(tn − µ) ever, because S need only be computed once in the single model case and the EM algorithm is iterative, potential efficiency gains depend on the number of iterations required to obtain the desired accuracy of solution, as well as the ratio of d to q. For example, in our implementation of the model using q = 2 for data visualization, we found that an iterative approach could be more efficient for d > 20. A.6 Rotational Ambiguity. If W is determined by the above algorithm, or any other iterative method that maximizes the likelihood (see equation 3.8), then at convergence, WML = Uq (Λq − σ 2 I)1/2 R. If it is desired to find the true principal axes Uq (and not just the principal subspace) then the arbitrary rotation matrix R presents difficulty. This rotational ambiguity also exists in factor analysis, as well as in certain iterative PCA algorithms, where it is usually not possible to determine the actual principal axes if R 6= I (although there are algorithms where the constraint R = I is imposed and the axes may be found). However, in probabilistic PCA, R may actually be found since WTML WML = RT (Λq − σ 2 I)R
(A.29)
implies that RT may be computed as the matrix of eigenvectors of the q × q matrix WTML WML . Hence, both Uq and Λq may be found by inverting the rotation followed by normalization of WML . That the rotational ambiguity may be resolved in PPCA is a consequence of the scaling of the eigenvectors by (Λq − σ 2 I)1/2 prior to rotation by R. Without this scaling, WTML WML = I, and the corresponding eigenvectors remain ambiguous. Also, note that while finding the eigenvectors of S directly requires O(d3 ) operations, to obtain them from WML in this way requires only O(q3 ). Appendix B: Optimal Least-Squares Reconstruction One of the motivations for adopting PCA in many applications, notably in data compression, is the property of optimal linear least-squares reconstruction. That is, for all orthogonal projections x = AT t of the data, the
476
Michael E. Tipping and Christopher M. Bishop
least-squares reconstruction error, E2rec =
N 1 X ktn − BAT tn k2 , N n=1
(B.1)
is minimized when the columns of A span the principal subspace of the data covariance matrix, and B = A. (For simplification, and without loss of generality, we assume here that the data has zero mean.) We can similarly obtain this property from our probabilistic formalism, without the need to determine the exact orthogonal projection W, by finding the optimal reconstruction of the posterior mean vectors hxn i. To do this we simply minimize E2rec =
N 1 X ktn − Bhxn ik2 , N n=1
(B.2)
over the reconstruction matrix B, which is equivalent to a linear regression problem giving B = SW(WT SW)−1 M,
(B.3)
where we have substituted for hxn i from equation A.22. In general, the resulting projection Bhxn i of tn is not orthogonal, except in the maximum likelihood case, where W = WML = Uq (Λq − σ 2 I)1/2 R, and the optimal reconstructing matrix becomes BML = W(WT W)−1 M,
(B.4)
and so ˆtn = W(WT W)−1 Mhxn i, = W(WT W)−1 WT tn ,
(B.5) (B.6)
which is the expected orthogonal projection. The implication is thus that in the data compression context, at the maximum likelihood solution, the variables hxn i can be transmitted down the channel and the original data vectors optimally reconstructed using equation B.5 given the parameters W and σ 2 . Substituting for B in equation B.2 gives E2rec = (d − q)σ 2 , and the noise term σ 2 thus represents the expected squared reconstruction error per “lost” dimension. Appendix C: EM for Mixtures of Probabilistic PCA In a mixture of probabilistic principal component analyzers, we must fit a mixture of latent variable models in which the overall model distribution
Mixtures of Probabilistic Principal Component Analyzers
477
takes the form p(t) =
M X
πi p(t|i),
(C.1)
i=1
where p(t|i) is a single probabilistic PCA model and πi is the corresponding mixing proportion. The parameters for this mixture model can be determined by an extension of the EM algorithm. We begin by considering the standard form that the EM algorithm would take for this model and highlight a number of limitations. We then show that a two-stage form of EM leads to a more efficient algorithm. We first note that in addition to a set of xni for each model i, the missing data include variables zni labeling which model is responsible for generating each data point tn . At this point we can derive a standard EM algorithm by considering the corresponding complete-data log-likelihood, which takes the form
LC =
M N X X
zni ln{πi p(tn , xni )}.
(C.2)
n=1 i=1
Starting with “old” values for the parameters πi , µi , Wi , and σi2 , we first evaluate the posterior probabilities Rni using equation 4.3 and similarly evaluate the expectations hxni i and hxni xTni i: T hxni i = M−1 i Wi (tn − µi ),
T hxni xTni i = σi2 M−1 i + hxni ihxni i ,
(C.3) (C.4)
with Mi = σi2 I + WTi Wi . Then we take the expectation of LC with respect to these posterior distributions to obtain hLC i =
M N X X n=1 i=1
½ ´ d 1 ³ Rni ln πi − ln σi2 − tr hxni xTni i 2 2 1 1 ktni − µi k2 + 2 hxni iT WTi (tn − µi ) 2σi2 σi ³ ´¾ 1 T T tr W W hx x i , − i ni ni i 2σi2 −
(C.5)
where h·i denotes the expectation with respect to the posterior distributions of both xni and zni and terms independent of the model parameters have been omitted. The M-step then involves maximizing equation C.5 with respect to πi , µi , σi2 , and Wi to obtain “new” values for these parameters. The maximization with respect to πi must take account of the constraint that
478
Michael E. Tipping and Christopher M. Bishop
P
i πi = 1. This can be achieved with the use of a Lagrange multiplier λ (see Bishop, 1995) and maximizing
hLC i + λ
à M X
! πi − 1 .
(C.6)
i=1
Together with the results of maximizing equation C.5 with respect to the remaining parameters, this gives the following M-step equations: 1X Rni N n P e i hxni i) Rni (tni − W P ei = n µ n Rni " #−1 #" X X T T e ei )hxni i Rni (tn − µ Rni hxni xni i Wi = e πi =
n
e σi2 =
d
½X
1 P
n Rni
n
+
(C.7) (C.8) (C.9)
n
ei k2 − 2 Rni ktn − µ
X
e T (tn − µ ei ) Rni hxni iT W i
n
³ Rni tr
X
ei e TW hxni xTni iW i
´¾ (C.10)
n
where the symboledenotes “new” quantities that may be adjusted in the Me i , given by equations C.8 ei and W step. Note that the M-step equations for µ and C.9, are coupled, and so further (albeit straightforward) manipulation is required to obtain explicit solutions. In fact, simplification of the M-step equations, along with improved speed of convergence, is possible if we adopt a two-stage EM procedure as follows. The likelihood function we wish to maximize is given by
L=
N X
ln
( M X
n=1
) πi p(tn |i) .
(C.11)
i=1
Regarding the component labels zni as missing data, and ignoring the presence of the latent x variables for now, we can consider the corresponding expected complete-data log-likelihood given by bC = L
M N X X
© ª Rni ln πi p(tn |i) ,
(C.12)
n=1 i=1
where Rni represent the posterior probabilities (corresponding to the expected values of zni ) and are given by equation 4.2. Maximization of equation C.12 with respect to πi , again using a Lagrange multiplier, gives the
Mixtures of Probabilistic Principal Component Analyzers
479
M-step equation (4.4). Similarly, maximization of equation C.12 with respect to µi gives equation 4.5. This is the first stage of the combined EM procedure. bC , In order to update Wi and σi2 , we seek only to increase the value of L and not actually to maximize it. This corresponds to the generalized EM bC as our likelihood of (or GEM) algorithm. We do this by considering L interest and, introducing the missing xni variables, perform one cycle of the EM algorithm, now with respect to the parameters Wi and σi2 . This second bC , and therefore L as desired. stage is guaranteed to increase L ei The advantages of this approach are twofold. First, the new values µ calculated in the first stage are used to compute the sufficient statistics of the posterior distribution of xni in the second stage using equations C.3 and C.4. By using updated values of µi in computing these statistics, this leads to improved convergence speed. A second advantage is that for the second stage of the EM algorithm, there is a considerable simplification of the M-step updates, since when ei (and not µi ) equation C.5 is expanded for hxni i and hxni xTni i, only terms in µ appear. By inspection of equation C.5, we see that the expected completedata log-likelihood now takes the form hLC i =
N X M X n=1 i=1
½ ´ d 1 ³ Rni ln e πi − ln σi2 − tr hxni xTni i 2 2 1 1 ei k2 + 2 hxTni iWTi (tn − µ ei ) ktni − µ 2 2σi σi ³ ´¾ 1 T T tr W W hx x i . − i ni i ni 2σi2 −
(C.13)
Now when we maximize equation C.13 with respect to Wi and σi2 (keeping ei fixed), we obtain the much simplified M-step equations: µ e i = Si Wi (σ 2 I + M−1 WT Si Wi )−1 , W i i i ´ 1 ³ 2 −1 e T e σi = tr Si − Si Wi Mi Wi , d
(C.14) (C.15)
where Si =
N 1 X ei )(tn − µ ei )T . Rni (tn − µ e πi N n=1
(C.16)
Iteration of equations 4.3 through 4.5 followed by equations C.14 and C.15 in sequence is guaranteed to find a local maximum of the likelihood (see equation 4.1). Comparison of equations C.14 and C.15 with equations A.26 and A.27 shows that the updates for the mixture case are identical to those of the
480
Michael E. Tipping and Christopher M. Bishop
single PPCA model, given that the local responsibility-weighted covariance matrix Si is substituted for the global covariance matrix S. Thus, at stationary points, each weight matrix Wi contains the (scaled and rotated) eigenvectors of its respective Si , the local covariance matrix. Each submodel is then performing a local PCA, where each data point is weighted by the responsibility of that submodel for its generation, and a soft partitioning, similar to that introduced by Hinton et al. (1997), is automatically effected. Given the established results for the single PPCA model, there is no need to use the iterative updates (see equations C.14 and C.15) since Wi and σi2 may be determined by eigendecomposition of Si , and the likelihood must still increase unless at a maximum. However, as discussed in appendix A.5, the iterative EM scheme may offer computational advantages, particularly for q ¿ d. In such a case, the iterative approach of equations P C.14 and C.15© can be used,ª taking care to evaluate Si Wi efficiently as n Rni (tn − ei )T Wi . In the mixture case, unlike for the single model, Si must e µi ) (tn − µ be recomputed at each iteration of the EM algorithm, as the responsibilities Rni will change. As a final computational note, it might appear that the necessary calculation of p(t|i) would require inversion of the d × d matrix C, an O(d3 ) operation. However, (σ 2 I + WWT )−1 = {I − W(σ 2 I + WT W)−1 WT }/σ 2 and so C−1 may be computed using the already calculated q × q matrix M−1 . Acknowledgments This work was supported by EPSRC contract GR/K51808: Neural Networks for Visualization of High Dimensional Data, at Aston University. We thank Michael Revow for supplying the handwritten digit data in its processed form. References Anderson, T. W. (1963). Asymptotic theory for principal component analysis. Annals of Mathematical Statistics, 34, 122–148. Anderson, T. W., & Rubin, H. (1956). Statistical inference in factor analysis. In J. Neyman (Ed.), Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability (Vol. 5, pp. 111–150). Berkeley: University of California, Berkeley. Bartholomew, D. J. (1987). Latent variable models and factor analysis. London: Charles Griffin & Co. Ltd. Basilevsky, A. (1994). Statistical factor analysis and related methods. New York: Wiley. Bishop, C. M. (1995). Neural networks for pattern recognition. Oxford: Clarendon Press. Bishop, C. M., Svens´en, M., & Williams, C. K. I. (1998). GTM: The generative topographic mapping. Neural Computation, 10(1), 215–234.
Mixtures of Probabilistic Principal Component Analyzers
481
Bishop, C. M., & Tipping, M. E. (1998). A hierarchical latent variable model for data visualization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(3), 281–293. Bregler, C., & Omohundro, S. M. (1995). Nonlinear image interpolation using manifold learning. In G. Tesauro, D. S. Touretzky, & T. K. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 973–980). Cambridge, MA: MIT Press. Broomhead, D. S., Indik, R., Newell, A. C., & Rand, D. A. (1991). Local adaptive Galerkin bases for large-dimensional dynamical systems. Nonlinearity, 4(1), 159–197. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, B39(1), 1–38. Dony, R. D., & Haykin, S. (1995). Optimally adaptive transform coding. IEEE Transactions on Image Processing, 4(10), 1358–1370. Hastie, T., & Stuetzle, W. (1989). Principal curves. Journal of the American Statistical Association, 84, 502–516. Hinton, G. E., Dayan, P., & Revow, M. (1997). Modelling the manifolds of images of handwritten digits. IEEE Transactions on Neural Networks, 8(1), 65–74. Hinton, G. E., Revow, M., & Dayan, P. (1995). Recognizing handwritten digits using mixtures of linear models. In G. Tesauro, D. S. Touretzky, & T. K. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 1015–1022). Cambridge, MA: MIT Press. Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24, 417–441. Hull, J. J. (1994). A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 550–554. Japkowicz, N., Myers, C., & Gluck, M. (1995). A novelty detection approach to classification. In Proceedings of the Fourteenth International Conference on Artificial Intelligence (pp. 518–523). Jolliffe, I. T. (1986). Principal component analysis. New York: Springer-Verlag. Jordan, M. I., & Jacobs, R. A. (1994). Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6(2), 181–214. Kambhatla, N. (1995). Local models and gaussian mixture models for statistical data processing. Unpublished doctoral dissertation, Oregon Graduate Institute, Center for Spoken Language Understanding. Kambhatla, N., & Leen, T. K. (1997). Dimension reduction by local principal component analysis. Neural Computation, 9(7), 1493–1516. Kramer, M. A. (1991). Nonlinear principal component analysis using autoassociative neural networks. AIChE Journal, 37(2), 233–243. Krzanowski, W. J., & Marriott, F. H. C. (1994). Multivariate analysis part 2: Classification, Covariance structures and repeated measurements. London: Edward Arnold. Lawley, D. N. (1953). A modified method of estimation in factor analysis and some large sample results. In Uppsala Symposium on Psychological Factor Analysis. Nordisk Psykologi Monograph Series (pp. 35–42). Uppsala: Almqvist and Wiksell.
482
Michael E. Tipping and Christopher M. Bishop
Oja, E. (1983). Subspace methods of pattern recognition. New York: Wiley. Ormoneit, D., & Tresp, V. (1996). Improved gaussian mixture density estimates using Bayesian penalty terms and network averaging. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 542–548). Cambridge, MA: MIT Press. Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. London, Edinburgh and Dublin Philosophical Magazine and Journal of Science, Sixth Series, 2, 559–572. Petsche, T., Marcantonio, A., Darken, C., Hanson, S. J., Kuhn, G. M., & Santoso, I. (1996). A neural network autoassociator for induction motor failure prediction. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 924–930). Cambridge, MA: MIT Press. Rao, C. R. (1955). Estimation and tests of significance in factor analysis. Psychometrika, 20, 93–111. Rubin, D. B., & Thayer, D. T. (1982). EM algorithms for ML factor analysis. Psychometrika, 47(1), 69–76. Tibshirani, R. (1992). Principal curves revisited. Statistics and Computing, 2, 183– 190. Tipping, M. E., & Bishop, C. M. (1997). Mixtures of principal component analysers. In Proceedings of the IEE Fifth International Conference on Artificial Neural Networks, Cambridge (pp. 13–18). London: IEE. Titterington, D. M., Smith, A. F. M., & Makov, U. E. (1985). The statistical analysis of finite mixture distributions. New York: Wiley. Webb, A. R. (1996). An approach to nonlinear principal components analysis using radially symmetrical kernel functions. Statistics and Computing, 6(2), 159–168. Received June 19, 1997; accepted May 19, 1998.
LETTER
Communicated by Robert Jacobs
Boosted Mixture of Experts: An Ensemble Learning Scheme Ran Avnimelech Nathan Intrator Department of Computer Science, Sackler Faculty of Exact Sciences, Tel-Aviv University, Tel-Aviv, Israel
We present a new supervised learning procedure for ensemble machines, in which outputs of predictors, trained on different distributions, are combined by a dynamic classifier combination model. This procedure may be viewed as either a version of mixture of experts (Jacobs, Jordan, Nowlan, & Hinton, 1991), applied to classification, or a variant of the boosting algorithm (Schapire, 1990). As a variant of the mixture of experts, it can be made appropriate for general classification and regression problems by initializing the partition of the data set to different experts in a boostlike manner. If viewed as a variant of the boosting algorithm, its main gain is the use of a dynamic combination model for the outputs of the networks. Results are demonstrated on a synthetic example and a digit recognition task from the NIST database and compared with classical ensemble approaches. 1 Introduction The mixture-of-experts approach has great potential for improving performance in machine learning. The improved classification and regression performance achieved by using an ensemble of networks rather than a single net for classification and regression tasks is well established (Hansen & Salamon, 1990; Wolpert, 1992; Breiman, 1996c; Perrone & Cooper, 1993; Raviv & Intrator, 1996). Earlier work focused on voting schemes—majority and plurality—but in later studies, averaging of the outputs was usually found to be superior. Advanced methods for combining the output of different classifiers are suggested in Ho, Hull, and Srihari (1994). Logistic regression (perceptron) is applied on the output of the classifiers to achieve better results than simple averaging; furthermore, the static combination of experts is replaced by a dynamic model (DCS), so that only one of several logistic regression functions is chosen, according to the input or to the classifier outputs. Generally there are two approaches to combining outputs of different classifiers: selection, or choosing the locally best classifier, and averaging, or reducing the variance by combining outputs that are not fully correlated. DCS and other methods combine these approaches by using a dynamic weighted average. Neural Computation 11, 483–497 (1999)
c 1999 Massachusetts Institute of Technology °
484
Ran Avnimelech and Nathan Intrator
Stacking is another framework for combining estimators that uses a nonsymmetric combination (Wolpert, 1992; Breiman, 1996c). The principle is to use several levels of learners, in a manner that is basically an extension of choosing a learner by cross-validation. To avoid training the combination level on overfit outputs of the lower-level learners, each input pattern to the combination learner is extracted by copies of the learners trained on the data, excluding that pattern. The algorithm is applicable for either multiple learners or a single learner. The popular form of stacking uses two levels with a linear combination model, possibly with constrained coefficients (e.g., nonnegative, sum to 1). Other methods use dynamic linear combination models, using a confidence measure of the ensemble members regarding each pattern. Different measures of the confidence of each predictor can be used for determining the relative contribution of each expert (Tresp & Taniguchi, 1995; Shimshoni & Intrator, 1996). All of these algorithms train the individual classifiers independently for the same goal. More specifically, the different parts of the training set that are used to train individual classifiers are all drawn from the same distribution. This holds when different types of classifiers are used, in cross-validation (Meir, 1995; Krogh & Vedelsby, 1995), or when different noisy bootstrap copies are used (Raviv & Intrator, 1996). A different approach is training the classifiers on different parts of the training set, partitioned in a manner such that their distributions differ. Such an approach, which is presented here, combines two algorithms: boosting and mixture of experts. Sections 2 and 3 describe the boosting and adaptive mixture-of-experts algorithms. These algorithms are compared in section 4, and various ways to combine them are suggested in section 5. Following this discussion we present in section 6 the basic and advanced versions of the new algorithm. The empirical evaluation of the algorithm on a demonstration problem and on a character recognition task from the NIST database is reported in section 7. 2 Theory of Boosting The boosting algorithm can improve the performance of learning machines (Schapire, 1990). Its theoretic basis relies on a proof of the equivalence of the strong and weak PAC (probably approximately correct) learning models. In the standard PAC model, for any distribution of patterns and for arbitrary small δ and ², the learner must be able to produce a hypothesis about the underlying concept, with an error rate of at most ² with a probability of at least (1 − δ). The weak PAC model, however, requires just ² < 1/2—slightly better than a random guess on this two-class model. Schapire proved the equivalence of the two models by proposing a technique for converting any weak learning algorithm (on any given distribution) to a strong learning algorithm. He termed this provably correct tech-
Boosted Mixture of Experts
485
nique boosting. The basis of the technique is creating different distributions on which different subhypotheses are trained. Schapire has proved that if three such weak subhypotheses, which have an error rate of α < 1/2 (on the respective distributions), are combined, the resulting ensemble hypothesis will have an error rate of 3α 2 − 2α 3 , which is smaller than α. Schapire suggested hierarchical combinations of classifiers, such that an arbitrarily low error rate can be achieved. A procedure for creating appropriate distributions is the following: A classifier is trained on the original distribution. Fifty percent of the training set for the second classifier are patterns misclassified by the first classifier, and 50% are patterns correctly classified by it (no change in the internal distribution of each of these two groups). The third classifier is designed to break ties. Its training set contains only patterns on which the first two classifiers disagree. Real-world machine learning tasks do not necessarily match the weak PAC model, and even if they did, the assured performance for worst-case scenario would not necessarily be higher than the practically achieved performance of simple classifiers. Still, boosting proved to be not just a theoretical technique, but also a practical tool for enhancing performance. Drucker, Schapire, and Simard (1993) demonstrated its advantage over a combination of independently trained classifiers (parallel machine) on a handwritten recognition task. Recently, boosting achieved an extremely low error rate on the same problem (Bottou et al., 1994). Various improvements have been made to the original boosting algorithm. Freund (1990) suggested using a simpler structure for combining many subhypotheses: instead of having a tree of majority gates, all subhypotheses are presented to one majority gate. AdaBoost (Freund & Schapire, 1995) is a more advanced algorithm, in which each pattern is assigned a different probability to appear in the training set presented to the new learner. This version also prefers a flat structure for combining the classifiers rather than a hierarchical one. Another idea mentioned within the AdaBoost framework is the use of a weighted combination of the individual classifiers. Recently several applications of AdaBoost have been reported (Breiman, 1996b; Schwenk & Bengio, 1997). Breiman regards boosting as one example of an algorithm performing adaptive resampling of the training set and suggests other such algorithms. He applied these algorithms to decision trees (CARTs) on various data. Schwenk and Bengio applied Adaboost to multilayer perceptrons (MLPs) and autoencoder-based classifiers (“diabolo networks”) on character recognition tasks. 3 The Mixture-of-Experts Learning Procedure The adaptive mixture of local experts (Jacobs et al., 1991) is a learning procedure that achieves improved performance in certain problems by assigning different subtasks to different learners. Its basic idea is concurrently to train
486
Ran Avnimelech and Nathan Intrator
several expert classifiers (or regression estimators) and a gating function. The gating function assigns probability to each of the experts based on the current input. In the training stage, this value states the probability of a pattern’s appearing in an expert’s training set. In the test stage, it defines the relative contribution of each expert to the ensemble. The training attempts to achieve two goals: (1) for a given expert, find the optimal gating function, and (2) for a given gating function, train each expert to achieve maximal performance on the distribution assigned to it by the gating function. This decomposition of the learning task motivates an expectation-maximization version of the algorithm, though simultaneous training was also used. Much emphasis is given in this framework to making the experts local, which is a key to improving performance over ensembles of networks trained on similar distributions. A basic level of locality is achieved by targeting each expert for maximal performance on its distribution instead of having it compensate for errors of other experts. Further localization is achieved by giving higher learning rates to the better-performing expert on each pattern. This idea was later extended into a tree structure termed hierarchical mixture of experts (HME), in which experts may be built from lower-level experts and gating functions (Jordan & Jacobs, 1992). In later work, the EM algorithm was used for training the HME (Jordan & Jacobs, 1994). Waterhouse and Robinson (1996) describe how to grow these recursive learning machines gradually. The mixture-of-experts procedure achieves superior generalization and fast learning when the learning task corresponds to different subtasks for distinct portions of the input space. The mixture-of-experts algorithm differs from other ensemble algorithms in the relation between the combination model and the basic learners (and our algorithm follows it). Most ensemble learning algorithms, such as stacking, first train the basic predictors (or use existing predictors) and then try to tune the combination model. The mixture-of-experts algorithm trains the combination model simultaneously with the basic learners, and the current model determines the data sets provided to each learner for its further training. 4 Comparison of the Two Algorithms Boosting and mixture of experts were developed for different types of problems and thus have different advantages and weaknesses. Any attempt to combine principles from both should address their limitations and overcome them by combining elements of the other method. The mixture of experts is suitable when the patterns can be naturally divided into simpler (homogeneous) subsets, and the learning task in each of these subsets is not as difficult as the original one. However, real-world problems may not exhibit this property, and, furthermore, even when such a partition exists, the required gating function may be complex and the initial
Boosted Mixture of Experts
487
stage of localizing the experts has a chicken-and-egg nature. In boosting, the distributions are selected to encourage each classifier to become an expert on patterns on which the previous classifiers err or disagree1 —difficult patterns—while maintaining a reasonably good performance on easier patterns. The two main advantages of the mixture of experts are localization of the different experts and use of a dynamic model for combining the outputs. In boosting, the first classifier is trained on all patterns, and the localization criterion for the distributions presented to the two other classifiers is the level of difficulty of the patterns as measured by classification performance. The limitation of this criterion is that it cannot be applied to unlabeled data, therefore disabling the use of a dynamic model based on a similar criterion. 5 Combining Boosting and HME Algorithms There are several approaches for combining features of boosting and mixture of experts: • Improved boosting. Adding a dynamic model for combining the outputs of the classifiers. (This feature is not unique to mixture of experts.) • Initialized mixture of experts. The main boosting feature one would like to introduce to the mixture-of-experts framework is the ability to initialize a split of the training set to different experts. • Multilevel approach. Using a mixture-of-experts classifier as the second or third boosting classifier can solve two problems: The difficult patterns may be more easily partitioned to subgroups, while the second and third boosting classifiers usually handle a more difficult problem from the original one. This approach incorporates classifier selection and classifier combination. Waterhouse and Cook (1997) have attempted to combine boosting with the mixture of experts using the first two approaches. They report that using a dynamic model for combining boost-trained networks achieved improved performance versus simple addition. They also report that the mixture of experts was best when bootstrapped from boosted networks (bootstrapping from simple ensemble was also superior to starting from random weights). 6 The Boosted Mixture of Experts The work presented here attempts to design a new algorithm that applies principles of both boosting and the mixture of experts and has high performance on classification or regression problems. The proposed boostedmixture-of-experts (BME) algorithm may be considered either as a boost1 More precisely, patterns on which the output may have maximal influence on the ensemble’s classification.
488
Ran Avnimelech and Nathan Intrator
wise initialized mixture of experts or as a variant of boosting that uses a dynamic model for combining output of the classifiers. The main boosting feature we want to include in our scheme is the ability to initialize a split of the training set to different experts. This split is based on a difficulty criterion. In boosting, this difficulty criterion is the errors of the first classifier or the disagreement between the first two classifiers. We prefer using a confidence measure rather than errors as our difficulty criterion. This has several advantages: the size of the difficult set is more flexible (a flexible error-oriented criterion is actually error plus confidence), it focuses on the patterns that could be classified correctly, and it avoids focusing on mislabels. This also enables using other confidence-oriented methods. (Such an approach is actually used for constructing the training set of the third classifier in boosting.) Our method includes an important component that boosting lacks: a dynamic model for combining the outputs of the classifiers. This requires a method for assigning each of the unlabeled patterns to the best-fitting classifier (or weighted combination). We follow the mixture-of-experts scheme and use the same gating function used for partitioning the data between the experts during training as the gating function for combining the outputs. Instead of training a separate gating function, we use a confidence measure, which is available for unlabeled patterns too. 6.1 The Basic Algorithm. The algorithm is designed for an arbitrary number of experts as the ensemble is constructed gradually by adding a new expert and repartitioning the data. The experts used in our work are neural nets, though any classifier with a good confidence measure is appropriate. The confidence measure is a key to achieving improved performance, and the flexibility in choosing it extends the range of applications of the algorithm. Basically, the algorithm trains several learners on different (possibly overlapping) portions of the data. The confidence measure—Ci (x) = C(oi (x))— is a scalar function of the basic learner’s output vector, which is used as a gating function. It determines the probability of patterns to be assigned to the data set of any learner; thus, these training sets may change as the learners evolve and their output vectors change. In addition to the confidence, the gating may be influenced from the basic reliability of each learner: gi (x) = wi · Ci (x). The reliability may be calculated by finding the optimal weighted average of the (output*confidence) of each classifier, and its value changes as the learners evolve. The output of this gating function is also used in the dynamic combination model as the coefficient assigned to each predictor for this pattern. The confidence measure may be based on specifics of the predictor used. For an MLP performing classification, with continuous-valued output, it may be some function of the output vector. The confidence should increase as the highest output is higher and decrease as any of the other outputs
Boosted Mixture of Experts
489
is higher. Other confidence measures, reported in machine learning literature, may also be used. Tresp and Taniguchi (1995) use various confidence measures of different predictors, in their combination model. One they use is the variance of the predictor as it is measured by the local sensitivity to changes in weights. Another approach they mention is assuming that the different predictors were trained on different data sets (e.g., American versus European digit data), and a hidden input indicates the set to which a pattern belongs. Estimating that value may be used to extract confidence information. Tresp and Taniguchi also suggest a unified approach, of which these two methods are extreme cases. Shimshoni and Intrator (1996) used base-level ensembles of several similar estimators as the different experts. The variance within each base-level ensemble indicates its confidence. A monotone function can be applied to the confidence measure to determine whether a soft or a hard partition of the data is to be used. The confidence measure we used on a multiclass classification task was based on the difference between the two highest outputs. This is the network’s estimate of its confidence margin and is a “natural” confidence measure provided by the MLP. We found that in order to encourage good localization, it was better to apply some power higher than 1 to the basic confidence measure. With a continuous-valued output, with rankings {R}, the confidence of the ith expert is Ci (x) = [OiR1 (x) − OiR2 (x)]n . The algorithm for constructing a BME consists of several procedures: • A procedure for training a single classifier on a given training set (we used a variant of BP). • A procedure for adding a classifier to an existing ensemble—assigning a training set for its initial training. We took a predefined portion from the training set of each of the experts, consisting of the patterns on which it was less confident. • A refining procedure. Repartition the data according to the current confidence level of each expert on each pattern. This can be done deterministically, by assigning each pattern to the most confident expert, or stochastically, by which the probability of assigning a pattern to a certain expert is proportional to its confidence (we used the stochastic version). The following algorithm describes how these different components fit into the constructive algorithm for creating a BME: Algorithm. 1. Train the first expert on all the training set. 2. Assign the patterns on which the current experts are not confident to the initial training set of the new expert and train it. 3. Refining stage: for i=1:N
490
Ran Avnimelech and Nathan Intrator
• Partition the data according to the confidence of each expert on each pattern. • Train each expert on its training set. 4. If more experts are required, return to step 2. Once the experts are trained, they may be used as an ensemble. The classifier combination model is based on the same gating function used for the localization of the experts. The exact choice of the gating function—both the confidence measure and the applied function—defines a specific variant of this algorithm. This gives the algorithm its flexibility and enables further improvement by handcrafting a confidence measure matching the specific problem (although we did not find this extra tuning necessary). The flexible nature of this algorithm makes it appropriate for most pattern recognition problems. The choice of the function may depend on specific features of the problem and of the basic learners. The effective number of parameters used by a BME ensemble is greater than that used by an ensemble that averages similar classifiers, trained on the same data set (parallel machine). A parallel machine with k classifiers, each with N effective parameters, also has N effective parameters. A BME effectively has more parameters because of the difference between the data sets (because the confidence measure is a constant simple function of the output vector, it adds no parameters). The upper bound is kN parameters, but the actual number is much closer to the lower bound. 6.2 Multilevel Ensembles: Model Selection Plus Averaging. We emphasized the different advantages of two basic combination schemes: classifier selection and averaging. We argue that by applying two levels of ensembles—one for selection and the other for averaging—the advantages of each ensemble approach may be exploited in a better way than by a compromise. Most studies state that from a certain number of classifiers, the performance of an ensemble becomes steady. When the training set is partitioned between different experts, the effect of overfit may cause a decline in the performance as the number of experts increases and the training set size for each expert becomes too small. We suggest two ways of combining ensemble averaging and expert selection to improve performance. The first approach is training several sets of BMEs and using them in a multilevel ensemble: The output of this ensemble is the simple average of the outputs of the various BMEs, each extracted as previously described. Some of the gain here is due to overcoming the “stitch” effect: patterns in the boundaries between regions covered by different experts may yield poor performance. Using different sets of BMEs with different partitions might help overcome this. The ability to gain from such a multilevel approach relies on the lower-
Boosted Mixture of Experts
491
level ensemble’s being a selection ensemble. For ensembles based on averaging learners trained with similar data, this would just be a larger ensemble. At the other extent, decision trees may be considered as selection-style ensembles of simpler tree predictors. Ensembles, combining the output of trees trained on bootstrapped copies of the same data (bagging), effectively improve performance (Breiman, 1996a). Ensemble methods that encourage diverse training sets may gain from such a method if the data partitions vary. Using a dynamic combination models makes the ensemble even more a selection-style ensemble. Therefore, this approach is most appropriate for use with the BME algorithm. Another approach follows ideas from the query-by-committee framework (Seung, Opper, Sompolinsky, 1992; Freund, Seung, Shamir, & Tishby, 1993). According to this approach, a disagreement in an ensemble marks interesting patterns that are located in information gaps. Committees may be used as the basic experts, with the average as the expert’s output and the disagreement between the committee members as a measure to the expert’s confidence. It is likely that the agreement between the different members of a committee is higher because the presented patterns are more similar to those in the committee’s training set. This also follows the principle used in Perrone and Cooper (1993). They suggest that in order to achieve an ensemble with minimum variance, the coefficient for each member should be inversely proportional to its variance (versus the ground truth). We assume that because of the different training sets, the members of each committee have different variances that vary in different regions of the input space. This follows the use of the internal variance in each committee as an estimate to its error rate (Shimshoni & Intrator, 1996). 7 Results 7.1 Synthetic Example. We first demonstrate the capabilities of the algorithm on a synthetic two-class two-dimensional problem (see Figure 1), to provide more intuition about the way it works. Each class is a mixture of gaussians. Patterns of the first class are drawn with a probability of 80% from the leftmost gaussian (x ∼ N(−6, 1), y ∼ N(0, 1.5)) and with probability of 20% from the lower central gaussian (x ∼ N(1, 1), y ∼ N(−0.4, 0.1)). Patterns of the second class are similarly drawn from the gaussians centered at (6,0) and (−1, 0.4). We performed tests with 2000 points drawn with equal probability from both classes. We used a simple perceptron as our basic learner. A single learner achieved a 16% error rate (all induced by the small gaussians). An ensemble composed of two to four independent learners combined by a weighted average achieved similar performance. A multilayer perceptron with two hidden units also had 16% error. The BME ensemble used the absolute value of the perceptron output (which was in [−1, 1]) as its confidence score and a gating function, com-
492
Ran Avnimelech and Nathan Intrator
Figure 1: Input distribution of the synthetic task.
bining the confidence function and a constant coefficient for each of the two basic learners (a hard partition was used for training). The BME ensemble achieved a 3% error rate on this task. The “first” learner performs a horizontal separation: the main gaussians are classified correctly, with high confidence, and patterns in the small gaussians get a low confidence score. The second learner performs a vertical separation, but it tends to overestimate its confidence. However, the first learner is assigned a higher reliability coefficient; thus, the output of the second learner has influence only when the first one is not confident. In the initialization of the second learner (step 2 in the algorithm drawing), it was presented with a subset consisting of the 15 to 20% of the patterns whose confidence was lower than 0.3. This subset included most of the patterns belonging to the small clusters. It also had a small number of patterns from the main clusters. As the learner took into account all of the patterns, its decision boundary was a diagonal line from upper left to lower right. Thus, the difficult subset included data points at one vertical edge of each main cluster (and data points horizontally far from the centers of their gaussians). In the refining stage (step 3 in the algorithm drawing), the basic reliability coefficients for each learner were recalculated at each refining cycle, and then the data were split in a deterministic manner: each data point was assigned to the learner whose product of the confidence score on it and the reliability coefficient was higher. The refining stage had effect mostly on the first learner, which was able to produce a better estimation of the classification for the main gaussians. In this example, the refining stage did not contribute much. We also
Boosted Mixture of Experts
493
Figure 2: (A) Examples of digits from the NIST database. (B) Their representation by the first 32 principal components.
performed a slightly different variant of this problem in which the BME ensemble had a 6% error rate before refining, and after a few refining cycles it dropped to 4%. The first learner initially performed a compromise of the two separations, and when it had to perform only one separation, its performance improved. 7.2 Digit Recognition Results. The BME algorithm was empirically evaluated on digits from the NIST database (see Figure 2A). Preprocessing operations, similar to those described in Bottou et al. (1994), were applied to the digits. Digits were size normalized to fit a 20 × 20 pixel box (gray scale) centered within a 28 × 28 image. We then performed principal component analysis and used the first 32 components as input to our classifiers (see Figure 2B). The basic classifier used was a feedforward neural network, trained via the backpropagation algorithm (with momentum). The network’s input layer had 32 units, and its single hidden layer consisted of 16 units. The 10-dimensional output vector was used to extract the output digit and its confidence level. In order to evaluate the unique contribution of the new algorithm, we compared it to a standard ensemble (parallel machine). This ensemble consisted of several learners trained independently, each with different starting conditions. The combination model used to extract the ensemble output was averaging of the output vectors of the different classifiers and decision according to the highest output. Increasing the number of networks improved the ensemble’s performance. We tested the performance ensembles trained with the BME algorithm. The initial training set for new learners added to the ensemble was constructed by choosing from the training set of each of the other learners those patterns on which it was less confident (we took 1/(a + b ∗ n) of its set, where n is the current size of the ensemble and a, b are arbitrary constants). The confidence score of each pattern and a specific classifier was (P1 − P2 )4 , where P1 is the highest output of the classifier on the pattern and P2 is its
494
Ran Avnimelech and Nathan Intrator
Table 1: Performance of Various Ensembles on a Digit Recognition Task. Number of Nets
2 3 4 5 8 10
Parallel Machine
Boosted Mixture of Experts
Multilevel Ensemble (2∗ N nets)
Mean
SD
Mean
SD
Mean
93.75% 94.35 94.6 94.65 94.65 94.65
0.35% 0.45 0.45 0.3 0.4 0.4
94.65% 95.15 95.3 95.4 95.6 95.7
0.3% 0.3 0.35 0.3 0.3 0.3
95.15% 95.5 95.7 95.8 96 96.1
SD 0.3% 0.35 0.35 0.25 0.25 0.25
second highest output (probabilities were normalized to sum to 1 for any pattern). The gating function used at the refining step of the training, to get the probability of assigning a pattern to the training set of a specific classifier, was this confidence score (no global reliability coefficient was used). This gating function was also used in the combination model, as the weight given to each classifier in the weighted average of the output vectors. We also performed a test of the multilevel ensemble. A simple average was applied to the output of two independently trained BME ensembles of N classifiers. Such an ensemble combines the advantages of an ensemble choosing the appropriate classifier for each pattern and an averaging ensemble. Table 1 presents the performance of the three ensemble methods over a wide range of ensemble sizes. These results were collected using five different partitions of the data into a 49,000-digit training set and a 10,000digit test set. The basic MLP used had 32 inputs, 10 outputs, and 16 hidden units. By a naive counting, this gives N = (32 + 1) ∗ 16 + (16 + 1) ∗ 10, which is almost 700 free parameters. The effective number N is of the same order of magnitude. The naive number of parameters for both a parallel machine and a BME ensemble of k nets is kN, and for the multilevel ensemble it is 2kN. Effectively, it is N parameters in the parallel machine, and for both the BME and the multilevel ensemble it is between N and kN . We tried to check whether the reported effect was due to only the increased number of parameters in the BME ensemble. The BME’s number of parameters may be similar to that of a parallel machine, similar to that of a single classifier with a k-times larger hidden layer, or some intermediate case. For k = 3, the success rate of a parallel machine was 94.35%, the success rate for a larger net was 94.2%, and for a BME it was 95.15%. An average of two large nets had a success rate of 94.9%, while the multilevel ensemble had 95.5% success. The results indicate that the performance of an ensemble machine trained
Boosted Mixture of Experts
495
with the BME algorithm (and combined appropriately) is significantly better than a standard ensemble (parallel machine). The improvement rate is similar to that achieved using boosting (Drucker, Cortes, Jackel, Lecun, & Vapnik, 1994). It is encouraging that this improvement rate is kept even for a high number of classifiers (20% error reduction for 10 classifiers). The improved performance for a large ensemble was achieved despite the fact that the classifiers in this scheme were trained on a small portion of the data set. The improvement due to the BME algorithm beyond ensemble performance may be even larger when greater training sets are used (e.g., by multiplying samples using invariant transformations, as in Bottou et al., 1994). The results further demonstrate the potential of combining the two basic schemes for ensemble machines in a multilevel approach. Our ensemble used a weighted average of classifiers, which tended to select the locally best classifier rather than average classifiers. Averaging the outputs of two such ensembles yielded further improvement in the results. These results are not fully contrasted with other ensembles of similar size, but when they are (two ensembles of 4 to 5 classifiers versus 8 to 10 classifiers) they have a slight advantage. Furthermore, because most studies claim that adding classifiers beyond a certain number is not expected to improve the performance further, the constant incremental improvement is encouraging. 8 Conclusions This study analyzed two of the more advanced frameworks for ensembles of learning machines: boosting and the mixture of experts. We discussed the advantages and weaknesses of each algorithm and reviewed several ways in which the principles of these algorithms may be combined to achieve improved performance, including variants of each algorithm incorporating elements of the other. We suggested a flexible procedure for constructing an ensemble machine based on principles of these two algorithms. The essential components are: • Training several classifiers on subsets of the data with a significantly different distribution and using them in an ensemble. • Dynamic classifier selection, which is common to the training and the test stages. • Usage of a confidence measure for each of the classifiers as the gating function (in mixture-of-experts terminology), which determines their contribution to the ensemble output. These principles lead to outperforming conventional ensemble machines. The flexibility of the procedure is due mostly to the use of a confidence measure, which may be adjusted specifically for any classification or regression problem. This makes boostwise algorithms appropriate for regression problems as well. We further suggest an all-purpose confidence measure
496
Ran Avnimelech and Nathan Intrator
by using a committee of simple learners as the basic learner in our algorithm. The disagreement among a committee for a given pattern becomes a confidence measure. We have made a distinction between two groups of ensemble machines: classifier selectors and classifier averagers. These two mechanisms provide different advantages for ensembles: using local experts may reduce bias, while averaging tends to reduce variance. We claim that a multilevel approach combining selection and averaging is capable of improving the performance of ensembles and that it may be better than a compromise between selection and averaging. A digit recognition task from the NIST database was used to demonstrate the advantages of the BME and multilevel ensemble and achieve a significant reduction of the error rate over standard ensembles.
Acknowledgments We thank NIST and H. Drucker for the handwritten digits database we used.
References Bottou, L., Cortes, C., Denker, J., Drucker, H., Guyon, I., Jackel, L., LeCun, Y., Sackinger, U. M. E., Simard, P., & Vapnik, V. (1994). Comparison of classifier methods: A case study in handwritten digit recognition. In Proceedings Int. Conf. on Pattern Recognition (Vol. 12, pp. 77–82). Breiman, L. (1996a). Bagging predictors. Machine Learning, 24, 123–140. Breiman, L. (1996b). Bias, variance and arcing classifiers (Tech. Rep. TR-460). Berkeley: Department of Statistics, University of California, Berkeley. Breiman, L. (1996c). Stacked regressions. Machine Learning, 24, 49–64. Drucker, H., Cortes, C., Jackel, L., LeCun, Y., & Vapnik, V. (1994). Boosting and other ensemble methods. Neural Computation, 6(6), 1289–1301. Drucker, H., Schapire, R., & Simard, P. (1993). Improving performance in neural networks using a boosting algorithm. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems, 5 (pp. 42–49). San Mateo, CA: Morgan Kaufmann. Freund, Y. (1990). Boosting a weak learning algorithm by majority. In 3rd Annual Workshop on Computational Learning Theory (pp. 202–216). Freund, Y., & Schapire, R. (1995). A decision-theoretic generalization of on-line learning and an application to boosting. In 2nd European Conference on Computational Learning Theory. Freund, Y., Seung, H., Shamir, E., & Tishby, N. (1993). Information, prediction and query by committee. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems, 5 (pp. 483–490). San Mateo, CA: Morgan Kaufmann. Hansen, L. K., & Salamon, P. (1990). Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10), 993–1001.
Boosted Mixture of Experts
497
Ho, T., Hull, J., & Srihari, S. (1994). Decision combination in multiple classifier systems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(1), 66–75. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation, 3(1), 79–87. Jordan, M. I., & Jacobs, R. A. (1992). Hierarchies of adaptive experts. In J. E. Moody, S. J. Hanson, & R. P. Lippmann (Eds.), Advances in neural information processing systems, 4 (pp. 985–992). San Mateo, CA: Morgan Kaufmann. Jordan, M. I., & Jacobs, R. A. (1994). Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6(2), 181–214. Krogh, A., & Vedelsby, J. (1995). Neural network ensembles, cross validation, and active learning. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 231–238). Cambridge, MA: MIT Press. Meir, R. (1995). Bias, variance and the combination of least square estimators. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 295–302). Cambridge, MA: MIT Press. Perrone, M. P., & Cooper, L. N. (1993). When networks disagree: Ensemble method for neural networks. In R. J. Mammone (Ed.), Neural networks for speech and image processing. London: Chapman-Hall. Raviv, Y., & Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. Connection Science (Special Issue), 8, 356–372. Schapire, R. (1990). The strength of weak learnability. Machine Learning, 5(2), 197–227. Schwenk, H., & Bengio, Y. (1997). Adaptive boosting of neural networks for character recognition (Tech. Rep. TR-1072). Montreal: Department d’Informatique et Recerche Operationnelle, Universit´e d’Montreal. Seung, H. S., Opper, M., & Sompolinsky, H. (1992). Query by committee. In Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory (pp. 287–294). Shimshoni, Y., & Intrator, N. (1996). Classifying seismic signals by integrating ensembles of neural networks. In S. Amari, L. Xu, L. W. Chan, I. King, & K. S. Leung (Eds.), Proceedings of ICONIP Hong Kong. Progress in Neural Information Processing (Vol. 1, pp. 84–90). New York: Springer-Verlag. Tresp, V., & Taniguchi, M. (1995). Combining estimators using non-constant weighting function. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 7. Cambridge, MA: MIT Press. Waterhouse, S. R., & Cook, G. (1997). Ensemble methods for phoneme classification. In M. Mozer, J. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems, 9. Cambridge, MA: MIT Press. Waterhouse, S. R., & Robinson, A. J. (1996). Constructive algorithms for hierarchical mixtures of experts. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8. Cambridge, MA: MIT Press. Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5, 241–259. Received January 10, 1997; accepted December 10, 1997.
LETTER
Communicated by Robert Jacobs
Boosting Regression Estimators Ran Avnimelech Nathan Intrator Department of Computer Science, Sackler Faculty of Exact Sciences, Tel-Aviv University, Tel-Aviv, Israel
There is interest in extending the boosting algorithm (Schapire, 1990) to fit a wide range of regression problems. The threshold-based boosting algorithm for regression used an analogy between classification errors and big errors in regression. We focus on the practical aspects of this algorithm and compare it to other attempts to extend boosting to regression. The practical capabilities of this model are demonstrated on the laser data from the Santa Fe times-series competition and the Mackey-Glass time series, where the results surpass those of standard ensemble average. 1 Introduction Boosting algorithms are ensemble learning algorithms that achieve improved performance by training different learners on different distributions of the data and combining their output (Schapire, 1990; Freund & Schapire, 1995). Boosting was found to be an effective method to achieve improved performance in many classification tasks. The success and increasing interest in ensemble methods for regression tasks encourage the application of boosting to regression tasks. There have been several different suggestions regarding the way in which this extension should be performed, each one considering a different analogy between classification errors and regression errors. We follow the version suggested by Freund (1995) and study the practical effects of the difference between classification and regression errors and the modifications that may lead to better performance in practice. We find that this version of boosting for regression can reduce the error rate when a small number of big errors contribute a significant part of the mean squared error (MSE). In addition to the theoretical analysis of the algorithm, we present empirical tests, including a case study of the behavior of the different predictors in one of the tests and how it contributes to error reduction. Section 2 reviews the basic boosting algorithm and the AdaBoost algorithm and their applications. Section 3 reviews the usage of ensemble algorithms for regression tasks. Section 4 presents the algorithm and a theoretical analysis of it within an appropriate model. Section 5 reviews other attempts to extend boosting to regression and highlights the advantages and Neural Computation 11, 499–520 (1999)
c 1999 Massachusetts Institute of Technology °
500
Ran Avnimelech and Nathan Intrator
disadvantages of each suggestion. We also claim that the algorithm may be effective in many tasks that are not fully compliant with the model, although it is not analytically shown. Empirical results are shown in section 6. 2 The Boosting Algorithm 2.1 The Original Boosting Algorithm. The original boosting algorithm (Schapire, 1990) was suggested in the context of the PAC learning model (Valiant, 1984). Theoretically, it enables achieving an arbitrarily low error rate, requiring the basic learners only to be able to achieve performance that is slightly better than random guessing on any input distribution. The algorithm trains the first learner on the original data set, and new learners are trained on data sets enriched with difficult patterns—patterns misclassified by some of the previous classifiers. There have been various improvements to the original boosting algorithm. Original boosting uses hierarchies of three-classifier ensembles; boosting by majority uses a simple ensemble that may consist of more classifiers, thus achieving the same improvement with less classifiers (Freund, 1995). The boosting algorithm was successfully used in various real-world classification tasks, despite the fact that the assumptions of the PAC model do not hold in this more general context. Ensembles of neural networks, constructed by the boosting algorithm, significantly outperformed a single network or a simple ensemble on a digit recognition task (Drucker, Schapire, & Simard, 1993) and on a phoneme recognition task (Waterhouse & Cook, 1997). In this article, we extend the boosting algorithm to regression problems by introducing the notion of weak and strong learning and an appropriate equivalence theorem between them. Practical implications are demonstrated on the laser data set. 2.2 AdaBoost. AdaBoost is an extension of boosting that applies to a more general context and takes into consideration the different error levels of the various classifiers (Freund & Schapire, 1995). The error rate determines the reweighting applied to the data when they are adaptively resampled for providing a training set for the next classifier. The error rate also determines the coefficient of each classifier when computing the ensemble output. This modification makes boosting effective when performance on the difficult sets is much worse than on the original set. The reweighting procedure tries to construct decelerated classifiers by assigning weights to the new training set s.t. the (weighted) error rate of the previous learner on it would be 0.5. Initially the weights are uniform µ (wi = 1/N) Let c(µ) be the true class of pattern µ (for a two-class task), ht the hypothesis generated at step t, dt (µ) the boolean that is equal to 1 when ht (µ) 6= c(µ), ²t the (weighted) error rate of ht , and βt = ²t /(1 − ²t ).
Boosting Regression Estimators
501 µ
µ
d (µ)
The weight updates used by AdaBoost are wt+1 = wt · βt t · Ct , where P µ Ct is a normalization factor s.t. mu wt+1 = 1. The combination model is a weighted voting of the classifiers, with weights µ ¶ µ ¶ 1 1 − ²t = ln . αt = ln βt ²t If the errors of different classifiers were independent (because of the stepwise deceleration), this would be the optimal Bayes decision. Recently, several successful applications of AdaBoost have been reported (Breiman, 1996b; Schwenk & Bengio, 1997). Breiman applied AdaBoost to decision trees on various data sets and achieved improved performance (compared to bagging). Schwenk and Bengio applied AdaBoost to multilayer perceptrons (MLPs) and autoencoder-based classifiers (“diabolo networks”) on character recognition tasks. There have been several explanations as to why AdaBoost works. Breiman (1996b) suggested that AdaBoost and other algorithms that adaptively resample the training data and combine the classifiers (arcing) gain by increasing the variance component of the error and reducing the variance, which enables gaining more from the combination. Schapire, Freund, Bartlett, and Lee (1997) studied the success of boosting low-variance algorithms and suggested an alternative explanation. They present examples in which the error rate on the test set keeps decreasing when new hypotheses are added to the ensemble, even after the training error reaches zero. They explain this apparent contradiction with standard learning curves and the Occam’s razor principle by studying the behavior of a quantity they term margin. For a convex function (i.e., linear combination with coefficients summing to 1) of a set of classifiers, the margin is defined (for any pattern) as a weighted sum of +1 for correct classifications and −1 for errors. Ensembles gain by increasing the minimal value of the margin. AdaBoost is specifically designed to concentrate on the patterns with low margin, and this leads to its improved performance. This article also includes a statistical bound to the error rate on unseen data based on the minimal margin and the VC-dimension of the individual learners, which is lower than the bound when the margin is not considered. This explains why in some cases the error on test data kept decreasing even after training error reached 0. A quantity edge, which is an extension of margin, was suggested in Breiman (1997). AdaBoost and other arcing algorithms are optimization algorithms for minimizing some function of the edge. 3 Ensemble of Predictors in Regression Setup Regression learning may exhibit complex behavior such as nonlinearities, chaotic behavior (especially in time-series prediction), local effects or nonstationarity, and high levels of noise. Ensemble averaging of predictors can improve the performance of single predictors and add to its robustness
502
Ran Avnimelech and Nathan Intrator
under these circumstances. This occurs when the errors made by different predictors are independent, and thus the ensemble average reduces the variance portion of the error (Geman, Bienenstock, & Doursat, 1992). There are various ways to increase the independence of the errors. The simplest is to split the data into independent sets (Meir, 1995); however, such a reduction in the number of training patterns may degrade the results of each predictor too much. Another approach is to bootstrap several training sets with a small percentage of nonoverlapping patterns (Efron & Tibshirani, 1993) or training sets constructed by sampling with repetition (Breiman, 1996a). A recently proposed method increases independence between the predictors by adding large amounts of noise to the training patterns (Raviv & Intrator, 1996). A different approach to ensemble averaging is the adaptive mixture of experts (Jacobs, Jordan, Nowlan, & Hinton, 1991). This method is a divideand-conquer algorithm that cotrains a gating network for (soft) partitioning the input space and expert networks modeling the underlying function in each of these partitions. In this article, we introduce a boosting algorithm for regression as a method for training an ensemble of predictors so as to optimize their collective performance. The algorithm is based on a fundamental observation that often the MSE of a predictor is significantly greater than the squared median of the error due to a small number of large errors. By reducing the number of large errors, we are able to reduce the MSE. 4 The Regressor-Boosting Algorithm 4.1 Model Definitions. We introduce a regression notion of weak learning taken from the PAC framework (Schapire, 1990). The essence of the regression problem is constructing a function f (x) based on a training set (x1 , y1 ), . . . , (xN , yN ), for the purpose of approximating y at future observations of x. Usually the goal is to approximate just E[y|x] and not the conditional distribution P[y|x], that is, to approximate the underlying function G: y = G(x) + ε(x) with ε(x) a zero-mean (for any x) noise. It is assumed that the samples in the training set were drawn from the unknown distribution P(x, y) = P(x) · P(y|x) and the goal is to approximate E[y|x]. The model we describe includes some simplifying assumptions, and the more realistic case is discussed at the end of this section. We assume no noise is present, y = G(x), and this unknown function should be approximated . Another assumption is the existence of unlimited number of pairs hx, G(x)i drawn from the joint distribution P(x, y). Therefore, we ignore within this analysis the different performance on training data and unseen data. In this context, we regard various input distributions by applying selective filters to the input: P0 (x) ≡ P(‘filter’, x) = P(x) · P(‘filter’|x). Given an unknown function G and a source of pairs hx, G(x)i, the following definitions refer to learners—algorithms that approximate a function G
Boosting Regression Estimators
503
by a function fD that depends on the distribution D ≡ PD (x): γ -Weak learner. A learner for which exists some α < 1/2, such that it is capable, for any given distribution D and δ > 0, of finding, with probability (1−δ),1 a function fD such that PrD [| fD (x)−G(x)| > γ ] < α. γ -Strong learner. A learner capable (for any given D, δ) of finding, with probability (1−δ), a function FD such that PrD [|FD (x)−G(x)| > γ ] < ² for any ² > 0. Big error (with reference to γ ). An error g.t. γ . The big error rate (BER) for h on D0 is PrD0 [|h(x) − G(x)| > γ ]. When the term big error is mentioned not in the context of a specific γ , it applies to any γ for which the learning algorithm estimating the underlying function is a γ -weak learner. γ -Weak (γ -Strong) learnability. Whether any γ -weak (γ -strong) learner for G exists. Some immediate results are obvious from these definitions: • Any γ -strong learner is a γ -weak learner. • For γ1 < γ2 any γ1 -weak (strong) learner is a γ2 -weak (strong) learner. • A γ -weak learner can find, for any distribution D, a function fD whose median error is smaller than γ (i.e., BER < 12 ). It is not clear from the definitions whether γ -weak learnability is equivalent to γ -strong learnability. The rest of this section will prove the equivalence of the two terms and emphasize the significance of this equivalence. 4.2 The Regressor-Boosting Theorem. The following theorem suggests that in problems for which a γ -weak learner exists, an arbitrarily low rate of big errors may be achieved. This can reduce the MSE. Theorem. Given an unknown function G, a source of pairs hx, G(x)i, and γ : If a γ -weak learner is available, then a γ -strong learner may be constructed too. In other words, γ -weak learnability is equivalent to γ -strong learnability. The contribution of the big errors to the general MSE is a function not only of their percentage but also of their MSE. However, given ²—the BER of the ensemble—and the error distribution of the individual predictors, this contribution is bounded. At most they contribute as much as the ² highest errors of a simple predictor. Each such error is at most the median error of several functions generated by the weak learning algorithm at this input point. Similarly, the error distribution on the inputs, which originally were 1
The probability limit is for a misrepresentative data set.
504
Ran Avnimelech and Nathan Intrator
not big errors, is not only bounded by γ , but it also is not likely to get worse. This theorem implies that if the MSE is dominated by a small number of relatively large errors (i.e., the average (RMS) error is much greater than median error), the RMS error can be reduced to values close to the median error. This occurs in many cases. (Figure 3 shows such an example of the error distribution taken from the empirical tests we performed.) Another example is gaussian error distribution, where 32% errors that are higher than the RMS error contribute about 80% of the MSE. The choice of γ is arbitrary (as long as there exists a γ -weak learner), and there is a trade-off between lower γ and lower ². Therefore, a γ could be chosen that would achieve a greater error reduction. 4.3 Proof. The following constructive proof follows the proof for boosting in classification (Schapire, 1990). Instead of the majority vote of an ensemble used for classification tasks, the median of an ensemble is used in regression problems. We discuss several variants: Boost1 is the variant suggested in Freund (1995). Boost2, which is also appropriate for the proof, is a variant more focused in reducing the MSE by considering the different kinds of errors. Another variant, BOOST3, which does not fit this proof, is focused on the MSE. The essence of the algorithm is the construction of ensembles of three estimators and combining them to reduce the BER from α to (3α 2 − 2α 3 ) or less. We present two versions of this step (BOOST1, in Figure 1, and BOOST2). Using hierarchies of such ensembles leads to an arbitrarily low BER.
BOOST1: The first estimator in such an ensemble is trained on the original input distribution. Fifty percent of the data set used for training the second estimator are patterns on which the first estimator has a big error and 50% are those on which it has not (no change in the internal distribution of each of the two groups). The training set for the third estimator consists only of patterns on which exactly one of the previous estimators had a big error. The ensemble output is the median of the outputs of the different estimators. Figure 1 shows the different distributions of the training sets of the three predictors and how they are combined to achieve a low BER. BOOST2: Similar to BOOST1, but the training set of the third estimator contains patterns on which the previous estimators had big errors of different signs, in addition to those on which exactly one of them had a big error. This algorithm ensures an error rate even lower than (3α 2 − 2α 3 ). A summary description of the algorithm is presented below. It includes
Boosting Regression Estimators
505
Figure 1: Main step of the boosting algorithm (BOOST1). Three predictors trained on different input distributions are combined. Training set of B contains the patterns on which A has a big error and a similar amount of patterns on which it does not have a big error. A and B may be considered an ensemble whose output pairs are evaluated according to what they imply on the median of them and another output: whether it would necessarily have a big error (Error) or it could not have a big error (Correct), or it depends on the value of the third output (Reject). The training set of C contains only Rejects—patterns on which A XOR B had a big error. This scheme shows that only on (3α 2 − 2α 3 ) of the patterns 2-3 predictors have a big error. The median may have a big error only on these patterns. See proof for the mathematical details
BOOST3, a modification of BOOST2 described and justified later: 1. Split the training set to three Sets. The first set should be smaller than the other two sets, because it is used as a whole for training. 2. Train the first expert on Training Set 1 = Set 1. 3. Assign to Training Set 2 all the patterns from Set 2 on which expert 1 has a big error and a similar number of patterns from this Set 2 on which it does not have a big error and train the next expert on it.
506
Ran Avnimelech and Nathan Intrator
4. Assign all patterns from Set 3 on which the third expert may have a critical influence on the median to Training Set 3 for the third expert and train. The criterion for this set may differ by version: Boost1 Any pattern on which exactly one of the first experts has a big error. Boost2 Set (BOOST1) + Any pattern on which both experts have a big error, but these errors have different signs. Boost3 Set (BOOST2) + Any pattern on which both experts have a big error, but there is a “big” difference (see details in section 4.4) between the magnitude of the errors. 5. The ensemble output for any test pattern is the median of the outputs of the three experts. Proof. Given a test set and three weak learners trained as above, each with big error rate α on appropriate distribution, we show that in an ensemble of three such estimators, (3α 2 − 2α 3 ) of the patterns are those on which two to three estimators have big errors. Therefore, the median has a big error rate of (3α 2 − 2α 3 ) at most (the median may have a big error only if two to three estimators have such error, but if two estimators have big errors with different signs, the median would be the estimator with a small error).
BOOST1: We term the ratio of patterns on which the first two estimators have big errors as ErrorAB and that of patterns on which one of them has a big error as RejectAB (see Figure 1). ErrorABC is the number of patterns on which most of the estimators have big errors. The unknown BER of B on patterns for which A produces a γ -accurate prediction is marked as β and the BER of B on patterns on which A has a big error is marked as β2 . (It should be noted that as we randomly choose patterns from an infinite source, there are no two distinct groups of patterns on which A produces a γ -accurate prediction—those used and those not used for training of B .) The weak learning assumption states that B ’s BER on its training set distribution is at most α. Half of B ’s training set is patterns on which A had a big error. By applying the weak learning assumption on this distribution, we get: 12 · β + 12 · β2 ≤ α. Assuming worse case (equality), B ’s BER on A’s big errors is β2 = 2α − β, leading to: ErrorAB = α · (2α − β) RejectAB = β · (1 − α) + (1 − (2α − β)) · α ErrorABC = ErrorAB + α · RejectAB = α · (2α − β) + α · [β · (1 − α) + (1 + β − 2α) · α] = 2α 2 − αβ + αβ − α 2 β + α 2 + α 2 β − 2α 3 = 3α 2 − 2α 3
BOOST2: Error0AB is the percentage of patterns on which the median would have a big error regardless of the third estimator (i.e., both A and B
Boosting Regression Estimators
507
have big errors with common sign). Reject0AB is the percentage of patterns for which the median would have a big error iff the third predictor had a big error. The unknown ratio of patterns on which A and B had big errors of different signs (relatively to the total number of common big errors) is marked ζ : Error0AB = α · (2α − β) · (1 − ζ )
Reject0AB = β · (1 − α) + (1 − (2α − β)(1 − ζ )) · α
Error0ABC = Error0AB + α · Reject0AB
= 3α 2 − 2α 3 − αζ (2α − β) + α 2 ζ (2α − β) = 3α 2 − 2α 3 − ζ α(1 − α)(2α − β) 4.4 Practical Considerations and Limitations. The above model refers to a threshold γ for big errors. However, in regression problems, the goal is usually to reduce the MSE, and this presents a dilemma about the desired value of γ . Theoretically, the choice may be the lowest value for which we have a γ -weak learner. In practice there are several considerations: • The size of the data set is finite. • Only a limited number of estimators are combined. • Boosting may be effective, although our learner is not a γ -weak learner (just like in classification). The optimal γ is one for which the big errors are responsible for a significant part of the MSE, but the BER is low (usually the sets on which the second and third estimators are trained are more difficult and have a higher BER). In most cases, the choice of a good γ may require tuning. The use of this boosting algorithm may also change the basic learner appropriate for the problem. It encourages the use of learners that are robust to the presence of outliers (e.g., minimize MSE0 ≡ MSE excluding x% greatest errors). In the training stage, the worse points are less crucial, as they will be learned by the other estimators. When the ensemble is used for prediction, the worst of the three estimates (for any sample point) is not relevant, as the median must be one of the other estimates. In practical cases, noise will also be present. Its various effects on the algorithm are mentioned in Section 4.6, but the bottom line is that boosting may be effective when the errors of its basic estimators are large compared to the noise level. One of the limits of boosting in regression is that while its focus on the real goal (MSE reduction) is limited, the basic learner already focuses on the harder patterns (e.g., the learning rule in backpropagation practically assigns a higher weight to patterns on which there is a big error). In iterative prediction, however, training a predictor optimized for stepwise prediction and using it iteratively is usually simpler and more effective than designing
508
Ran Avnimelech and Nathan Intrator
a predictor specifically for iterative prediction. Yet it is simple to reweight the training set according to the performance of the iterative prediction, while training the individual predictors (using the weighted training sets) with a single-step prediction goal. An extension of the principle that guided us to suggest BOOST2 is including some of the patterns on which the first two predictors have big errors (of same sign) in the training set of the third set. This may increase the BER of the median predictor, but if these are patterns, for which there is a big difference between the errors of the first two estimators, it is likely to reduce its MSE. The two extents are using the original BOOST2 or adding all such patterns to the set. A criterion for choosing some of the patterns should consider the effect of choosing between the two predictors (i.e., the difference in their squared errors) and the expected cost to performance on other patterns of including it in the training set (i.e., the pattern’s difficulty). A simple criterion we use is including those patterns on which the difference between the predictions is g.t. the threshold γ by which we defined big errors. 4.5 Extensions of the Algorithm. 4.5.1 AdaBoost. Applying AdaBoost to regression tasks is done in a similar way to the way in which boosting was applied to regression. The description of the actual algorithm is as follows: µ
1. Initialize uniform weights to patterns: w1 = 1/N. 2. Train single predictor according to current weights. µ
3. Compute error ratio ²t according to pattern errors ²t : ²t = µ
P
µ
µ:²t >γ µ
4. If ²t > 0.5 update weights: If pattern had a big error: wt+1 = wt · µ µ 0.5 . otherwise: wt+1 = wt · 1−² t
µ
wt . 0.5 ²t ,
5. If halting condition has not been reached, return to step 2; halting condition is either maximal number of hypotheses (predictors) or maximal number of consecutive failures (BER > 0.5). The final prediction is a weighted median of the predictors with coefficients t αt = ln 1−² ²t . In practice, AdaBoost may be less effective due to several reasons. In many cases, the first regressor is significantly better than other regressors PN (because it is evaluated on the original distribution) and α1 > t=2 αt . Therefore, the median is the first estimator. Another practical limit is the fact that the first estimator is already a compromise between better local estimations and the fact that in regression, the bigger errors already have a big effect due to minimizing the squared error (unlike classification).
Boosting Regression Estimators
509
Because we are interested in reducing the MSE rather than the BER, a nonweighted combination may result in a performance similar to or better than that achieved by a weighted combination. The coefficients in the weighted combination model are determined according to the BER of each predictor, which is just a partial indicator of the quality of the predictor. Similarly, combining the predictors through the mean is likely to be as good as using the median. 4.5.2 Iterative Approach: Several Error Thresholds. A disadvantage of the algorithm presented here is that it is effective only in reducing the errors to be below some threshold (which cannot be lower than the median error). A solution is using an iterative approach. Once the errors of the vast majority of the patterns are lower than the threshold, a lower threshold can be chosen, and small ensembles may be trained on the training set reweighted according to this threshold. Assuming the error sizes are uniformly distributed beneath the old threshold (a pessimistic assumption), a new threshold that is 0.7 the old one (half the squared error) will still have a sufficiently small BER. 4.6 Boosting and Noisy Estimation. The description of the model ignored noise. The presence of noise poses the problem of overfitting the data—learning the noise rather than just the underlying function. There are cases in which the use of boosting may increase the sensitivity to noise. Therefore, boosting will be effective when the error level of the basic estimators is higher than the noise level. Noise will not only interrupt the learning process of each predictor, an effect emphasized by the split and “waste” of the data, but also mark good estimates as big errors, thus overemphasizing them. A complete analysis of the behavior of boosting in the presence of noise depends on the error distribution of the basic learner, the noise distribution, the exact effect the reweighting has on the basic estimators, and other factors. However, if we have some estimate of the noise level, there may be some rules of thumb that may indicate when noise may disturb boosting. The reweighting should not emphasize the noise; the noise should be low compared with the error level of the single estimator. It should also be low relative to the threshold γ ; noise higher than γ or close to it (which may mark small errors as big) should be rare. When a series of estimators is used, the weights of patterns that had big errors by most or all the estimators should be bounded according to the probability of noise > γ . This limits the number of estimators constructed and combined by the algorithm. This limit may also be encountered if the threshold is too high compared to the error level of a single estimator (because that will cause massive weight updates). These limits may become relevant not only as a result of noise, but also due to the capacity of the estimators. Another effect that should be noticed in the presence of noise is the relation between the BER and the MSE. If γ is much
510
Ran Avnimelech and Nathan Intrator
higher than the average error, estimators trained on the reweighted set may have an MSE that is higher than or similar to that of previous estimators but a BER that is much lower. This may lead to giving them unjustified high coefficients in the weighted median used by AdaBoost. Such estimators may overfit a few points with high noise. The combination model of boosting has better immunity to noise than the generation of new estimators. One advantage is that the (weighted) median is at least as smooth as the single estimators. If there exist some metric D and some function G s.t. for every estimator | fi (x) − fi (y)| < G(Dxy ) then the same smoothness condition is also true for the median. Furthermore, the estimators that suffer from overfitting usually have a BER closer to 0.5 and will have lower coefficients. When the noise level varies (or the confidence level of the estimator), it may be useful to estimate it locally, and let that influence the flexible threshold γ (x). Using such a threshold may focus more on errors of the estimator rather than noise. 5 Other Attempts to Extend Boosting to Regression There have been several other attempts to extend boosting to regression tasks, one of which was also empirically tested (Freund & Schapire, 1995; Drucker, 1997). 5.1 Function Estimation as a Set of Boolean Queries. Freund and Schapire (1995) suggest an approach that considers a function estimation task as an infinite series of boolean queries: c(xi , y) = (yi > y). A squared error cost is achieved by the distribution of queries: P(y|xi ) ∝ |y − yi | (the possible values of y are bounded). Applying AdaBoost directly changes the distribution of pairs; the conditional distribution P(y|x) changes as well as P(x). This method has several advantages. It attempts to reduce the MSE directly, rather than the number of big errors. It attempts to reduce the error to zero (but stops when the hypothesis error exceeds 0.5). AdaBoost.R also favors errors whose sign is opposite to the sign of previous errors. The main disadvantage of this method is related to its implementation: The weak learner has to perform a task that is more complex than minimizing the weighted MSE. It may also be inappropriate for learning algorithms that rely on the gradient of the error function (e.g., backpropagation). The gradient increases only in the range between the target value and the current prediction. Thus, the patterns that will have a small error at an early stage of the training of an additional predictor will be emphasized. Another disadvantage is the massive weight changes when the errors are small compared with the y bounds. The initial massive weight changes also cause the coefficient of the first regressor in the combination model to be much greater than any other coefficient (even if error distribution is similar). Thus, in small en-
Boosting Regression Estimators
511
sembles, the median is actually the first predictor. This method also contains a hidden hyperparameter: extending the y bounds would change the percentage of errors (by adding many easy queries), thus severely changing the reweighting factors. 5.2 Boosting for Regression Using a Continuous Loss Function. Drucker (1997) suggests a different approach. A loss function Lµ is assigned to each pattern (for each estimator) that is a function (e.g., identity, square, exponent) of the ratio between its error and the maximal error (in the range ¯ L. ¯ The [0, 1]). The error rate is the (weighted) average loss, βt = (1 − L)/ µ µ L µ weight updates are wt+1 = wt · β · Ct . The algorithm terminates when L¯ exceeds 0.5. Drucker reports that the algorithm outperforms bagging on a set of benchmark synthetic tasks and on the Boston housing data set. Drucker’s method is characterized by the use of a continuous loss function for the reweighting of the patterns. The main advantages of this method are that it concentrates on the big errors with no need to adjust a parameter to the typical error, and it does not increase the complexity of the task presented to the basic learner. Its main disadvantage is the dependence of the loss function on the maximal error. This means that two estimators with the same error distribution relative to the maximal error—but with the maximal error of one double that of the other (i.e., ∀x : P(²2 = x) = P(²1 = 2x))—are considered to have similar performance. This also leads to big changes in the weighting as a single extreme value varies. If the algorithm is used just for reweighting the patterns, while the combination is through a nonweighted median/mean (which may be appropriate, as previously suggested), this disadvantage becomes less significant. 5.3 Strengths and Drawbacks of Threshold-Based Boosting. The main disadvantages of the method used in this work and in Freund (1995) are the fact that it is limited to reducing the errors to the chosen threshold and not down to zero and the need to choose this threshold. The significance of the first limit varies with the error distribution. If further error reduction is required, the method may be applied recursively, using a different threshold at each level. The need to choose the threshold for big errors may actually be an advantage at nontrivial tasks by providing flexibility and suggesting a choice of threshold based on the error distribution. (For simple tasks, a threshold slightly higher than the RMS error should be fine.) Another advantage of this algorithm is the simplicity of its implementation. A summary of such a comparison cannot state that one of the variants is always superior to another. The performance of each method relies on the specifics of the problem and the basic learner used. The mere fact that a method can theoretically reduce the error to zero does not imply such results in practice. AdaBoost.R is most appealing theoretically, but may
512
Ran Avnimelech and Nathan Intrator
Figure 2: Typical segment of laser-intensity time series.
suffer severely from practical drawbacks, such as its massive weight updates and its error gradient. Drucker’s variant may suffer from the fact that its coefficients for combining the predictors are not directly related to the performance of each predictor (if a weighted combination is used). The approach we followed may be limited theoretically and in its maximal contribution but has two advantages for the layman. One advantage is that it is simple to implement and does not affect the basic learner. The other advantage is that by observing the error distribution and choosing the threshold, one may be aware in advance of the effect this version of boosting may have on the overall performance. 6 Results 6.1 Boosting on Laser Data. We demonstrate the capabilities of the boosting algorithm on laser data from the Santa Fe times-series competition2 (data set A) (Weigend & Gershenfeld, 1993). This time series is the intensity of a NH3 -FIR laser, which exhibits Lorenz-like chaos (see Figure 2). It has a sampling noise due to the A/D conversion to 256 discrete values. The behavior of the time series may be described as having “normal” behavior and several types of “collapses.” Many models may adequately fit the “normal” behavior while failing to learn the “catastrophic” behavior (see Figure 3). The comparison of the performance is followed by a detailed analysis of the behavior of each of the estimators and the resulting median estimator, which may provide a greater sense of how this algorithm actually works. We compared the performance of standard “bagging” ensembles and boosted ensembles, all consisting of three networks. The basic learners were 2
http://www.cs.colorado.edu/ andreas/Time-Series/SantaFe.html.
Boosting Regression Estimators
513
Figure 3: Error distribution of neural net predictor on test set. Dotted lines mark the RMS error. Table 1: Normalized MSE X10−3 of Different Types of Three-Predictor Ensembles on the Laser Data. Ensemble Type
Average Predictor Median Predictor
Bagging (3)
Simple Boosting
AdaBoost-3
BOOST1
BOOST2
BOOST3
Nonweighted
Weighted
2.8 (0.3)
2.6 (0.2)
2.5 (0.2)
2.6 (0.2)
2.7 (0.3)
2.7 (0.4)
3.1 (0.3)
2.7 (0.1)
2.6 (0.2)
2.7 (0.2)
3.0 (0.3)
3.1 (0.3)
two-layer feedforward neural networks predicting the next value according to the 16 previous values that were used as the net input. The hidden layer consisted of six units. The results presented were collected using the first 8000 points (of the combined set of original training and continuation data) as the training set and the following 2000 points as test data. Table 1 compares the performance of ensembles implementing bagging, the different variants of simple boosting, and AdaBoost. For each method we presented two results: the performance of the ensemble with an averagebased combination model and its performance using a median-based combination model. The results are presented in normalized mean squared errors (NMSE). NMSE is the MSE divided by the variance across the data. (When scaled to the range [−1, 1] the laser data had mean −0.5326 and S.D. 0.3676.) The performance achieved by our ensembles is significantly better than that re-
514
Ran Avnimelech and Nathan Intrator
ported in Nix and Weigend (1995) and the better-performing participants in the Santa Fe time-series competition (Weigend & Gershenfeld, 1993): Nix and Weigend report NMSE = 0.0139 for a single predictor (they used just 1000 points—unsampled them with an FFT method with factor 32—for training.) The results presented in the competition are mostly of iterated prediction. According to the partial results presented for single-step prediction, the overall NMSE of the best method is about 0.01. The NMSE of the other competitors is significantly higher. 6.1.1 Analysis of the Behavior of the Three Estimators. Figure 4 shows how the different estimators behave on different patterns in one of the tests we performed. The estimators colored red, green, and blue, respectively, were analyzed on two groups of patterns: The easy patterns are the 50% of the data on which both the first and second estimator had an error smaller than 0.02 (this is lower than the threshold γ used in this test). The difficult patterns are the 3% patterns on which at least one of the first two patterns had an error g.t. 0.1 (this is higher than the γ used).3 6.1.2 Errors on Easy Patterns. The first estimator was trained on a data set that represented the original distribution and has very accurate predictions on these patterns (MSE ' 0.6 · 10−4 , NMSE = MSE · 7.4). The second estimator was trained on a set in which these patterns were underrepresented and had accurate predictions (MSE ' 1.2 · 10−4 ). The third estimator was trained only on more difficult patterns, so its error level on these patterns (MSE ' 9 · 10−4 ) is higher. The ensemble output is the median, so it is hardly influenced by the worst result and has an MSE slightly lower than that of the first estimator. Another effect, which is evident in these graphs, is the lack of correlation in size and sign between errors of the different estimators on the patterns in this group. 6.1.3 Errors on Difficult Patterns. The first estimator (MSE ' 0.025 on these patterns) and the second estimator (MSE ' 0.03) were influenced mainly from the “normal” patterns and err on these patterns. Because the errors of both estimators on these patterns are quite decorrelated (compared to independently trained estimators), a better estimator can influence the median. The figure also shows that some of the patterns that were estimated accurately by the first estimator were underrepresented in the train set of the second estimator, and it had big errors on them.
3 The criteria for the two groups create some of the asymmetry between the estimators because they filter according to a condition on the error of the first two estimators and not the third one, but this effect is significantly smaller than the actual asymmetry revealed in the graphs.
Boosting Regression Estimators
515
Figure 4: Errors of the three estimators on the easy patterns (top) and the difficult patterns.
516
Ran Avnimelech and Nathan Intrator
The third estimator was trained only on “difficult” patterns where its output may have a great impact on the ensemble output. Therefore, it has a better performance on such patterns (MSE ' 0.01). Its error on these patterns is almost always similar to the better estimator of the previous two, or better. The ensemble output—the median—has MSE ' 0.012 on these patterns, due to the relatively good performance of the third estimator and the relatively weak correlation between errors of the first two estimators. 6.2 Iterative Time-Series Prediction. We performed a further set of tests on the laser data. In these tests, the goal was to predict the next 16 values. The error measure was the sum of the squared errors on all 16 predicted values. The predictors used were the same 16-input, 6-hidden neural networks used previously. These networks were trained according to the single-step prediction goal. However, the reweighting used by the boosting algorithm was based on the performance of the iterative prediction. We compare the performance of ensembles constructed by the several variants of the boosting algorithm to those based on bagging and to single networks. For each ensemble, there are four possible outputs: The combination may be performed either at each step or on the final predictions, and it may use mean or median. The NMSE of the individual networks was 0.0112 ± 0.022. (The first networks in the boosting esembles were slightly better: .0095 ± 0.0037.) Table 2 compares bagging ensembles, AdaBoost ensembles, and two variants of simple Boosting (BOOST2 is meaningless in this context). The threshold used for boosting was NMSE of 0.22; typically 5% to 10% of the patterns have an error beyond this threshold, and their contribution to the total MSE is 85% to 90%. There was no significant difference between the performance of bagging as the size of the ensemble changed from two to six. We present just these two ensembles. The ensemble mean in both variants of simple boosting is typically with NMSE of order of 1 and is not presented in Table 2. The combination model uses uniform weights unless stated otherwise. For the five-predictor AdaBoost ensembles, we present two weighted combinations: using the weights specified in AdaBoost, or these weights if they are positive and 0 otherwise. These results demonstrate the advantage of boosting over standard ensembles in scenarios in which its explicit reweighting of the patterns differs from the implicit one of the individual predictors. Boosting becomes more effective than bagging, even when only two predictors are combined. Using five predictors within an ensemble, a 50% error reduction is achieved. These results also support the MSE-oriented modification we introduced to simple boosting. (The advantage of BOOST3 is greater than the standard deviation in the table implies, as it was rarely inferior to BOOST1 using the same first two predictors.) The advantage of the overall combination model over the stepwise combination may be attributed to the reweighting, although
Boosting Regression Estimators
517
Table 2: Normalized MSE X10−2 of Different Types of Ensembles on the Iterated Prediction Task. Combination Model Combination Model Bagging—2 nets Boosting—2 nets Bagging—6 nets AdaBoost—3 nets BOOST1 BOOST3 AdaBoost—4 nets AdaBoost—5 nets AdaBoost—5 nets weighted averages AdaBoost—5 nets positive weights
Step-wise Step-wise Mean Median 8.2 (1.0) 6.9 (1.1) 8.5 (0.8) 9.0 (0.5) 5.5 (0.8) 5.6 (1.0) — 6.7 (1.4) — 5.8 (1.4) 5.3 (1.4) 5.0 (1.4) 5.0 (1.1) 4.8 (1.3) 6.6 (1.6) 9.3 (3.9) 6.5 (1.6)
9.5 (3.7)
Overall Overall Mean Median 7.8 (1.2) 6.3 (1.3) 7.4 (0.3) 7.9 (0.7) 4.8 (0.6) 5.0 (1.2) — 6.4 (0.8) — 5.4 (1.0) 4.5 (1.0) 4.3 (1.1) 4.2 (0.8) 4.3 (1.0) 5.5 (1.6) 9.4 (3.7) 5.4 (1.6)
9.5 (3.7)
a smaller but similar effect exists in bagging too. Median is the combination model for simple boosting, but in AdaBoost it has no advantage over mean (in bagging, mean is better). Our tests also found simple averaging (mean or median) to outperform the weighted versions. This is due to the large coefficients of the earlier predictors (especially the first one), while the performance of the different predictors is similar (actually, for the first few predictors, there is a gradual improvement). The median was usually the first predictor. The mean was less affected. It clearly outperforms bagging but is inferior to the nonweighted mean. 6.3 Mackey-Glass Time Series. The Mackey-Glass differential-delay equation (Mackey & Glass, 1977), x(t − τ dx(t) = −bx(t) + a d(t) 1 + x(t − τ )1 0
(6.1)
and the time series resulting from its integration have attracted much focus as a statistical learning benchmark (Moody, 1989; Crowder, 1990). We performed tests on a data set of this time series in the CMU repository.4 This specific time series was generated with τ = 17, a = 0.2, and b = 0.1. The input data used are x(t − 18), x(t − 12), x(t − 6), x(t), and the task is to predict x(t + 6). Training data consist of 3000 data points, and the test set consists of 500 data points, starting 1800 time steps after the end of the training data. We compared the performance of ensembles constructed by the AdaBoost algorithm to those constructed by bagging. The basic learner we 4
http://www.boltz.cs.cmu.edu/benchmarks/mackey-glass.html
518
Ran Avnimelech and Nathan Intrator
Table 3: Normalized RMS X10−2 of different types of ensembles on MackeyGlass data Ensemble-Type
Bagging
Combination Method 3 predictors 4 predictors 5 predictors
Median
Mean
AdaBoost Weighted Weighted Nonweighted Nonweighted Median Mean Median Mean
1.30 (0.07) 1.25 (0.04) 1.28 (0.04) 1.13 (0.03) 1.24 (0.03) 1.23 (0.03) 1.22 (0.05) 1.10 (0.01) 1.24 (0.04) 1.22 (0.04) 1.21 (0.05) 1.08 (0.02)
1.20 (0.04) 1.12 (0.02) 1.09 (0.02)
1.13 (0.02) 1.10 (0.01) 1.07 (0.01)
used was a two-layer neural network with 20 hidden units. Such a learner achieved normalized RMS of 0.014, which compares favorably with other results achieved on these data. Table 3 shows the normalized RMS of ensembles constructed by bagging and by AdaBoost, using either the average or the median as the ensemble output. For AdaBoost we present the performance of both the weighted and nonweighted median/average. The normalized RMS of simple boosting (three predictors) was 0.0124 ± 0.0004 using the median as the output and 0.0148 ± 0.0012 using the mean. (This result is inferior to AdaBoost and similar to Bagging.) These results show the advantage of AdaBoost over bagging. They also demonstrate that the nonweighted averages may be at least as good as the weighted averages and that the mean may be as least as good as the median. The relatively good performance of the nonweighted versions is due to the limit of the BER in indicating the quality. The BER provides some measure of the relative performance of each estimator, but because these coefficients ignore the finer details of the error distribution, they lack an advantage over a simple average. The relatively poor performance of the weighted mean results from the fact that in some cases, it is exactly the first predictor, due to its higher coefficient. The dichotomy between these cases and the cases in which all the predictors are used also leads to the higher variance in its performance. 7 Discussion This work reviews the extension of the boosting algorithm to fit regression problems and focuses on a threshold-based application of boosting for regression. This method is designed to fit tasks in which poor performance is due to the effect of difficult patterns, unfitted by the model. Various tasks, for which large data sets are available, exhibit this behavior and may gain from this new procedure. The basic principle of this method is regarding “big” estimation errors as “classification-like” errors and implementation of a mechanism to reduce their amount.
Boosting Regression Estimators
519
We focus on the practical aspects of boosting in regression. The model for extending the algorithm is based on an analogy of regression and classification errors. While this leads to a possible algorithm that reduces the number of errors beyond a given threshold and consequently reduces the MSE, certain minor modifications may be more appropriate for reducing the MSE. We also present a comparison between this method and other versions of boosting in regression (Freund & Schapire, 1995; Drucker, 1997). Although this comparison does not lead to a conclusive choice of one of these versions as superior to others, it emphasizes the advantages and drawbacks of each method. Other methods may be more appealing theoretically, but they seem to have practical drawbacks. The results achieved on the laser data and the Mackey-Glass time series demonstrate the potential of decorrelating the errors of different predictors, using threshold-based boosting, and combining them in a robust manner. Nonweighted averages and the mean of the predictors, rather than the median (in AdaBoost), may in many cases perform as least as good as the weighted median specified in the theoretical model. This is due to the fact that the weights are based on the BER rather than the MSE. The tests performed on an iterative prediction task emphasize the advantage of using boosting when the goal presented to the individual predictors does not fully represent the real goal. In such cases, the boosting mechanism may contribute to the learning, something that is not handled by the individual predictor, thus being more effective. Our analysis of the behavior of different estimators on different kinds of patterns (for simple boosting) provides insight into the way error reduction is achieved. The performance of the third estimator on easy patterns is not as good as of the first two, but it has almost no influence on the median. On the difficult patterns, however, it has a lower error rate; thus, the median will usually have either the smaller of the two errors or some intermediate value when these errors have different signs. References Breiman, L. (1996a). Bagging predictors. Machine Learning, 24 (TR-421), 123–140. Breiman, L. (1996b). Bias, variance and arcing classifiers (Tech. Rep. TR-460). Berkeley: Department of Statistics, University of California, Berkeley. Breiman, L. (1997). Arcing the edge (Tech. Rep. TR-486). Berkeley: Department of Statistics, University of California, Berkeley. Crowder, S. (1990). Predicting the Mackey-Glass timeseries with cascade correlation learning. In Connectionist Models: Proceedings of the 1990 Summer School. Drucker, H. (1997). Improving regressors using boosting techniques. In 14th International Conference on Machine Learning. San Mateo, CA: Morgan Kaufmann. Drucker, H., Schapire, R., & Simard, P. (1993). Improving performance in neural networks using a boosting algorithm. In S. J. Hanson, J. D. Cowan, &
520
Ran Avnimelech and Nathan Intrator
C. L. Giles (Eds.), Advances in neural information processing systems, 5 (pp. 42– 49). San Mateo, CA: Morgan Kaufmann. Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. New York: Chapman & Hall. Freund, Y. (1995). Boosting a weak learning algorithm by majority. Information and Computation, 121(2), 256–285. Freund, Y., & Schapire, R. (1995). A decision-theoretic generalization of on-line learning and an application to boosting. In 2nd European Conference on Computational Learning Theory. Geman, S., Bienenstock, E., & Doursat, R. (1992). Neural networks and the biasvariance dilemma. Neural Computation, 4, 1–58. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation, 3(1), 79–87. Mackey, M., & Glass, L. (1977). Oscilations and chaos in physiological control systems. Science, (197). Meir, R. (1995). Bias, variance and the combination of least square estimators. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 295–302). Cambridge, MA: MIT Press. Moody, J. (1989). Fast learning in multi-resolution hierarchies. In Advances in neural information processing systems, 1 (pp. 29–39). San Mateo, CA: Morgan Kaufmann. Nix, D. A., & Wiegend, A. S. (1995). Learning local error bars for nonlinear regression. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 489–496). Cambridge, MA: MIT Press. Raviv, Y., & Intrator, N. (1996). Bootstrapping with noise: An effective regularization technique. Connection Science, 8(3/4), 355–372. Schapire, R. (1990). The strength of weak learnability. Machine Learning, 5(2), 197–227. Schapire, R., Freund, Y., Bartlett, P., & Lee, W. (1997). Boosting the margin: A new explanation for the effectiveness of voting methods. In Machines That Learn—Snowbird. Schwenk, H., & Bengio, Y. (1997). Adaptive boosting of neural networks for character recognition (Tech. Rep. TR-1072). Montreal: Department d’Informatique et Recerche Operationnelle, Universit´e d’Montreal. Valiant, L. G. (1984). A theory of the learnable. Communications of the ACM, 27, 1134–1142. Waterhouse, S. R., & Cook, G. (1997). Ensemble methods for phoneme classification. In M. Mozer, M. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems, 9. Cambridge, MA: MIT Press. Weigend, A. S., & Gershenfeld, N. A. (Eds.). (1993). Time series prediction: Forecasting the future and understanding the past. Reading, MA: Addison-Wesley. Received June 5, 1997; accepted April 20, 1998.
LETTER
Communicated by John Platt
An On-Line Agglomerative Clustering Method for Nonstationary Data Isaac David Guedalia Center for Neural Computation and Institute of Computer Science, Hebrew University, 91904 Jerusalem, Israel
Mickey London Institute of Life Sciences and Center for Neural Computation, Hebrew University, 91904 Jerusalem, Israel
Michael Werman Institute of Computer Science, Hebrew University, 91904 Jerusalem, Israel
An on-line agglomerative clustering algorithm for nonstationary data is described. Three issues are addressed. The first regards the temporal aspects of the data. The clustering of stationary data by the proposed algorithm is comparable to the other popular algorithms tested (batch and on-line). The second issue addressed is the number of clusters required to represent the data. The algorithm provides an efficient framework to determine the natural number of clusters given the scale of the problem. Finally, the proposed algorithm implicitly minimizes the local distortion, a measure that takes into account clusters with relatively small mass. In contrast, most existing on-line clustering methods assume stationarity of the data. When used to cluster nonstationary data, these methods fail to generate a good representation. Moreover, most current algorithms are computationally intensive when determining the correct number of clusters. These algorithms tend to neglect clusters of small mass due to their minimization of the global distortion (Energy). 1 Introduction 1.1 Scale. Cluster analysis is the process of finding the intrinsic structure in a data set without relying on a priori knowledge. Given a data set and some measure of distance, or similarity, between data points, the goal in most clustering algorithms is to assign each data point (pattern) to a cluster “such that the patterns in a cluster are more similar to each other than to patterns in different clusters” (Jain & Dubes, 1988). However, the structure determined by the measure of similarity is a function of scale. While two data points at a high resolution may seem very different, when viewed at a lower resolution they appear similar. Neural Computation 11, 521–540 (1999)
c 1999 Massachusetts Institute of Technology °
522
Isaac David Guedalia, Mickey London, and Michael Werman
Figure 1: Example of scale-dependent intrinsic structure. The data in this figure are composed of nine gaussian clusters. Each cluster contains 1000 points, except the right cluster of the upper three, which contains 2000 points. Note how the data can be grouped into either three or nine clusters.
Figure 1 is an example of a data set that has at least two apparent scales. If the data points in the left corner are analyzed in isolation (at high resolution), they appear as three clusters. However, the same data when viewed in the larger picture are part of a single larger cluster. Hence, the answer to the question, “How many clusters are there ?” in this data set is twofold (either three or nine). The “correct” answer is application dependent. Moreover, finite resources may limit the possible computable answers. Clustering algorithms that minimize the global distortion1 using a fixed number of centroids (see Jain & Dubes, 1988; Duda & Hart, 1973) ignore scale-dependent structures. Thus, in the previous example, Linde, Buzo, and Gray (1980), using 12 centroids, for example, would find 12 clusters, which does not capture the structure of the data (3 or 9). Many algorithms address this problem. Sebestyen (1962) used a thresholdbased adaptive approach to determine the number of clusters. MacQueen’s K-means algorithm (MacQueen, 1967) solves this issue by using two external parameters to define the coarseness and refinement of the clustering. Similarly, ISODATA (Ball & Hall, 1967), a batch algorithm, adjusts the number of clusters with an external threshold. A different approach taken 1
See Linde et al. (1980) for an extensive discussion of different measures of distortion.
On-Line Agglomerative Clustering Method
523
follows the minimum description length (MDL) criteria (Rissanen, 1989). This approach tries to minimize the total cost of the representation of the data when the cost is a parametric function of the distortion and of the model’s complexity (Gath & Geva, 1989; Fritzke, 1994; Buhmann & Kuhnel, 1993). However, although these methods find an “optimal” solution, the number of centroids in the final representation depend on an external parameter. This parameter’s effect on the outcome of the clustering must be determined experimentally, and small perturbations in either the parameter or the data can result in drastically different solutions. Another approach, which stems from statistical mechanics, uses a pseudotemperature to escape local minima in the energy (distortion) function (Rose, Gurewitz, & Fox, 1990). This approach presents a natural solution to the problem of scale-dependent structures. The clustering process, proposed by Rose et al., consists of a cooling schedule in which the pseudotemperature is lowered and a solution at each temperature is found. During this process, the energy function undergoes something similar to phase transitions. Each such transition reflects a scale-dependent solution. 1.2 Stationarity. Clustering algorithms can be divided into two classes: batch and on-line. Batch algorithms process the data off-line; hence, the temporal structure is ignored. Similarly, current on-line algorithms assume the data are produced by a stationary process2 (i.e., randomly drawn). In this situation the data can be sampled and clustered with a batch algorithm. There exist many real-world problems in which the data are produced by a certain type of nonstationary process. If a statistical sample of the data can be stored, then current algorithms can be used to cluster the data using either a batch method or an on-line method. However, this may require computational resources that are not always available. We address a set of problems that share the following property: on a short time scale, it is pseudostationary, while on the long time scale, the process has a sequential property. For example, in Figure 10, nine clusters of data were produced sequentially. The points in each cluster were generated in a stationary process. First, all the points from the first cluster arrived randomly. This is followed by the random arrival of all the points in the second cluster, and so on. In this example, the short time scale is the number of points in each cluster. The long time scale is the whole process.
2
The process of clustering involves an exposure to data points one at a time. This process can be viewed as a discrete-time real-valued stochastic process. Let t = 1, 2, 3, . . . be the time steps of points arrival, and let xt be a d-dimensional point. The sequence {xt } is a stochastic process. This process is a stationary process iff the joint distribution functions of (xt1 +h , xt2 +h , . . . , xtn +h ) and (xt1 , xt2 , . . . , xtn ) are the same for all h = 0, 1, 2, . . . and an arbitrary selection of t1 , t2 , . . . , tn .
524
Isaac David Guedalia, Mickey London, and Michael Werman
1.3 Small Clusters. Given a set of data that includes a few small, distinct clusters, how can the structure of the data be encoded such that the small clusters are represented? Existing algorithms that minimize the global distortion have the following dilemma: Either the clustering is performed at a high resolution, resulting in an overfitting, or a low-resolution clustering misses the small clusters. This is due to one of the following two reasons. If a batch method is used, then the effect the small clusters have on the global distortion is diluted by the larger clusters. Similarly, if the data are generated by a stationary process, on-line methods will have the same problem of dilution. Alternatively, if the data are produced by a nonstationary process, the problem becomes how to recognize that a new process began (arrival of a data from a new cluster) and to allocate a centroid to represent it.3 The ART1 algorithm presents a solution to this problem (Carpenter & Grossberg, 1990). Buhmann and Kuhnel (1993) have proposed batch and on-line clustering algorithms that minimize a complexity term composed of the global distortion and the scale (complexity) of the model. The complexity term helps to solve the previous dilemma by increasing the effect that distant points have on the system and minimizing the overfitting of the larger clusters. Unfortunately, the tuning of the scale parameter is very difficult. Moreover, the on-line algorithm assumes the stationarity of the data. 1.4 Example. As an example of a real-world application concerned with the issues mentioned, one can consider the problem of quality control of fruit. The problem is how to classify a fruit into a quality class, based on a series of feature vectors measured from the fruit. One solution is to use a sample of fruits, cluster their feature vectors, and correlate their features with the predefined quality classes. Then use the relationship between the clusters and the classes to classify the fruits. Due to the huge amount of data needed, an on-line method should be used, but stationarity of the data cannot be assumed. Some features of the fruits (for example, weather damages) tend to occur in bursts; for example, fruit that is damaged by a cold spell will appear at intervals determined by the weather. These features, which correlate with damages, are very meaningful for classifying the fruit, and although very distinct, they are infrequent. Thus, the problem of quality control encapsulates the three issues raised: the data is nonstationary, there exist small, meaningful clusters, and the structure is scale dependent. The proposed algorithm uses a novel approach toward such cases. The basic idea is that each point of data can belong to a new cluster. Thus, a new centroid is placed on each and every new point. Due to the limitation of finite memory, this implies that a centroid must be allocated at the cost
3 In such situations, once the centroid is placed, it will continue to represent the cluster even though it is relatively small. This is due to its relatively distant location.
On-Line Agglomerative Clustering Method
525
of the existing representation (centroids). This is done by merging the two closest centroids into one, at every step, minimizing the necessary loss of information. The resulting algorithm does not neglect small clusters, regardless if the data are produced by a stationary process. Furthermore, if a small cluster is distinct enough, it will not be lost by being merged into an existing cluster. Finally, if the data point was distinct but no other points were close enough to it to be merged with it (e.g., distant noise), the centroid can be removed at the end of the process (revealed by a very small weight). 2 Proposed On-Line Algorithm The proposed algorithm is simple and fast. The algorithm can be summarized in the following three steps: For each data point arriving: 1. Move the closest centroid toward the point. 2. Merge the two closest centroids. This results in the creation of a redundant centroid. 3. Set the redundant centroid equal to the data point. The algorithm can be understood as follows: Three criteria are addressed at each time step: minimization of the within-cluster variance, maximization of the distances between the centroids, and adaptation to temporal changes in the distribution of the data. In the first step, the within-cluster variance is minimized by updating the representation in a manner similar to the K-means algorithm (MacQueen, 1967). The second step maximizes the distances between the centroids by merging the two centroids with the minimum distance (not considering their weight). The merging is similar to most agglomerative methods (see Sneath & Sokal, 1973, for a review and Wong, 1993, for a recent paper). Finally, temporal changes in the distribution of the data are anticipated by treating each new point as an indication to a potential new cluster. The detailed description of the proposed algorithm for on-line clustering follows (note that we follow the notation used by Buhmann & Kuhnel, 1993). For each centroid α, let yα be the location and cα the counter (the number of points this centroid represents) of the centroid. The scale of the desired solution is specified by the maximum number of centroids available (i.e., size of memory). We denote this parameter as Kmax . The number of centroids participating in the final solution may be less than Kmax due to the postprocessing described below. Thus, the true structure of the data is revealed by the remaining centroids: 1. Initialize the system with zero centroids: K = 0. 2. Get data point x.
526
Isaac David Guedalia, Mickey London, and Michael Werman
3. The centroid closest to the data point is defined as the winner: winner = α s.t. kyα − xk is minimal 4. Update the location of the closest centroid and its counter, that is, compute the running average: ywinner ← ywinner +
x − ywinner cwinner + 1
cwinner ← cwinner + 1 5. If there remains free memory, allocate a new centroid, that is, if K < Kmax then K ← K + 1, set δ ← K. Go to step 8. 6. Find the redundant pair of centroids—the two centroids whose representation of the data is most similar (closest to each other): {γ , δ} = argmin kyγ − yδ k γ ,δ,γ 6=δ
7. Merge the two redundant centroids by computing their weighted average location and cumulative number of points (counter): yγ ←
yγ cγ + yδ cδ cγ + cδ
cγ ← cγ + cδ 8. Initialize the new centroid with the last data point. It may indicate the start of a new process (the arrival of a new cluster of data). yδ = x; cδ = 0 9. While there remains data to be clustered, go to step 2. 10. Postprocess: Remove all clusters with a negligible weight: ∀α if cα < ² perf orm steps 6 and 7 with δ ≡ α, (Kmax ← Kmax − 1) This algorithm can cluster the data in a single pass, with performance (minimization of the global distortion) comparable to existing clustering algorithms running in batch mode. Moreover, the proposed algorithm follows new data while preserving the existing structure; even small clusters are represented. The next section presents results of two different sets of simulations. The first set of simulations demonstrates the robustness of the algorithm and quantitatively compares the proposed clustering algorithm to a popular batch algorithm (deterministic annealing) and two on-line methods (Kmeans and EquiDistortion). The results indicate that the new algorithm’s performance in minimizing the global distortion (Energy) is comparable to the other methods. This is true even though the proposed algorithm clusters on-line nonstationary data (K-means fails completely to cluster the
On-Line Agglomerative Clustering Method
527
nonstationary processes). Furthermore, we introduce a new measure of performance, the local distortion. Results from these experiments demonstrate the superior performance of the new algorithm in minimizing the local distortion (i.e., representing the smaller clusters). The second set of experiments, is an example of how the proposed algorithm determines the solution given an indication of the desired scale. 3 Results of Simulations 3.1 Quantitative Analysis: Random-Generated Clusters. To analyze quantitatively the performance of the proposed algorithm, a series of randomly generated gaussian mixtures were generated. Four different methods were compared: K-means (MacQueen, 1967) (an on-line method), EquiDistortion (Ueda & Nakano, 1994) (modified to be on-line) deterministic annealing (Rose et al., 1990) (a batch method), and the proposed algorithm (AddC). The K-means and deterministic annealing method were chosen to represent baseline performance of an on-line and batch method. These algorithms determine their representation of the data by moving their K centroids, no merging or splitting is performed. The EquiDistortion method merges and splits centroids as a function of their relative variance; centroids with a relatively large variance are split, and those with a relatively small variance are merged. The EquiDistortion method was modified to run in an on-line mode (Guedalia, Werman, & Edan, 1995). The data were presented to the on-line algorithms in either a stationary process or a nonstationary fashion. The nonstationary process has the following feature: on a short time scale, it is random, while on the long time scale, the process has a sequential property. For example, the data in Figure 10 have nine small clusters, which were produced sequentially. The points in each cluster were generated in a stationary process. Thus, all the points from the first cluster arrived randomly, followed by the points in the second cluster, and so on. The number of centroids was equal to number of clusters. Deterministic annealing ran with β = 1 through β = 11,357.8 incremented by 10%. At each β step, the system ran until convergence (maximum 30 epochs). Experimentally it was noted that at most β steps, convergence occurred relatively early. It is worth noting that β = 11,357.8 was not large enough to be considered infinity (we stopped at this value due to lack of computing resources). The number of gaussian mixtures generated was systematically varied from 5 through 24. Ten sets of data were generated for each of the different cases. The data were divided into a training set and test (generalization) set. All results were averaged over 10 runs. Each of the gaussian mixtures had a randomly generated number of points and shape. After the training data were clustered by the different methods, the global and local distortion was
528
Isaac David Guedalia, Mickey London, and Michael Werman
Figure 2: A comparison of the performance of the different stationary methodologies tested. Plot of the energy of the system (averaged over 10 runs) as a function of the different data sets (i.e., different numbers of clusters). The solutions found for one instance of 16 clusters are depicted in Figure 4. The deterministic annealing (RGF) method would have reached a lower energy throughout had the process not been stopped early.
measured on the test set. The global distortion was calculated as follows: S 1X min kyα − xi k, S i=1 α
where S is the size of the data set and the “distance” is computed as the sum of squares. 3.1.1 Global Distortion. Figures 2 and 3 present the global distortion as a function of the number of gaussian mixtures generated. The deterministic annealing energy would probably approach the K-means given more time (β = ∞). An example of the results can be seen in Figures 4 and 5. The proposed method succeeds in approaching batch results—in minimization of the global distortion—even though it clustered the data in a single sequential pass. Moreover, it better preserved the representation of the data by allocating centroids for the small, distant clusters. Figures 6 and 7 present the global distortion as a function of the dimension of the data with nonstationary and stationary data, respectively. In this situation as well, the proposed method succeeds in approaching batch results: minimization of the global distortion.
On-Line Agglomerative Clustering Method
529
Figure 3: A comparison of the performance of the different sequential methodologies tested. Plot of the energy of the system (averaged over 10 runs) as a function of the different data sets (i.e., different numbers of clusters). The solutions found for one instance of 16 clusters are depicted in Figure 5.
3.1.2 Local Distortion. While the global distortion provides a measure of the average performance, it is not a good measure of the quality of the representation of each individual cluster. Hence, the local distortion is determined as follows: N X 1 X min kyα − xk, α S n=1 n x∈C n
where N is the number of clusters generated, Cn is the nth cluster, Sn the number of points in Cn , and the distance kyα − xk is the sum of squares. The distortion of each point is the distance between the point and its most representative centroid, normalized by the size of its originating cluster. This ensures that the effect each cluster has on the performance measure is relatively equal. Even small clusters influence the final result. Figures 8 and 9 graph the local distortion (averaged over 10 runs) as a function of the number of clusters. The K-means and deterministic annealing methods, which minimize the global distortion (β = ∞), perform relatively poorly. This is because they ignore small clusters even if they are quite distinct. As the number of clusters increase, the effect of missing a single cluster is diminished. By preserving the small, distant clusters, the proposed method also minimizes the local distortion. 3.1.3 Stationarity. The on-line methods were tested on data that were presented once in a pseudostationary (random) mode and once in a non-
530
Isaac David Guedalia, Mickey London, and Michael Werman
Figure 4: An example of clustering of randomly generated stationary data by four different methods, Proposed method: Add constantly (AddC), K-means, EquiDistortion, and deterministic annealing (RGF). Note how the K-means and EquiDistortion methods missed two small clusters in the center left section (the deterministic annealing missed one of them and another one at the bottom right). This contributes to the relatively high local distortion of these methods as compared with the proposed method.
stationary (sequential) mode. While the K-means method successfully clustered the stationary data, it failed to capture the structure of the nonstationary data. The reason for its poor performance is demonstrated in Figure 5. The K-means method follows the arrival of the latest set of data. Hence, most of the centroids are located within the central cluster. This is in contrast to the performance of the proposed method in clustering both the stationary and nonstationary data. 3.2 Scale dependence. To demonstrate the algorithm’s ability to follow the structure as a function of scale, the data from Figure 1 were clustered with the new algorithm. Figure 10 depicts the clustering of the data while constraining the memory to four centroids. Four stages in the process are presented, after the presentation of the first 1000, 3000, 6000, and 10,000 data points. In the first stage, all the centroids are placed on the existing data. Next, the centroids represent the three clusters that exist in the bottom-
On-Line Agglomerative Clustering Method
531
Figure 5: An example of clustering of randomly generated nonstationary data by three on-line methods, Proposed method: Add constantly (AddC), K-means, and EquiDistortion. Note how the proposed method successfully clusters the data even though they are presented in a sequential fashion. Furthermore, the solution found by the proposed method here is virtually identical to the solution obtained when the data are processed randomly (see Figure 4).
right corner. The introduction of data at a relatively large distance from the previous data modifies the perspective. Hence, the previously subdivided clusters are merged into a single large cluster. The final representation of the data with four centroids uses three of the centroids, placing them in the center of mass of each group of data. The fourth centroid represents the last data point and should be merged into the system. In comparison, Figure 11 presents the results when using 10 centroids. Similar to the previous example, the first stage places all the centroids on the existing data. After 3000 data points arrive, the local structure is revealed; the data are properly represented by 3 centroids, with the other 7 appearing as satellites around the extremities. These centroids are allocated in the following stages. In the final stage (after the arrival of all 10,000 points) the local structure is preserved due to the relatively large number of centroids. Here again the extra centroid is needed to follow the last data point to arrive. Note that the nonstationarity in the final example is not a necessary condition for the final solution.
532
Isaac David Guedalia, Mickey London, and Michael Werman
Figure 6: A comparison of the performance of the different nonstationary methodologies tested. Plot of the global distortion as a function of the dimensionality of the data. Ten gaussian clusters were generated with dimensions 5 through 40 at increments of 5. The number of points in each cluster was fixed.
Figure 7: A comparison of the performance of the different stationary methodologies tested. Plot of the energy (global distortion) of the system (averaged over 10 runs) as a function of the dimensionality of the data. Ten gaussian clusters were generated with dimensions 5 through 40 at increments of 5. The number of points in each cluster was fixed.
On-Line Agglomerative Clustering Method
533
Figure 8: A comparison of the performance of the different sequential methodologies tested with respect to small clusters. Plot of the local distortion of the system (averaged over 10 runs) as a function of the different data sets (i.e., different numbers of clusters). The solutions found for one instance of 16 clusters are depicted in Figure 5. The proposed algorithm (AddC) succeeds in preserving the representation of even the small clusters. This is due to their relatively large distance from other clusters.
Figure 9: A comparison of the performance of the different stationary methodologies tested. Plot of the local distortion of the system (averaged over 10 runs) as a function of the different data sets (i.e., different numbers of clusters). The solutions found for one instance of 16 clusters are depicted in Figure 4. The proposed algorithm (AddC) succeeds in preserving the representation of even the small clusters. This is due to their relatively large distance from other clusters.
534
Isaac David Guedalia, Mickey London, and Michael Werman
Figure 10: Sequential presentation of data from Figure 1. Four stages of the clustering by the proposed algorithm are presented. Kmax = 4.
Perhaps the most important aspect of the algorithm is its relative insensitivity to the exact choice of Kmax . In other words, one should specify only the order of magnitude of Kmax . This is demonstrated in Figure 12. A single gaussian centroid (stationary) was clustered with Kmax equal 2 through 7. After the clustering process, all centroids that represented less than 0.5% of the number of points were merged. Figure 13 graphs the Energy (global distortion) as a function Kmax . The effect of increasing Kmax is negligible until a “phase transition” occurs and a split. The reasoning behind this is as follows. Assume a single gaussian cluster of data that arrives in a stationary process. Let us assume Kmax is equal to 3. Assume that it has been correctly clustered, and we will label the centroids µ, ν, and ξ , where ξ is the actual center (mean). When a new data point arrives, it forces the merging of the two closest centroids. In order for the centroids in the periphery to accumulate points, they must merge with each other. However, since the probability that the distance between µ and ν is smaller than the distance between ξ and either µ or ν is small, it is more likely that they will merge with the ξ , hence, strengthening the center and weakening the periphery. For the centroids on the periphery to
On-Line Agglomerative Clustering Method
535
Figure 11: Sequential presentation of data from Figure 1. Four stages of the clustering by the proposed algorithm are presented. Kmax = 10.
Figure 12: An example of the lack of sensitivity of the proposed algorithm to the choice of Kmax . A single gaussian centroid (stationary) was clustered with Kmax equal 2 through 7. After the clustering process, all centroids that represented less than 0.5% of the number of points were merged (postprocessing).
536
Isaac David Guedalia, Mickey London, and Michael Werman
Figure 13: Plot of the energy (global distortion) as a function of the addition of Kmax . A single gaussian cluster was clustered with the proposed method at different Kmax . Next, all centroids that represented less than 0.5% of the total number of points were merged. This was averaged over 10 runs. The energy (global distortion) function demonstrates that there is a clear plateau in which there is no change in the solutions found. This is in contrast to methods that minimize the energy and would use all the centroids available.
have a large mass, they must be closer to each other than to the center (an unlikely event) and this must occur for many time steps consecutively (a very unlikely event). Figure 14 presents a measure of order which quantifies this process. This process of phase transitions is similar to the one described by Rose et al. (1990). Figures 15 and 16 present the results of clustering the same data using the deterministic annealing algorithm. Note the similarity of the behavior of the Energy function in Figures 13 and 16. 4 Summary and Conclusions Yet another clustering classifier? The proposed algorithm is the first explicitly to address the issue of on-line clustering nonstationary data. The method can be seen as an extension of the work presented by Buhmann and Kuhnel (1993) or an on-line version of the clustering by melting algorithm presented by Wong (1993) in which each data point is assigned a centroid. Quantitative analysis of the new algorithm performance in clustering simulated data demonstrated its superior performance in minimizing the local distortion and comparable performance in minimizing the global distortion to existing clustering algorithms. This is even more pronounced when clustering nonstationary data.
On-Line Agglomerative Clustering Method
537
Figure 14: Plot of order parameter as a function of Kmax . A single 2D or 3D gaussian with 1 million data points was presented randomly and clustered with the proposed method at different Kmax ’s. The following order parameter was calcuK max X cα lated: 1 − ( PKmax )2 . The order parameter shows clear phase transitions α=1 γ =1 cγ indicating the method’s robustness to Kmax The phase transition indicates a sudden change in the number of nonredundant centroids. Furthermore, as the dimensionality increases, this occurs at a larger Kmax .
Unfortunately, the new algorithm is sensitive to data that include drastically different scales. For example, if the data seen in Figure 1 are corrupted with noise (a very wide gaussian placed in the center of the data), performance drops (see Figures 17 and 18). The proposed method attaches equal importance to every point. Each new point is potentially the beginning of a new cluster. The solution to this is to assume knowledge of the time scale of the smallest process and further assume that the smallest process is larger than a certain threshold. Then after each time step, merge all centroids whose counter is below the threshold. Currently the algorithm is being tested on the difficult problem of quality control of agricultural produce. Preliminary results indicate that the algorithm shows significantly better results than other on-line clustering algorithms. 5 Acknowledgments We thank Haim Sompolinksy and Yael Edan for their help in preparation of this article. This research was supported by Binational Agricultural and Research Development Fund No. US-1992-91 and partially supported by the Paul Ivanier Center for Robotics Research and Production Management.
538
Isaac David Guedalia, Mickey London, and Michael Werman
Figure 15: Clustering of data from Figure 1 by deterministic annealing. Four stages of the clustering at different β are presented.
Figure 16: Energy as a function of time (decreasing temperature) during the deterministic Annealing clustering of the data from Figure 1. The four stages of the clustering depicted in Figure 15 are noted by vertical lines.
On-Line Agglomerative Clustering Method
539
Figure 17: Plot of the energy (global distortion) as a function of the addition of noise. The data from Figure 1 with the addition of noise were clustered with either K-means or AddC. Note how the AddC method immediately reacts to the addition of noise, while the K-means method slowly degrades.
Figure 18: Performance of Kmeans and AddC in clustering data from Figure 1 with the addition of noise. Note, how the AddC method immediately reacts to the addition of noise, while the Kmeans method slowly degrades.
540
Isaac David Guedalia, Mickey London, and Michael Werman
Please address correspondence to either
[email protected] or
[email protected]. A demo program of the AddC algorithm is available from: ftp://lobster.ls.huji.ac.il/pub/mikilon/Cluster/ addcdemo.zip. References Ball, G., & Hall, D. (1967). A clustering technique for summarizing multivariate data. Behavioral Science, 12, 153–155. Buhmann, J., & Kuhnel, H. (1993). Complexity optimized data clustering by competitive neural networks. Neural Computation, 5, 75–88. Carpenter, G. A., & Grossberg, S. (1990). Adaptive resonance theory: Neural network architectures for self-organizing pattern recognition. In R. Eckmiller, G. Hartmann, and G. Hauske (Eds.), Parallel processing in neural systems and computers (pp. 383–389). Amsterdam: North-Holland. Duda, R. O., & Hart, P. E. (1973). Pattern classification and scene analysis. New York: Wiley. Fritzke, B. (1994). Growing cell structures—A self-organizing network for unsupervised and supervised learning. Neural Networks, 7, 1441–1460. Gath, I., & Geva, A. B. (1989). Unsupervised optimal fuzzy clustering. IEEE Trans. on Pattern Analysis and Machine Intelligence, 11, 773–781. Guedalia, I. D., Werman, M., & Edan, Y. (1995). A new method for on-line clustering of sparse data (ASAE Paper No. 95-3606). Jain, A. K., & Dubes, R. C. (1988). Algorithms for clustering data. Englewood Cliffs NJ: Prentice Hall. Linde, Y., Buzo, A., & Gray, R. M. (1980). An algorithm for vector quantizer design. IEEE Trans. On Communications, 28(1), 84–95. MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. Proc. 5th Berkeley Symp. Mathematical Statist. Probability (pp. 281–297). Rissanen, J. (1989). Stochastic complexity in statistical inquiry. Singapore: World Scientific. Rose, K., Gurewitz, E., & Fox, G. (1990). A deterministic annealing approach to clustering. Patt. Rec. Letters, 11(4), 589–594. Sebestyen, G. S. (1962). Pattern recognition by an adaptive process of sample set construction. IRE Trans. Info. Theory, 8, S82–S91. Sneath, P. H. A., & Sokal, R. R. (1973). Numerical taxonomy. San Francisco: W. H. Freeman. Ueda, N., & Nakano, R. (1994). A new competitive learning approach based on an equidistortion principle for designing optimal vector quantizers. Neural Networks, 7(8), 1211–1227. Wong, Y. (1993). Clustering data by melting. Neural Computation, 5, 89–104. Received June 5, 1996; accepted February 20, 1998.
LETTER
Communicated by Christopher Williams
Hidden Neural Networks Anders Krogh Center for Biological Sequence Analysis, Building 208, Technical University of Denmark, 2800 Lyngby, Denmark
Søren Kamaric Riis Department of Mathematical Modeling, Section for Digital Signal Processing, Technical University of Denmark, Building 321, 2800 Lyngby, Denmark
A general framework for hybrids of hidden Markov models (HMMs) and neural networks (NNs) called hidden neural networks (HNNs) is described. The article begins by reviewing standard HMMs and estimation by conditional maximum likelihood, which is used by the HNN. In the HNN, the usual HMM probability parameters are replaced by the outputs of state-specific neural networks. As opposed to many other hybrids, the HNN is normalized globally and therefore has a valid probabilistic interpretation. All parameters in the HNN are estimated simultaneously according to the discriminative conditional maximum likelihood criterion. The HNN can be viewed as an undirected probabilistic independence network (a graphical model), where the neural networks provide a compact representation of the clique functions. An evaluation of the HNN on the task of recognizing broad phoneme classes in the TIMIT database shows clear performance gains compared to standard HMMs tested on the same task. 1 Introduction Hidden Markov models (HMMs) is one of the most successful modeling approaches for acoustic events in speech recognition (Rabiner, 1989; Juang & Rabiner, 1991), and more recently they have proved useful for several problems in biological sequence analysis like protein modeling and gene finding (see, e.g., Durbin, Eddy, Krogh, & Mitchison, 1998; Eddy, 1996; Krogh, Brown, Mian, Sjolander, ¨ & Haussler, 1994). Although the HMM is good at capturing the temporal nature of processes such as speech, it has a very limited capacity for recognizing complex patterns involving more than firstorder dependencies in the observed data. This is due to the first-order state process and the assumption of state-conditional independence of observations. Multilayer perceptrons are almost the opposite: they cannot model temporal phenomena very well but are good at recognizing complex patNeural Computation 11, 541–563 (1999)
c 1999 Massachusetts Institute of Technology °
542
Anders Krogh and Søren Kamaric Riis
terns. Combining the two frameworks in a sensible way can therefore lead to a more powerful model with better classification abilities. The starting point for this work is the so-called class HMM (CHMM), which is basically a standard HMM with a distribution over classes assigned to each state (Krogh, 1994). The CHMM incorporates conditional maximum likelihood (CML) estimation (Juang & Rabiner, 1991; N´adas, 1983; N´adas, Nahamoo, & Picheny, 1988). In contrast to the widely used maximum likelihood (ML) estimation, CML estimation is a discriminative training algorithm that aims at maximizing the ability of the model to discriminate between different classes. The CHMM can be normalized globally, which allows for nonnormalizing parameters in the individual states, and this enables us to generalize the CHMM to incorporate neural networks in a valid probabilistic way. In the CHMM/NN hybrid, which we call a hidden neural network (HNN), some or all CHMM probability parameters are replaced by the outputs of state-specific neural networks that take the observations as input. The model can be trained as a whole from observation sequences with labels by a gradient-descent algorithm. It turns out that in this algorithm, the neural networks are updated by standard backpropagation, where the errors are calculated by a slightly modified forward-backward algorithm. In this article, we first give a short introduction to standard HMMs. The CHMM and conditional ML are then introduced, and a gradient descent algorithm is derived for estimation. Based on this, the HNN is described next along with training issues for this model, and finally we give a comparison to other hybrid models. The article concludes with an evaluation of the HNN on the recognition of five broad phoneme classes in the TIMIT database (Garofolo et al., 1993). Results on this task clearly show a better performance of the HNN compared to a standard HMM. 2 Hidden Markov Models To establish notation and set the stage for describing CHMMs and HNNs, we start with a brief overview of standard hidden Markov models. (For a more comprehensive introduction, see Rabiner, 1989; Juang & Rabiner, 1991.) In this description we consider discrete first-order HMMs, where the observations are symbols from a finite alphabet A. The treatment of continuous observations is very similar (see, e.g., Rabiner, 1989). The standard HMM is characterized by a set of N states and two concurrent stochastic processes: a first-order Markov process between states modeling the temporal structure of the data and an emission process for each state modeling the locally stationary part of the data. The state process is given by a set of transition probabilities, θij , giving the probability of making a transition from state i to state j, and the emission process in state i is described by the probabilities, φi (a), of emitting symbol a ∈ A in state i. The φ’s are usually called emission probabilities, but we use the term
Hidden Neural Networks
543
match probabilities here. We observe only the sequence of outputs from the model, and not the underlying (hidden) state sequence, hence the name hidden Markov model. The set 2 of all transition and emission probabilities completely specifies the model. Given an HMM, the probability of an observation sequence, x = x1 , . . . , xL , of L symbols from the alphabet A is defined by P(x|2) =
X π
P(x, π|2) =
L XY π
θπl−1 πl φπl (xl ).
(2.1)
l=1
Here π = π1 , . . . , πL is a state sequence; πi is the number of the ith state in the sequence. Such a state sequence is called a path through the model. An auxiliary start state, π0 = 0, has been introduced such that θ0i denotes the probability of starting a path in state i. In the following we assume that state N is an end state: a nonmatching state with no outgoing transitions. The probability 2.1 can be calculated efficiently by a dynamic programming-like algorithm known as the forward algorithm. Let αi (l) = P(x1 , . . . , xl , πl = i | 2), that is, the probability of having matched observations x1 , . . . , xl and being in state i at time l. Then the following recursion holds for 1 ≤ i ≤ N and 1 < l ≤ L, αi (l) = φi (xl )
X
αj (l − 1)θji ,
(2.2)
j
and P(x|2) = αN (L). The recursion is initialized by αi (1) = θ0i φi (x1 ) for 1 ≤ i ≤ N. The parameters of the model can be estimated from data by an ML method. If multiple sequences of observations are available for training, they are assumed independent, and the total likelihood of the model is just a product of probabilities of the form 2.1 for each of the sequences. The generalization from one to many observation sequences is therefore trivial, and we will consider only one training sequence in the following. The likelihood of the model, P(x|2), given in equation 2.1, is commonly maximized by the Baum-Welch algorithm, which is an expectation-maximization (EM) algorithm (Dempster, Laird, & Rubin, 1977) guaranteed to converge to a local maximum of the likelihood. The Baum-Welch algorithm iteratively reestimates the model parameters until convergence, and for the transition probabilities the reestimation formulas are given by P nij nij (l) = P , θij ← P l 0 j0 l0 nij0 (l ) j0 nij0
(2.3)
where nij (l) = P(πl−1 = i, πl = j | x, 2) is the expected number of times a transition from state i to state j is used at time l. The reestimation equations
544
Anders Krogh and Søren Kamaric Riis
for the match probabilities can be expressed in a similar way by defining ni (l) = P(πl = i | x, 2) as the expected number of times we are in state i at time l. Then the reestimation equations for the match probabilities are given by P ni (l)δxl ,a ni (a) . = P φi (a) ← P l 0 0 la0 ni (l)δxl ,a a0 ni (a )
(2.4)
The expected counts can be computed efficiently by the forward-backward algorithm. In addition to the forward recursion, a similar recursion for the backward variable βi (l) is introduced. Let βi (l) = P(xl+1 , . . . , xL | πl = i, 2), that is, the probability of matching the rest of the sequence xl+1 , . . . , xL given that we are in state i at time l. After initializing by βN (L) = 1, the recursion runs from l = L − 1 to l = 1 as βi (l) =
N X
θij βj (l + 1)φj (xl+1 ),
(2.5)
j=1
for all states 1 ≤ i ≤ N. Using the forward and backward variables, nij (l) and ni (l) can easily be computed: nij (l) = P(πl−1 = i, πl = j | x, 2) = ni (l) = P(πl = i | x, 2) =
αi (l − 1)θij φj (xl )βj (l) P(x|2)
αi (l)βi (l) . P(x|2)
(2.6) (2.7)
2.1 Discriminative Training. In many problems, the aim is to predict what class an input belongs to or what sequence of classes it represents. In continuous speech recognition, for instance, the object is to predict the sequence of words or phonemes for a speech signal. To achieve this, a (sub)model for each class is usually estimated by ML independent of all other models and using only the data belonging to this class. This procedure maximizes the ability of the model to reproduce the observations in each class and can be expressed as ˆ ML = argmax P(x, y|2) = argmax[P(x|2y )P(y|2)], 2 2
2
(2.8)
where y is the class or sequence of class labels corresponding to the observation sequence x and 2y is the model for class y or a concatenation of submodels corresponding to the observed labels. In speech recognition, P(x|2y ) is often denoted the acoustic model probability, and the language model probability P(y|2) is usually assumed constant during training of the acoustic models. If the true source producing the data is contained in
Hidden Neural Networks
545
the model space, ML estimation based on an infinite training set can give the optimal parameters for classification (N´adas et al., 1988; N´adas, 1983), provided that the global maximum of the likelihood can be reached. However, in any real-world application, it is highly unlikely that the true source is contained in the space of HMMs, and the training data are indeed limited. This is the motivation for using discriminative training. To accommodate discriminative training, we use one big model and assign a label to each state; all the states that are supposed to describe a certain class C are assigned label C. A state can also have a probability distribution ψi (c) over labels, so that several labels are possible with different probabilities. This is discussed in Krogh (1994) and Riis (1998a), and it is somewhat similar to the input/output HMM (IOHMM) (Bengio & Frasconi, 1996). For brevity, however, we here limit ourselves to consider only one label for each state, which we believe is the most interesting for many applications. Because each state has a class label or a distribution over class labels, this sort of model was called a class HMM (CHMM) in Krogh (1994). In the CHMM, the objective is to predict the labels associated with x, and instead of ML estimation, we therefore choose to maximize the probability of the correct labeling, ˆ CML = argmax P(y|x, 2) = argmax 2 2
2
P(x, y|2) , P(x|2)
(2.9)
which is also called conditional maximum likelihood (CML) estimation (N´adas, 1983). If the language model is assumed constant during training, CML estimation is equivalent to maximum mutual information estimation (Bahl, Brown, de Souza, & Mercer, 1986). From equation 2.9, we observe that computing the probability of the labeling requires computation of (1) the probability P(x, y|2) in the clamped phase and (2) the probability P(x|2) in the free-running phase. The term free running means that the labels are not taken into account, so this phase is similar to the decoding phase, where we wish to find the labels for an observation sequence. The constraint by the labels during training gives rise to the name clamped phase; this terminology is borrowed from the Boltzmann machine literature (Ackley, Hinton, & Sejnowski, 1985; Bridle, 1990). Thus, CML estimation adjusts the model parameters so as to make the freerunning recognition model as close as possible to the clamped model. The probability in the free-running phase is computed using the forward algorithm described for standard HMMs, whereas the probability in the clamped phase is computed by considering only paths C (y) that are consistent with the observed labeling, P(x, y|2) =
X π∈C(y)
P(x, π|2).
(2.10)
546
Anders Krogh and Søren Kamaric Riis
This quantity can be calculated by a variant of the forward algorithm to be discussed below. Unfortunately the Baum-Welch algorithm is not applicable to CML estimation (see, e.g., Gopalakrishnan, Kanevsky, N´adas, & Nahamoo, 1991). Instead, one can use a gradient-descent-based approach, which is also applicable to the HNNs discussed later. To calculate the gradients, we switch to the negative log-likelihood, and define
L = − log P(y|x, 2) = Lc − L f Lc = − log P(x, y|2) L f = − log P(x|2).
(2.11) (2.12) (2.13)
The derivative of L f for the free-running model with regard to a generic parameter ω ∈ 2 can be expressed as, ∂ Lf 1 ∂P(x|2) =− ∂ω P(x|2) ∂ω X 1 ∂P(x, π |2) =− P(x|2) ∂ω π X P(x, π|2) ∂ log P(x, π|2) =− P(x|2) ∂ω π X ∂ log P(x, π |2) . P(π|x, 2) =− ∂ω π
(2.14)
This gradient is an expectation over all paths of the derivative of the complete data log-likelihood log P(x, π |2). Using equation 2.1, this becomes X ni (l) ∂φi (xl ) X nij (l) ∂θij ∂ Lf =− − . ∂ω φi (xl ) ∂ω θij ∂ω l,i l,i,j
(2.15)
The gradient of the negative log-likelihood Lc in the clamped phase is computed similarly, but the expectation is taken only for the allowed paths C (y), X mi (l) ∂φi (xl ) X mij (l) ∂θij ∂ Lc =− − , ∂ω φi (xl ) ∂ω θij ∂ω l,i l,i,j
(2.16)
where mij (l) = P(πl−1 = i, πl = j | x, y, 2) is the expected number of times a transition from state i to state j is used at time l for the allowed paths. Similarly, mi (l) = P(πl = i | x, y, 2) is the expected number of times we are in state i at time l for the allowed paths. These counts can be computed using the modified forward-backward algorithm, discussed below.
Hidden Neural Networks
547
For a standard model, the derivatives in equations 2.15 and 2.16 are simple. When ω is a transition probability, we obtain mij − nij ∂L =− . ∂θij θij
(2.17)
is of exactly the same form, except that mij and nij are The derivative ∂φ∂L i (a) replaced by mi (a) and ni (a), and θij by φi (a). When minimizing L by gradient descent, it must be ensured that the probability parameters remain positive and properly normalized. Here we use the same method as Bridle (1990) and Baldi and Chauvin (1994) and do gradient descent in another set of unconstrained variables. For the transition probabilities, we define ezij θij = P zij0 , j0 e
(2.18)
where zij are the new unconstrained auxiliary variables, and θij always sum ∂L yields to one by construction. Gradient descent in the z’s by zij ← zij − η ∂z ij a change in θ given by ∂L θij exp(−η ∂z ) ij . θij ← P ∂L j0 θij0 exp(−η ∂z 0 )
(2.19)
ij
The gradients with respect to zij can be expressed entirely in terms of θij and mij − nij , X ∂L = −[mij − nij − θij (mij0 − nij0 )], ∂zij j0
(2.20)
and inserting equation 2.20 into 2.19 yields an expression entirely in θ s. Equations for the emission probabilities are obtained in exactly the same way. This approach is slightly more straightforward than the one proposed in Baldi and Chauvin (1994), where the auxiliary variables are retained and the parameters of the model calculated explicitly from equation 2.18 after updating the auxiliary variables. This type of gradient descent is very similar to the exponentiated gradient descent proposed and investigated in Kivinen and Warmuth (1997) and Helmbold, Schapire, Singer, and Warmuth (1997). 2.2 The CHMM as a Probabilistic Independence Network. A large variety of probabilistic models can be represented as graphical models (Lauritzen, 1996), including the HMM and its variants. The relation between HMMs and probabilistic independence networks is thoroughly described
548
Anders Krogh and Søren Kamaric Riis
xl−2
xl−1
xl
xl+1
xl+2
xl−2
xl−1
xl
xl+1
xl+2
πl−2
πl−1
πl
πl+1
πl+2
πl−2
πl−1
πl
πl+1
πl+2
Figure 1: The DPIN (left) and UPIN (right) for an HMM. xl−2
πl−2 yl−2
xl−1
xl
πl−1 yl−1
xl+1
πl yl
πl+1 yl+1
xl+2
πl+2 yl+2
xl−2
πl−2 yl−2
xl−1
xl
πl−1 yl−1
xl+1
πl yl
πl+1 yl+1
xl+2
πl+2 yl+2
Figure 2: The DPIN (left) and UPIN (right) for a CHMM.
in Smyth, Heckerman, and Jordan (1997), and here we follow their terminology and refer the reader to that paper for more details. An HMM can be represented as both a directed probabilistic independence network (DPIN) and an undirected one (UPIN) (see Figure 1). The DPIN shows the conditional dependencies of the variables in the HMM— both the observable ones (x) and the unobservable ones (π ). For instance, the DPIN in Figure 1 shows that conditioned on πl , xl is independent of x1 , . . . , xl−1 and π1 , . . . , πl−1 , that is, P(xl |x1 , . . . , xl−1 , π1 , . . . , πl ) = P(xl |πl ). Similarly, P(πl |x1 , . . . , xl−1 , π1 , . . . , πl−1 ) = P(πl |πl−1 ). When “marrying” unconnected parents of all nodes in a DPIN and removing the directions, the moral graph is obtained. This is a UPIN for the model. For the HMM, the UPIN has the same topology as shown in Figure 1. In the CHMM there is one more set of variables (the y’s), and the PIN structures are shown in Figure 2. In a way, the CHMM can be seen as an HMM with two streams of observables, x and y, but they are usually not treated symmetrically. Again the moral graph is of the same topology, because no node has more than one parent. It turns out that the graphical representation is the best way to see the difference between the CHMM and the IOHMM. In the IOHMM, the output yl is conditioned on both the input xl and the state πl , but more important, the state is conditioned on the input. This is shown in the DPIN of Figure 3 (Bengio & Frasconi, 1996). In this case the moral graph is different, because πl has two unconnected parents in the DPIN. It is straightforward to extend the CHMM to have the label y conditioned on x, meaning that there would be arrows from xl to yl in the DPIN for the
Hidden Neural Networks xl−2
πl−2 yl−2
xl−1
xl
πl−1 yl−1
xl+1
πl yl
549
πl+1 yl+1
xl+2
πl+2 yl+2
xl−2
πl−2 yl−2
xl−1
xl
πl−1 yl−1
xl+1
πl yl
πl+1 yl+1
xl+2
πl+2 yl+2
Figure 3: The DPIN for an IOHMM (left) is adapted from Bengio and Frasconi (1996). The moral graph to the right is a UPIN for an IOHMM.
CHMM. Then the only difference between the DPINs for the CHMM and the IOHMM would be the direction of the arrow between xl and πl . However, the DPIN for the CHMM would still not contain any “unmarried parents” and thus their moral graphs would be different. 2.3 Calculation of Quantities Consistent with the Labels. Generally there are two different types of labeling: incomplete and complete labeling (Juang & Rabiner, 1991). We describe the modified forward-backward algorithm for both types of labeling below. 2.3.1 Complete Labels. In this case, each observation has a label, so the sequence of labels denoted y = y1 , . . . , yL is as long as the sequence of observations. Typically the labels come in groups, that is, several consecutive observations have the same label. In speech recognition, the complete labeling corresponds to knowing which word or phoneme each particular observation xl is associated with. For complete labeling, the expectations in the clamped phase are averages over “allowed” paths through the model—paths in which the labels of the states agree with the labeling of the observations. Such averages can be calculated by limiting the sum in the forward and backward recursions to states with the correct label. The new forward and backward variables, α˜ i (l) and β˜i (l), are defined as αi (l) (see equation 2.2) and βi (l) (see equation 2.5), but with φi (xl ) replaced by φi (xl )δyl ,ci . The expected counts mij (l) and mi (l) for the allowed paths are calculated exactly as nij (l) and ni (l), but using the new forward and backward variables. If we think of αi (l) (or βi (l)) as a matrix, the new algorithm corresponds to masking this matrix such that only allowed regions are calculated (see Figure 4). Therefore the calculation is faster than the standard forward (or backward) calculation of the whole matrix. 2.3.2 Incomplete Labels. When dealing with incomplete labeling, the whole sequence of observations is associated with a shorter sequence of
550
Anders Krogh and Søren Kamaric Riis Begin
1
A
B
3
2
A
B
4
State
Sequence x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 Labels A A A B B B B A A B B A A A 1 A 3 B End
∼ α=0
2 A 4 B
∼ α=0
∼ α=0 ∼ α=0
∼ α=0
Figure 4: (Left) A very simple model with four states, two labeled A and two labeled B. (Right) The α˜ matrix for an example of observations x1 , . . . , x14 with complete labels. The gray areas of the matrix are calculated as in the standard forward algorithm, whereas α˜ is set to zero in the white areas. The β˜ matrix is calculated in the same way, but from right to left.
labels y = y1 , . . . , yS , where S < L. The label of each individual observation is unknown; only the order of labels is available. In continuous speech recognition, the correct string of phonemes is known (because the spoken words are known in the training set), but the time boundaries between them are unknown. In such a case, the sequence of observations may be considerably longer than the label sequence. The case S = 1 corresponds to classifying the whole sequence into one of the possible classes (e.g., isolated word recognition). To compute the expected counts for incomplete labeling, one has to ensure that the sequence of labels matches the sequence of groups of states with the same label.1 This is less restrictive than the complete label case. An easy way to ensure this is by rearranging the (big) model temporarily for each observation sequence and collecting the statistics (the m’s) by running the standard forward-backward algorithm on this model. This is very similar to techniques already used in several speech applications (see, e.g., Lee, 1990), where phoneme (sub)models corresponding to the spoken word or sentence are concatenated. Note, however, that for the CHMM, the transitions between states with different labels retain their original value in the temporary model (see Figure 5).
1 If multiple labels are allowed in each state, an algorithm similar to the forwardbackward algorithm for asynchronous IOHMMs (Bengio & Bengio, 1996) can be used; see Riis (1998a).
Hidden Neural Networks
Begin
A
A
1
2
551
B
B
3
4
A
A
1
2
B
B
3
4
A
A
1
End 2
Figure 5: For the same model as in Figure 4, this example shows how the model is temporarily rearranged for gathering statistics (i.e., calculation of m values) for a sequence with incomplete labels ABABA.
3 Hidden Neural Networks HMMs are based on a number of assumptions that limit their classification abilities. Combining the CHMM framework with neural networks can lead to a more flexible and powerful model for classification. The basic idea of the HNN presented here is to replace the probability parameters of the CHMM by state-specific multilayer perceptrons that take the observations as input. Thus, in the HNN, it is possible to assign up to three networks to each state: (1) a match network outputting the “probability” that the current observation matches a given state, (2) a transition network that outputs transition “probabilities” dependent on observations, and (3) a label network that outputs the probability of the different labels in this state. We have put “probabilities” in quotes because the output of the match and transition networks need not be properly normalized probabilities, since global normalization is used. For brevity we limit ourselves here to one label per state; the label networks are not present. The case of multiple labels in each state is treated in more detail in Riis (1998a). The CHMM match probability φi (xl ) of observation xl in state i is replaced by the output of a match network, φi (sl ; wi ), assigned to state i. The match network in state i is parameterized by a weight vector wi and takes the vector sl as input. Similarly, the probability θij of a transition from state i to j is replaced by the output of a transition network θij (sl ; ui ), which is parameterized by weights ui . The transition network assigned to state i has Ji outputs, where Ji is the number of (nonzero) transitions from state i. Since we consider only states with one possible label, the label networks are just delta functions, as in the CHMM described earlier. The network input sl corresponding to xl will usually be a window of context around xl , such as a symmetrical context window of 2K + 1
552
Anders Krogh and Søren Kamaric Riis
observations,2 xl−K , xl−K+1 , . . . , xl+K ; however, it can be any sort of information related to xl or the observation sequence in general. We will call sl the context of observation xl , but it can contain all sorts of other information and can differ from state to state. The only limitation is that it cannot depend on the path through the model, because then the state process is no longer first-order Markovian. Each of the three types of networks in an HNN state can be omitted or replaced by standard CHMM probabilities. In fact, all sorts of combinations with standard CHMM states are possible. If an HNN contains only transition networks (that is, φi (sl ; wi ) = 1 for all i, l) the model can be normalized locally by using a softmax output function as in the IOHMM. P However, if it contains match networks, it is usually impossible to make x∈X P(x|2) = 1 by normalizing locally even if the transition networks are normalized. A probabilistic interpretation of the HNN is instead ensured by global normalization. We define the joint probability 1 R(x, y, π |2) Z(2) 1 Y θπl−1 πl (sl ; uπl−1 )φπl (sl ; wπl )δyl ,cπl , = Z(2) l
P(x, y, π |2) =
where the normalizing constant is Z(2) =
P x,y,π
(3.1)
R(x, y, π |2). From this,
1 X 1 R(x, y|2) = R(x, y, π |2) Z(2) Z(2) π X 1 R(x, π |2), = Z(2) π∈C(y)
P(x, y|2) =
(3.2)
where R(x, π|2) =
Y
θπl−1 πl (sl ; uπl−1 )φπl (sl ; wπl ).
(3.3)
l
Similarly, P(x|2) = =
1 X 1 R(x|2) = R(x, y, π |2) Z(2) Z(2) y,π 1 X R(x, π |2). Z(2) π
(3.4)
2 If the observations are inherently discrete (as in protein modeling), they can be encoded in binary vectors and then used in the same manner as continuous observation vectors.
Hidden Neural Networks
xl−2
553
xl−1
πl−2 yl−2
xl
πl−1 yl−1
xl+1
πl yl
πl+1 yl+1
xl+2
πl+2 yl+2
Figure 6: The UPIN for an HNN using transition networks that take only the current observation as input (sl = xl ).
It is sometimes possible to compute the normalization factor Z, but not in all cases. However, for CML estimation, the normalization factor cancels out, P(y|x, 2) =
R(x, y|2) . R(x|2)
(3.5)
The calculation of R(x|2) and R(x, y|2) can be done exactly as the calculation of P(x|2) and P(x, y|2) in the CHMM, because the forward and backward algorithms are not dependent on the normalization of probabilities. Because one cannot usually normalize the HNN locally, there exists no directed graph (DPIN) for the general HNN. For UPINs, however, local normalization is not required. For instance, the Boltzmann machine can be drawn as a UPIN, and the Boltzmann chain (Saul & Jordan, 1995) can actually be described by a UPIN identical to the one for a globally normalized discrete HMM in Figure 1. A model with a UPIN is characterized by its clique functions, and the joint probability is the product of all the clique functions (Smyth et al., 1997). The three different clique functions are clearly seen in equation 3.1. In Figure 6 the UPIN for an HNN with transition networks and sl = xl is shown; this is identical to Figure 3 for the IOHMM, except that it does not have edges from x to y. Note that the UPIN remains the same if match networks (with sl = xl ) are used as well. The graphical representation as a UPIN for an HNN with no transition networks and match networks having a context of one to each side is shown in Figure 7 along with the three types of cliques. A number of authors have investigated compact representations of conditional probability tables in DPINs (see Boutilier, Friedman, Goldszmidt, & Koller, 1996, and references therein). The HNN provides a similar compact representation of clique functions in UPINs, and this holds also for models that are more general than the HMM-type graphs discussed in this article. The fact that the individual neural network outputs do not have to normalize gives us a great deal of freedom in selecting the output activation
554
Anders Krogh and Søren Kamaric Riis xl−2
xl−1
xl
xl+1
xl+2
xl−1
xl
xl+1
πl πl−2 yl−2
πl−1 yl−1
πl yl
πl+1 yl+1
πl+2 yl+2
πl−1
πl
πl
yl
Figure 7: (Left) The UPIN of an HNN with no transition networks and match networks having a context of one to each side. (Right) The three different clique types contained in the graph.
function. A natural choice is a standard (asymmetric) sigmoid or an exponential output activation function, g(h) = exp(h), where h is the input to the output unit in question. Although the HNN is a very intuitive and simple extension of the standard CHMM, it is a much more powerful model. First, neural networks can implement complex functions using far fewer parameters than, say, a mixture of gaussians. Furthermore, the HNN can directly use observation context as input to the neural networks and thereby exploit higher-order correlations between consecutive observations, which is difficult in standard HMMs. This property can be particularly useful in problems like speech recognition, where the pronunciation of one phoneme is highly influenced by the acoustic context in which it is uttered. Finally, the observation context dependency on the transitions allows the HNN to model the data as successive steady-state segments connected by “nonstationary” transitional regions. For speech recognition this is believed to be very important (see, e.g., Bourlard, Konig, & Morgan, 1994; Morgan, Bourlard, Greenberg, and Hermansky, 1994). 3.1 Training an HNN. As for the CHMM, it is not possible to train the HNN using an EM algorithm; instead, we suggest training the model using gradient descent. From equations 2.15 and 2.16, we find the following gradients of L = − log P(y|x, 2) with regard to a generic weight ωi in the match or transition network assigned to state i, X mi (l)−ni (l) ∂φi (sl ; wi ) X mij (l)−nij (l) ∂θij (sl ; ui ) ∂L = − − , ∂ωi φi (sl ; wi ) ∂ωi θij (sl ; ui ) ∂ωi l lj
(3.6)
where it is assumed that networks are not shared between states. In the backpropagation algorithm for neural networks (Rumelhart, Hinton, & Williams, 1986) the squared error of the network is minimized by gradient descent. For an activation function g, this gives rise to a weight update of the form ∂g 1w ∝ −E × ∂w . We therefore see from equation 3.6 that the neural networks
Hidden Neural Networks
555
are trained using the standard backpropagation algorithm where the quantity to backpropagate is E = [mi (l) − ni (l)]/φi (sl ; wi ) for the match networks and E = [mij (l) − nij (l)]/θij (sl ; ui ) for the transition networks. The m and n counts are calculated as before by running two forward-backward passes: once in the clamped phase (the m’s) and once in the free-running phase (the n’s). The training can be done in either batch mode, where all the networks are updated after the entire training set has been presented to the model, or sequence on-line mode, where the update is performed after the presentation of each sequence. There are many other variations possible. Because of the l dependence of mij (l), mi (l) and the similar n’s, the training algorithm is not as simple as for standard HMMs; we have to do a backpropagation pass for each l. Because the expected counts are not available before the forwardbackward passes have been completed, we must either store or recalculate all the neural network unit activations for each input sl before running backpropagation. Storing all activations can require large amounts of memory even for small networks if the observation sequences are very long (which they typically are in continuous speech). For such tasks, it is necessary to recalculate the network unit activations before each backpropagation pass. Many of the standard modifications of the backpropagation algorithm can be incorporated, such as momentum and weight decay (Hertz, Krogh, & Palmer, 1991). It is also possible to use conjugate gradient descent or approximative second-order methods like pseudo-Gauss-Newton. However, in a set of initial experiments for the speech recognition task reported in section 4, on-line gradient methods consistently gave the fastest convergence. 4 Comparison to Other Work Recently several HMM/NN hybrids have been proposed in the literature. The hybrids can roughly be divided into those estimating the parameters of the HMM and the NN separately (see, e.g., Renals, Morgan, Bourlard, Cohen, & Franco, 1994; Robinson, 1994; Le Cerf, Ma, & Compernolle, 1994; McDermott & Katagiri, 1991) and those applying simultaneous or joint estimation of all parameters as in the HNN (see, e.g., Baldi & Chauvin, 1996; Konig, Bourlard, & Morgan, 1996; Bengio, De Mori, Flammia, & Kompe, 1992; Johansen, 1994; Valtchev, Kapadia, & Young, 1993; Bengio & Frasconi, 1996; Hennebert, Ris, Bourlard, Renals, & Morgan, 1997; Bengio, LeCun, Nohl, & Burges, 1995). In Renals et al. (1994) a multilayer perceptron is trained separately to estimate phoneme posterior probabilities, which are scaled with the observed phoneme frequencies and then used instead of the usual emission densities in a continuous HMM. A similar approach is taken in Robinson (1994), but here a recurrent NN is used. A slightly different method is used in McDermott and Katagiri (1991) and Le Cerf et al. (1994), where the vector quantizer front end in a discrete HMM is replaced by a multilayer percep-
556
Anders Krogh and Søren Kamaric Riis
tron or a learning vector quantization network (Kohonen, Barna, & Chrisley, 1988). In contrast, our approach uses only one output for each match network whereby continuous and discrete observations are treated the same. Several authors have proposed methods in which all parameters are estimated simultaneously as in the HNN. In some hybrids, a big multilayer perceptron (Bengio et al., 1992; Johansen & Johnsen, 1994) or recurrent network (Valtchev et al., 1993) performs an adaptive input transformation of the observation vectors. Thus, the network outputs are used as new observation vectors in a continuous density HMM, and simultaneous estimation of all parameters is performed by backpropagating errors calculated by the HMM into the neural network in a way similar to the HNN training. Our approach is somewhat similar to the idea of adaptive input transformations, but instead of retaining the computationally expensive mixture densities, we replace these by match networks. This is also done in Bengio et al. (1995), where a large network with the same number of outputs as there are states in the HMM is trained by backpropagating errors calculated by the HMM. Instead of backpropagating errors from the HMM into the neural network, Hennebert et al. (1997) and Senior and Robinson (1996) use a two-step iterative procedure to train the networks. In the first step, the current model is used for estimating a set of “soft” targets for the neural networks, and then the network is trained on these targets. This method extends the scaled likelihood approach by Renals et al. (1994) to use global estimation where training is performed by a generalized EM (GEM) algorithm (Hennebert et al., 1997). The IOHMM (Bengio and Frasconi, 1996) and the CHMM/HNN have different graphical representations, as seen in Figures 2, 3, and 7. However, the IOHMM is very similar to a locally normalized HNN with a label and transition network in each state, but no match network. An important difference between the two is in the decoding, where the IOHMM uses only a forward pass, which makes it insensitive to future events but makes the decoding “real time.” (See Riis, 1998a, for more details.) 5 Experiments In this section we give an evaluation of the HNN on the task introduced in Johansen (1994) of recognizing five broad phoneme classes in continuous read speech from the TIMIT database (Garofolo et al., 1993): vowels (V), consonants (C), nasals (N), liquids (L) and silence (S) (see Table 1). We use one sentence from each of the 462 speakers in the TIMIT training set for training, and the results are reported for the recommended TIMIT core test set containing 192 sentences. An additional validation set of 144 sentences has been used to monitor performance during training. The raw speech signal is preprocessed using a standard mel cepstral preprocessor, which outputs a 26-dimensional feature vector each 10 ms (13 mel cepstral features and 13 delta features). These vectors are normalized to zero mean
Hidden Neural Networks
557
Table 1: Definition of Broad Phoneme Classes. Broad Class
TIMIT Phoneme Label
Vowel (V) Consonant (C) Nasal (N) Liquid (L) Silence (S)
iy ih eh ae ix ax ah ax-h uw uh ao aa ey ay oy aw ow ux ch jh dh b d dx g p t k z zh v f th s sh hh hv m n en ng em nx eng l el r y w er axr h# pau
and unit variance. Each of the five classes is modeled by a simple left-toright three-state model. The last state in any submodel is fully connected to the first state of all other submodels. (Further details are given in Riis & Krogh, 1997.) 5.1 Baseline Results. In Table 2, the results for complete label training are shown for the baseline system, which is a discrete CHMM using a codebook of 256 codebook vectors. The results are reported in the standard measure of percentage accuracy, %Acc = 100%−%Ins−%Del−%Sub, where %Ins, %Del and %Sub denote the percentage of insertions, deletions and substitutions used for aligning the observed and the predicted transcription.3 In agreement with results reported in Johansen (1994), we have observed an increased performance for CML estimated models when using a forward or all-paths decoder instead of the best-path Viterbi decoder. In this work we use an N-best decoder (Schwarz & Chow, 1990) with 10 active hypotheses during decoding. Only the top-scoring hypothesis is used at the end of decoding. The N-best decoder finds (approximatively) the most probable labels, which depends on many different paths, whereas the Viterbi algorithm finds only the most probable path. For ML-trained models, the N-best and Viterbi decoder yield approximately the same accuracy (see Table 2). As shown by an example in Figure 8, several paths contribute to the optimal labeling in the CML estimated models, whereas only a few paths contribute significantly for the ML estimated models. Table 2 shows that additional incomplete label training of a complete label trained model does not improve performance for the ML estimated model. However, for the CML estimation, there is a significant gain in accuracy by incomplete label training. The reason is that the CML criterion is very sensitive to mislabelings, because it is dominated by training sequences with an unlikely labeling. Although the phoneme segmentation (complete labeling) in TIMIT is done by hand, it is imperfect. Furthermore, it is often impossible—or even meaningless—to assign exact boundaries between phonemes.
3
The NIST standard scoring package “sclite” version 1.1 is used in all experiments.
558
Anders Krogh and Søren Kamaric Riis
ML
CML 1 0.8
10
0.6 0.4
5
0.2 50
0.8 10
0.6 0.4
5
0
100 150 200 Frame
1
15
State Posterior
State Posterior
15
0.2 50
HNN
0.8 10
0.6 0.4
5
0.2 100 150 200 Frame
1
15
State Posterior
State Posterior
0
Segmentation 1
15
50
100 150 200 Frame
0.8 10
0.6 0.4
5
0
0.2 50
100 150 200 Frame
0
Figure 8: State posterior plots (P(πl = i | x, 2)) for baseline and HNN for the test sentence “But in this one section we welcomed auditors” (TIMIT id: si1361). States 1–3 belong to the consonant model, 4–6 to the nasal model, 7–9 to the liquid model, 10–12 to the vowel model, and 13–15 to the silence model. (Top left) ML-trained baseline, which yield %Acc = 62.5 for this sentence. (Top right) CML-trained baseline (%Acc = 78.1). (Bottom left) HNN using both match and transition networks with 10 hidden units and context K = 1 (%Acc = 93.7). (Bottom right) The observed segmentation. Table 2: Baseline Recognition Accuracies. Viterbi
N-Best
75.9 76.6
76.1 79.0
75.8 78.4
75.2 81.3
Complete labels ML CML Incomplete labels ML CML
Note: The baseline system contains 3856 free parameters.
Hidden Neural Networks
559
CML gives a big improvement from an accuracy of around 76% for the ML estimated models to around 81%. Statistical significance is hard to assess, because of the computational requirements for this task, but in a set of 10 CML training sessions of random initial models, we observed a deviation of no more than ±0.2% in accuracy. For comparison, a MMI-trained model with a single diagonal covariance gaussian per state achieved a result of 72.4% accuracy in Johansen and Johnsen (1994). 5.2 HNN Results. For the HNN, two series of experiments were conducted. In the first set of experiments, only a match network is used in each state, and the transitions are standard HMM transitions. In the second set of experiments, we also use match networks, but the match distribution and the standard transitions in the last state of each submodel are replaced by a transition network. All networks use the same input sl , have the same number of hidden units, are fully connected, and have sigmoid output functions. This also applies for the transition networks; that is, a softmax output function is not used for the transition networks. Although the HNN with match networks and no hidden units has far fewer parameters than the baseline system, it achieves a comparable performance of 80.8% accuracy using only the current observation xl as input (K = 0) and 81.7% accuracy for a context of one left and right observation (K = 1) (see Table 3). No further improvement was observed for larger contexts. Note that the match networks without hidden units just implement linear weighted sums of input features (passed through a sigmoid output function). For approximately the same number of parameters as used in the baseline system, the HNN with 10 hidden units and no context (K = 0) yields 84.0% recognition accuracy. Increasing the context or number of hidden units for this model yields a slightly lower accuracy due to overfitting. In Johansen (1994) a multilayer perceptron was used as a global adaptive input transformation to a continuous density HMM with a single diagonal covariance gaussian per state. Using N-best decoding and CML estimation, a result of 81.3% accuracy was achieved on the broad phoneme class task. When using a transition network in the last state of each submodel, the accuracy increases, as shown in Table 3. Thus, for the model with context K = 1 and no hidden units, an accuracy of 82.3% is obtained compared to 81.7% for the same model with only match networks. The best result on the five broad class task is an accuracy of 84.4% obtained by the HNN with context K = 1, match and transition networks and 10 hidden units in all networks (see Table 3). 6 Conclusion In this article we described the HNN, which in a very natural way replaces the probability parameters of an HMM with the output of state-specific
560
Anders Krogh and Søren Kamaric Riis
Table 3: Recognition Accuracies for HNNs. Context K
Number of Parameters
Accuracy
0 1 1
436 1,216 2411
80.8 81.7 82.3
0 1 1
4,246 12,046 12,191
84.0 83.8 84.4
No hidden units HNN, match networks HNN, match networks HNN, match and transition networks Ten hidden units HNN, match networks HNN, match networks HNN, match and transition networks
Note: “HNN, match networks” are models using only match networks and standard CHMM transitions, whereas “HNN, match and transition networks” use both match and transition networks. Decoding is done by N-best.
neural networks. The model is normalized at a global level, which ensures a proper probabilistic interpretation of the HNN. All the parameters in the model are trained simultaneously from labeled data using gradientdescent-based CML estimation. The architecture is very flexible in that all combinations with standard CHMM probability parameters are possible. The relation to graphical models was discussed, and it was shown that the HNN can be viewed as an undirected probabilistic independence network, where the neural networks provide a compact representation of the clique functions. Finally, it was shown that the HNN improves on the results of a speech recognition problem with a reduced set of phoneme classes. The HNN has also been applied to the recognition of task-independent isolated words from the PHONEBOOK database (Riis, 1998b) and preliminary results on the 39 phoneme TIMIT problem are presented in Riis (1998a). Acknowledgments We thank Steve Renals and Finn T. Johansen for valuable comments and suggestions to this work. We also thank the anonymous referees for pointing our attention to graphical models. This work was supported by the Danish National Research Foundation. References Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985). A learning algorithm for Boltzmann machines. Cognitive Science, 9, 147–169. Bahl, L. R., Brown, P. F., de Souza, P. V., & Mercer, R. L. (1986). Maximum mu-
Hidden Neural Networks
561
tual information estimation of hidden Markov model parameters for speech recognition. In Proceedings of ICASSP’86 (pp. 49–52). Baldi, P., & Chauvin, Y. (1994). Smooth on-line learning algorithms for hidden Markov models. Neural Computation, 6(2), 307–318. Baldi, P., & Chauvin, Y. (1996). Hybrid modeling, HMM/NN architectures, and protein applications. Neural Computation, 8, 1541–1565. Bengio, S., & Bengio, Y. (1996). An EM algorithm for asynchronous input/output hidden Markov models. In Proceedings of the ICONIP’96. Bengio, Y., De Mori, R., Flammia, G., & Kompe, R. (1992). Global optimization of a neural network–hidden Markov model hybrid. IEEE Transactions on Neural Networks, 3(2), 252–259. Bengio, Y., & Frasconi, P. (1996). Input/output HMMs for sequence processing. IEEE Transactions on Neural Networks, 7(5), 1231–1249. Bengio, Y., LeCun, Y., Nohl, C., & Burges, C. (1995). Lerec: A NN/HMM hybrid for on-line handwritting recognition. Neural Computation, 7(5). Bourlard, H., Konig, Y., & Morgan, N. (1994). REMAP: Recursive estimation and maximization of a posteriori probabilities (Tech. Rep. TR-94-064). Berkeley, CA: International Computer Science Institute. Boutilier, C., Friedman, N., Goldszmidt, M., & Koller, D. (1996). Context-specific independence in Bayesian networks. In E. Horvitz & F. V. Jensen (Eds.), Proceedings of the Twelfth Conference on Uncertainty in Artificial Intelligence (pp. 115– 123). San Francisco: Morgan Kaufmann. Bridle, J. S. (1990). Alphanets: A recurrent “neural” network architecture with a hidden Markov model interpretation. Speech Communication, 9, 83–92. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Royal Statistical Society B, 39, 1–38. Durbin, R. M., Eddy, S. R., Krogh, A., & Mitchison, G. (1998). Biological sequence analysis. Cambridge: Cambridge University Press. Eddy, S. R. (1996). Hidden Markov models. Current Opinion in Structural Biology, 6, 361–365. Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallet, D. S., & Dahlgren, N. L. (1993). DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CDROM. Gaithersburg, MD: National Institute of Standards. Gopalakrishnan, P. S., Kanevsky, D., N´adas, A., & Nahamoo, D. (1991). An inequality for rational functions with applications to some statistical estimation problems. IEEE Transactions on Information Theory, 37(1), 107–113. Helmbold, D. P., Schapire, R. E., Singer, Y., & Warmuth, M. K. (1997). A comparison of new and old algorithms for a mixture estimation problem. Machine Learning, 27(1), 97–119. Hennebert, J., Ris, C., Bourlard, H., Renals, S., & Morgan, N. (1997). Estimation of global posteriors and forward-backward training of hybrid HMM/ANN systems. In Proceedings of EUROSPEECH’97. Hertz, J. A., Krogh, A., & Palmer, R. (1991). Introduction to the theory of neural computation. Redwood City, CA: Addison-Wesley. Johansen, F. T. (1994). Global optimisation of HMM input transformations. In Proceedings of ICSLP’94 (Vol. 1, pp. 239–242).
562
Anders Krogh and Søren Kamaric Riis
Johansen, F. T., & Johnsen, M. H. (1994). Non-linear input transformations for discriminative HMMs. In Proceedings of ICASSP’94 (Vol. 1, pp. 225–228). Juang, B. H., & Rabiner, L. R. (1991). Hidden Markov models for speech recognition. Technometrics, 33(3), 251–272. Kivinen, J., & Warmuth, M. K. (1997). Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132(1), 1–63. Kohonen, T., Barna, G., & Chrisley, R. (1988). Statistical pattern recognition with neural networks: Benchmarking studies. In Proceedings of ICNN’88 (Vol. 1, pp. 61–68). Konig, Y., Bourlard, H., & Morgan, N. (1996). REMAP: Recursive estimation and maximization of a posteriori probabilities—application to transition-based connectionist speech recognition. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 388– 394). Cambridge, MA: MIT Press. Krogh, A. (1994). Hidden Markov models for labeled sequences. In Proceedings of the 12th IAPR ICPR’94 (pp. 140–144). Krogh, A., Brown, M., Mian, I. S., Sjolander, ¨ K., & Haussler, D. (1994). Hidden Markov models in computational biology: Applications to protein modeling. Journal of Molecular Biology, 235, 1501–1531. Lauritzen, S. L. (1996). Graphical models. New York: Oxford University Press. Le Cerf, P., Ma, W., & Compernolle, D. V. (1994). Multilayer perceptrons as labelers for hidden Markov models. IEEE Transactions on Speech and Audio Processing, 2(1), 185–193. Lee, K.-F. (1990). Context-dependent phonetic hidden Markov models for speaker-independent continuous speech recognition. IEEE Transactions on Acoustics, Speech and Signal Processing, 38(4), 599–609. McDermott, E., & Katagiri, S. (1991). LVQ-based shift-tolerant phoneme recognition. IEEE Transactions on Acoustics, Speech and Signal Processing, 39, 1398–1411. Morgan, N., Bourlard, H., Greenberg, S., & Hermansky, H. (1994). Stochastic perceptual auditory-event-based models for speech recognition. In Proceedings of Interternational Conference on Spoken Language Processing (pp. 1943–1946). N´adas, A. (1983). A decision-theoretic formulation of a training problem in speech recognition and a comparison of training by unconditional versus conditional maximum likelihood. IEEE Transactions on Acoustics, Speech and Signal Processing, 31(4), 814–817. N´adas, A., Nahamoo, D., & Picheny, M. A. (1988). On a model-robust training method for speech recognition. IEEE Transactions on Acoustics, Speech and Signal Processing, 36(9), 814–817. Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of IEEE, 77(2), 257–286. Renals, S., Morgan, N., Bourlard, H., Cohen, M., & Franco, H. (1994). Connectionist probability estimators in HMM speech recognition. IEEE Transactions on Speech and Audio Processing, 2(1), 161–174. Riis, S. K. (1998a). Hidden Markov models and neural networks for speech recognition. Doctoral dissertation, IMM-PHD-1998-46, Technical University of Denmark. Riis, S. K. (1998b). Hidden neural networks: Application to speech recognition. In In Proceedings of ICASSP’98 (Vol. 2, pp. 1117–1121).
Hidden Neural Networks
563
Riis, S. K., & Krogh, A. (1997). Hidden neural networks: A framework for HMM/NN hybrids. In Proceedings of ICASSP’97 (pp. 3233–3236). Robinson, A. J. (1994). An application of recurrent nets to phone probability estimation. IEEE Transactions on Neural Networks, 5, 298–305. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. Saul, L. K., & Jordan, M. I. (1995). Boltzman chains and hidden Markov models. In G. Tesauro, D. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 435–442). San Mateo, CA: Morgan Kaufmann. Schwarz, R., & Chow, Y.-L. (1990). The N-best algorithm: An efficient and exact procedure for finding the N most likely hypotheses. In Proceedings of ICASSP’90 (pp. 81–84). Senior, A., & Robinson, T. (1996). Forward-backward retraining of recurrent neural networks. In D. Touretzky, M. Mozer, & M. Hasselmo (Eds.), Advances in neural information processing dystems, 8 (pp. 743–749). San Mateo, CA: Morgan Kaufmann. Smyth, P., Heckerman, D., & Jordan, M. I. (1997). Probabilistic independence networks for hidden Markov probability models. Neural Computation, 9, 227– 269. Valtchev, V., Kapadia, S., & Young, S. (1993). Recurrent input transformations for hidden Markov models. In Proceedings of ICASSP’93 (pp. 287–290). Received October 21, 1997; accepted May 7, 1998.
ARTICLE
Communicated by Stephen Lisberger
A Cerebellar Model of Timing and Prediction in the Control of Reaching Andrew G. Barto Andrew H. Fagg Nathan Sitkoff Department of Computer Science, University of Massachusetts, Amherst, MA 01003, U.S.A.
James C. Houk Department of Physiology, Northwestern University Medical School, Chicago, IL 60611, U.S.A.
A simplified model of the cerebellum was developed to explore its potential for adaptive, predictive control based on delayed feedback information. An abstract representation of a single Purkinje cell with multistable properties was interfaced, using a formalized premotor network, with a simulated single degree-of-freedom limb. The limb actuator was a nonlinear spring-mass system based on the nonlinear velocity dependence of the stretch reflex. By including realistic mossy fiber signals, as well as realistic conduction delays in afferent and efferent pathways, the model allowed the investigation of timing and predictive processes relevant to cerebellar involvement in the control of movement. The model regulates movement by learning to react in an anticipatory fashion to sensory feedback. Learning depends on training information generated from corrective movements and uses a temporally asymmetric form of plasticity for the parallel fiber synapses on Purkinje cells. 1 Introduction The neural commands that control rapid limb movements appear to comprise pulse components followed by smaller-step components (Ghez, 1979; Ghez & Martin, 1982), analogous to the pulse-step commands that control rapid eye movements (Robinson, 1975). In the case of eye movements, the pulse component serves to overcome the internal viscosity of the muscles, thus moving the eye rapidly to the target, whereupon the step component holds the eye at its final position. Limb movements involve more inertia than eye movements, so the pulse activation of the agonist muscle must end partway through the movement, and a braking pulse in the antagonist muscle is needed to decelerate the mass of the limb. Ghez and Martin (1982) showed that the braking pulse is produced by a stretch reflex in the anNeural Computation 11, 565–594 (1999)
c 1999 Massachusetts Institute of Technology °
566
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
tagonist muscle. The central control problem, therefore, is to terminate the pulse phase of the command sent to the agonist muscle at an appropriate time during the movement. The dynamics of the stretch reflex should then bring the movement to a halt at a desired end point. Since the pulse must terminate well in advance of the achievement of the desired end point, this is a problem of timing and prediction in control. In this article, we present a model of how the cerebellum may contribute to the predictive control of limb movements. The model is a simplified version of the adjustable pattern generator (APG) model being developed by Houk and colleagues (Berthier, Singh, Barto, & Houk, 1993; Houk, Singh, Fisher, & Barto, 1990; Sinkjær, Wu, Barto, & Houk, 1990) to test the computational competence of a conceptual framework for understanding the brain mechanisms of motor control (Houk, 1989; Houk & Barto, 1992; Houk, Keifer, & Barto, 1993; Houk & Wise, 1995; Houk, Buckingham, & Barto, 1996). The model has a modular architecture in which single modules generate elemental motor commands with adjustable time courses, and multiple modules cooperatively produce more complex commands. The APG model is constrained by the modular anatomy of the cerebellar cortex and its connections with the limb premotor network, by the physiology of the neurons comprising this network, and by properties of cerebellar Purkinje cells (PCs). However, it is purposefully abstract to allow us to explore control and learning issues in a computationally feasible manner. The model presented here corresponds to a single module of the APG model consisting of a single unit representing a PC. This unit is modeled as a collection of nonlinear switching elements, which we call dendritic zones, representing segments of a PC dendritic tree. Our previous modeling studies dealt mainly with two issues: (1) demonstration that a single module can learn to generate appropriate one-dimensional, variable-duration velocity commands (Houk et al., 1990) and (2) a preliminary demonstration that an array of 48 modules can learn to function cooperatively in the control of a simulated nondynamic, two-joint planar limb (Berthier et al., 1993). In these previous simulations, the input layer of the cerebellum, the representation of PCs, and the complexity of the learning problem were greatly simplified. In this article, we employ a more realistic input representation based on what is known about movementrelated mossy fiber (MF) signals in the intermediate cerebellum of the monkey (Van Kan, Gibson, & Houk, 1993a) and the Marr-Albus architecture of the granular layer (Tyrrell & Willshaw, 1992). In addition, we use a more complex dynamic spring-mass system (although it is still one-dimensional), and we include realistic conduction delays in the relevant signal pathways. The model also makes use of a trace mechanism in its learning rule. Preliminary results appear in Buckingham, Barto, and Houk (1995) and Barto, Buckingham, and Houk (1996). We first describe the nonlinear spring-mass system and discuss some of its properties from a control point of view. The following section presents
A Cerebellar Model of Timing and Prediction
567
the details of the model. We then present simulation results demonstrating the learning and control abilities of a single dendritic zone, followed by similar results for a model with multiple dendritic zones. We conclude with a discussion of these results. 2 Pulse-Step Control of a Nonlinear Plant The limb motor plant has prominent nonlinearities that have a strong influence on movement and its control. The plant model used in this study is a spring-mass system with a form of nonlinear damping based on studies of human wrist movement (Gielen & Houk, 1984; Wu, Houk, Young, & Miller, 1990): 1
˙ 5 + K(x − xeq ) = 0, Mx¨ + B(x)
(2.1)
where x is the position (in meters) of an object of mass M (kg) attached to the spring, xeq is the resting, or equilibrium, position, B is the damping coefficient, and K is the spring stiffness (see Figure 1a). This fractional power form of nonlinear damping is derived from a combination of nonlinear muscle properties and spinal reflex mechanisms, the latter driven mainly by feedback from muscle spindle receptors (Gielen & Houk, 1987). Setting M = 1, B = 3, and K = 30 produces trajectories that are qualitatively similar to those observed in human wrist movement (Wu et al., 1990). Nonlinear damping of this kind enables fast movements that terminate with little oscillation. Figure 1b is a graph of the damping force as a function of velocity. As velocity decreases, the effective damping coefficient (the curve’s slope) increases radically when the velocity gets sufficiently close to zero. This causes a decelerating mass generally to “stick” at a nonequilibrium position, thereafter drifting extremely slowly toward xeq . We call the position at which the mass sticks (defined here as the position at which the absolute value of its velocity falls and remains below 0.9 cm/sec) the end point of a movement, denoted xe . For all practical purposes, this is where the movement stops. The control signal in our model sets the equilibrium value xeq , which represents a central motor command setting the threshold of the stretch reflex (Feldman, 1966; Houk & Rymer, 1981). Pulse-step control is effective in producing rapid and well-controlled positioning of the mass in this system. As shown in Figure 1c, the control signal switches from a pulse level, xeq = xp , to a smaller step level, xeq = xs . Also shown are the time courses of the velocity (see Figure 1c, middle) and position (see Figure 1c, bottom) for the resulting movement. Inserting a low-pass filter in the command pathway, a common feature of muscle models, would produce velocity profiles more closely matching those of actual movements, but we have not been concerned with this issue.
568
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk Damping Force
M
0
x
Velocity
xeq (a)
(b) 35
Vel (cm/s)
30 20 10 0 15 xp 10 x 5 xT 0 s
25 20
Vel (cm/s)
Xeq (cm)
15 xp 10 5 xs 0 x0
Pos (cm)
30
15 10 5 0 −5
0
100 200 Time (ms)
(c)
300
400
500
x0 0
xs xT 2
4 6 Pos (cm)
xp 8
10
(d)
Figure 1: Pulse-step control of a simplified motor plant. (a) Spring-mass system. M, mass; x, position; xeq , resting, or equilibrium, position. (b) Nonlinear damping force as a function of velocity. The plant’s effective damping coefficient (the graph’s slope) increases rapidly as the velocity magnitude decreases to zero. (c) Pulse-step control. Control of a movement from initial position x0 = 0 to target end point xT = 5 cm. Top: The pulse-step command. Middle: Velocity as a function of time. Bottom: Position as a function of time. (d) Phase-plane trajectory. The bold line is the phase-plane trajectory of the movement of panel c. The dashed line is a plot of the states of the spring-mass system at which the command should switch from pulse to step so that the mass will stick at the end point xT = 5 cm starting from a variety of different initial states.
Figure 1d shows the phase-plane trajectory (velocity plotted against position) followed by the state of the spring-mass system during pulse-step control. When the pulse is being applied, the state follows a trajectory that would end at the equilibrium position xp = 10 cm if the pulse were to continue. When the step begins, the state switches to the trajectory that ends at the equilibrium position xs = 4 cm, but the mass sticks at the target end point, xT = 5 cm, before reaching this equilibrium position. Thus, simply setting the equilibrium position to the target end point as suggested by the
A Cerebellar Model of Timing and Prediction
569
equilibrium-point hypothesis (Bizzi, Hogan, Mussa-Ivaldi, & Gister, 1992; Feldman, 1966, 1974) is not a practical solution to the end point positioning task for this system. The dashed line in Figure 1d is an approximate plot of the states at which the switch from pulse to step should occur so that movements starting from a variety of initial states will stick at xT = 5 cm. This switching curve has to vary as a function of the target end point. If the switch from pulse to step occurs too soon (late), the mass will undershoot (overshoot) xT . In developing a model of pulse-step control of the limb, one can profit from analogies, where appropriate, with the extensive literature on pulsestep control of saccadic eye movements. However, an important difference between eye and limb control is the absence of a stretch reflex for regulating primate eye muscle activity (Keller & Robinson, 1971). As a consequence, models of the eye motor plant do not contain the nonlinear damping mechanism present in equation 2.1. The stretch reflex is important in generating a braking pulse in the antagonist muscles needed to decelerate the limb (Ghez & Martin, 1982). In fact, the stretch reflex is the predominant mechanism responsible for the entire decelerating portion of the trajectory in Figure 1D. The stretch reflex is also the main mechanism causing the limb to stick at a nonequilibrium position, as witnessed by the drift in limb position that occurs in deafferented patients who lack a stretch reflex (Ghez, Gordon, Ghilardi, Christakos, & Cooper, 1990). For eye movements, the prevention of postsaccadic drift is critically dependent on the precise regulation of the step component of the pulse-step command (Optican & Robinson, 1980). Although it is likely that the step component is also regulated for limb movements, relatively little is known about this mechanism. For the purposes of this article, we assume the presence of a fixed step component and rely on nonlinear damping for causing the limb to stick at an end point. 3 Model Architecture Both limb and saccadic control systems are highly distributed, involving the cerebral cortex, basal ganglia, cerebellum, tectum, brain stem, and spinal cord. The focus here is on the special role of the cerebellum, which exerts its influence on movement by way of premotor networks. For both limb movements and saccades, there are two levels of premotor network. The upper level is the cortico-rubro-cerebellar network for the limb (Houk et al., 1993) and the tecto-reticulo-cerebellar network for saccades (Houk, Galiana, & Guitton, 1992; Arai, Keller, & Edelman, 1994). These upper-level networks feed control signals to a lower level comprising a propriospinal network for the limb (Alstermark, Lundberg, Pinter, & Sadaki, 1987a) and a brain stem burst network for saccades (Robinson, 1975). Since the emphasis in this article is on the cerebellar cortex, the premotor networks will be given only a formal representation. We assume that the propriospinal network, in analogy with the brain stem burst network, can generate only relatively
570
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk τ3
target
τ5 MFs
extracerebellar CF corrective command sparse expansive PC encoding PFs
B
A premotor circuits
τ2
spring-mass system
1
y
τ4
efference copy
0
τ1
(a)
state
Tlow
T high
s
(b)
Figure 2: Model architecture. (a) Block diagram. PC, Purkinje cell; MFs, mossy fibers; PFs, parallel fibers; CF, climbing fiber; τi , i = 1, . . . , 5, conduction delays. The labels A and B mark places in the feedback loop to which we refer in discussing the model’s behavior. (b) Dendritic zone hysteresis. DZ activation, y, switches from 0 to 1 when the input weighted sum, s, exceeds threshold Thigh , and switches from 1 to 0 when s drops below Tlow .
crude commands that typically produce dysmetric movements when it operates on its own. However, the system is capable of orthometric control when the cerebellum and upper premotor networks are operative. In order to focus on the critical control functions of PCs in the cerebellar cortex, we will represent the cortico-rubro-cerebellar network as simply an inverting mechanism that converts the inhibitory output of PCs into a positive command signal. For simplicity, we further assume that the output of this cortico-rubro-cerebellar network acts directly on spinal output rather than functioning through the propriospinal network. The model’s main component is a single unit representing a cerebellar PC, whose input is derived from a sparse, expansive encoding of MF signals (see Figure 2a). In defining how the MFs encode information about the spring-mass system, we followed what is known about movement-related MF signals in the intermediate cerebellum of the monkey, where MFs exhibit discharge patterns involving diverse combinations of tonic and phasic components, as well as a variety of onset times relative to the time of movement onset (Van Kan et al., 1993a). To represent this diversity, the model has a total of 2000 MFs, 800 of ˙ xeq , or xT (200 MFs which encode information about single variables x, x, devoted to each), with the remaining 1200 MFs encoding information about pair-wise combinations of these variables. Each of the MFs representing a single variable uses a saturated ramp encoding. For example, as the mass’s position increases, the firing rate of a pure position-related MF remains zero until a threshold is exceeded and then increases linearly until saturating at a maximum firing rate. The thresholds are distributed uniformly over the
A Cerebellar Model of Timing and Prediction
571
relevant variable ranges, and several slopes and saturation levels are used.1 In addition, the signal conveyed by each pure position and velocity MF is delayed relative to spring-mass movement by an amount chosen uniformly at random from between 15 and 100 ms (τ1 in Figure 2a). The delay ranges for this and the following types of MFs are within those observed for the intermediate cerebellum of the monkey (Van Kan et al., 1993a). The signals of the efference copy MFs (representing xeq ) are delayed between 40 and 150 ms (uniform random) relative to the motor command (τ4 in Figure 2a). The signal of each target position MF (representing xT ) is delayed between 0 and 100 ms (uniform random) from the start of a trial (τ5 in Figure 2a). The signal conveyed by each of the 1200 MFs representing pair-wise combinations of the single variables is a weighted sum of the signals of two single-variable MFs: 400 are combinations of pure x and x˙ MFs, 400 are combinations of pure x and xeq MFs, and 400 are combinations of pure xT and x˙ MFs. Within these classes, the pairs of MFs were chosen uniformly at random, and the weights, which are positive and sum to one for each MF, were selected uniformly at random. The relative number of MFs in these various classes is consistent with the proportions observed by Van Kan et al., (1993a). The total number of MFs was chosen for computational reasons: we wanted to ensure that the model could accurately represent the transformation required by the control task. We did not rule out the possibility that fewer MFs might also suffice. We set the efferent delay from the PC to the spring-mass system via premotor circuits to 100 ms (τ2 in Figure 2a), which is within the range observed for this pathway in the intermediate cerebellum of the monkey (Van Kan, Houk, & Gibson, 1993b), although we experimented with other values as well (see section 5.1). With this delay, the MF delay ranges described imply that the onset of movement-related discharge of the MFs that use efference copy information can lead movement onset by as much as 60 ms or lag it by as much as 50 ms. On the other hand, movement-related discharge of MFs relying on only proprioceptive information always lags movement by between 15 and 100 ms. Patterns of MF activity are recoded to form sparse activity patterns over 40, 000 binary parallel fibers (PFs), which synapse on the PC. This form of PF state encoding is similar to that used in numerous models of the cerebellum, such as those of Marr (1969) and Albus (1971). We selected this number of PFs to ensure that the model could realize the required transformation. With
1 Thresholds are distributed at uniform intervals over the ranges of the relevant vari˙ [0, 1] for xeq ; and [3, 7] cm for xT ). The ables ([−0.5, 7.5] cm for x; [−25, 25] cm/sec for x; slopes were set so that the ramp covers 50%, 25%, or 12.5% of the variable’s range. Half of the slopes are negative, so that the MF decreases in activity as its coded variable increases. Saturation levels differ slightly as a function of threshold, with higher thresholds being associated with higher saturation levels. This roughly normalizes the average activity level of the MFs.
572
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
as few as 30,000 PFs, learning progresses at a slower rate and asymptotes at a higher average end point error. However, with 60,000 PFs, an improvement in learning performance is not observed. Each PF is the output of a granule unit that sums excitatory input from four randomly chosen MFs. We assumed that local competition takes place among granule units, allowing only 80 of the units to fire (output = 1) at the same time. Marr (1969) and Albus (1971) hypothesized that this competition arises from inhibitory interactions through Golgi cells. We implemented this competition by dividing the granule cell population into 80 Golgi-cell receptive fields, each comprising 500 granule units, and allowing only the most active unit in each field to fire at any time step of the simulation (although the model does not explicitly contain units representing Golgi cells). Thus, at each time step, the PF input to the PC is a pattern of 40,000 binary values containing 80 ones. The PC in the model consists of a number of dendritic zones (DZs) representing segments of the dendritic tree. Our representation of DZs is motivated by observations of plateau potentials in PC dendrites (Llin´as & Sugimori, 1980; Ekerot & Oscarsson, 1981; Campbell, Ekerot, Hesslow, & Oscarsson, 1983; Andersson, Campbell, Ederot, Hesslow, & Oscarsson, 1984). These long-lasting potentials (up to several hundred ms in duration) represent a form of bistability, which results from hysteresis produced by the dendritic ion channel system. A number of researchers have suggested that dendritic or neuronal bistability resulting from hysteresis can be computationally useful (Hoffman, 1986; Benson, Bree, Kinahan, & Hoffman, 1987; Kiehn, 1991; Houk et al., 1990; Wang & Ross, 1990; Gutman, 1991, 1994), and Yuen, Hockberger, and Houk (1995) showed how these properties can arise in a biophysical model of the PC dendrite. P Each DZ in the model is a linear threshold unit with hysteresis. Let s(t) = i wi (t)φi (t), where φi (t) denotes the activity of PF i at time t and wi (t) is the efficacy, or weight, at time step t of the synapse by which PF i influences the PC dendritic segment comprising the DZ. The activity of the DZ at time t, denoted y(t), is either 1 or 0, respectively, representing a state of high or low activity. DZ activity depends on two thresholds: Tlow and Thigh , where Tlow < Thigh . The activity state switches from 0 to 1 when s(t) > Thigh , and it switches from 1 to 0 when s(t) < Tlow (see Figure 2b). If Thigh = Tlow , the DZ is the usual linear threshold unit. Unlike plateau potentials, which tend to reset spontaneously after a few hundred milliseconds (Llin´as & Sugimori, 1980; Campbell et al., 1983; Andersson et al., 1984), the state of a DZ remains constant until actively switched by input. We have not yet explored the consequences of spontaneous resetting in our model. In the simulations reported below, we investigated the effects of several settings of Thigh and Tlow . The PC’s overall activity level at any time is equal to the fraction, f , of its DZs that are in state 1 at that time. In a more detailed model, the PC would inhibit nuclear cells, thereby regulating the buildup of activity in cortico-rubro-cerebellar loops from which motor commands are derived
A Cerebellar Model of Timing and Prediction
573
(Houk et al., 1993). The simpler model described here does not include an explicit representation of these premotor circuits. The motor command, xeq , is simply defined to be 4 f + 10(1 − f ), which means that when the PC is maximally active ( f = 1), the equilibrium position is the “near” position of 4 cm, and when it is minimally active ( f = 0), the equilibrium position is the “far” position of 10 cm. These values determine the range of target end points for which the model is able learn accurate positioning commands, but the model is not otherwise sensitive to the specific values. This definition of the motor command reflects the inhibitory effect the PC would have on cortico-rubro-cerebellar loops. As a result, pauses in the PC’s activity would disinhibit activity of premotor circuits, which activate an agonist muscle for rightward movement. We studied three versions of the model that differ in the number of DZs and how the PF input is distributed among them. In the simplest version, a single DZ receives input from all 40,000 PFs. In the other versions, the PC consists of 8 DZs. In one of these, each DZ receives input from all of the PFs; in the other, each DZ receives input from a separate subfield of 5000 PFs. The latter version of the model, which is more realistic due to the orthogonal relationship between PFs and the flattened dendritic trees of PCs in the cerebellum, learned somewhat slower than the other 8-DZ model, but its behavior was similar in other respects (see sec. 5.2). 4 Learning All the DZs comprising the PC in the model receive training information from a signal representing discharge of a climbing fiber (CF). This signal provides information about the spring-mass system with a delay of 20 ms (τ3 in Figure 2a), which is within the physiological range for CF signals in cats (Gellman, Gibson, & Houk, 1983). The nature of the training information supplied by the model’s CF is an extrapolation of what is known about the responsiveness of proprioceptive CFs, which respond to particular directions of limb movement and appear to signal “unexpected” passive movements, being suppressed during active (hence expected) movements (Gellman, Gibson, & Houk, 1985). We hypothesize that by monitoring the proprioceptive consequences of corrective movements generated by other structures, modules of the cerebellum can learn to regulate motor commands so that they produce more efficient and accurate movement. We follow Berthier et al. (1993) in assuming that the propriospinal premotor network generates simple corrective movements when a movement is inaccurate. These corrective movements do not have to be particularly accurate themselves; they only need to reduce the end point error. The literature on which these assumptions are based is reviewed in some detail in an earlier work (Berthier et al., 1993; see section 6). Although a sequence of such corrective movements alone can produce small final end point error, the sequence would be slow and dynamically erratic. The role of the cerebel-
574
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
lum, we hypothesize, is to eliminate the need for corrective movements by learning to suitably regulate the initial movement. In the model, whenever the mass is coming to rest at a point not near the target end point, an extracerebellar motor command in the form of a single rectangular pulse is generated, causing movement in the correct direction.2 In response to each rightward corrective movement, the model’s directionally-sensitive CF produces a single discharge. The CF is silent during leftward corrective movements (although the CF to a module activating a muscle for leftward movement, if one were present in the model, would discharge in this case). This follows a key assumption that the responsiveness of a PC’s CF to movement in a given direction is matched to the degree to which that PC’s module is capable of contributing to movement in that direction (see Berthier et al., 1993, for additional details). We also assume a low background firing rate for the CF in the absence of corrective movements. Letting c(t) denote CF activity at time step t, the model implements these assumptions by setting c(t) = 1 at the initiation of each rightward corrective movement, c(t) = 0 for the remainder of the rightward movement, c(t) = 0 during leftward corrective movements, and otherwise c(t) = β = 0.025, which represents a low background firing rate.3 As a result of a corrective movement, the weights of each DZ should change so that the PC contributes to a more accurate motor command. In response to a rightward corrective movement, the weights should change so as to increase the duration of the pulse phase of the command (since the movement stopped short of the target end point), and in response to a leftward corrective movement, the weights should change so as to decrease the duration of the pulse phase of the command (since the movement overshot the target end point). Accomplishing this with a simple learning rule is difficult because the training information in the form of CF activity is significantly delayed with respect to the relevant DZ activity due to the combined effects of movement duration and conduction latencies. To learn under these conditions, the model adopts Klopf’s (1972, 1982) hypothesis of synaptic “eligibility traces.” Appropriate activity at a synapse is hypothesized to set up a synaptically local memory trace that makes the synapse “eligible” for modification if and when the appropriate training information arrives within a short time period. This allows the learning rule to modify synaptic weights based on synaptic actions that occurred prior to the avail-
2 Whenever the mass has been “stuck” for 150 ms more than 0.1 cm from the target end point, the motor command, xeq , is set to a value that causes movement toward the target. Specifically, xeq = xT + a for undershoot and xeq = xT − a for overshoot, where a > 0 was chosen to be sufficiently large to overcome the high low-velocity viscosity. Here, we used a = 5 cm. 3 We experimented with a more realistic representation of background activity in which c(t) = 1 with probability β for each background time step t. The results were essentially the same, except that the learning process required about 2.5 times as many trials.
A Cerebellar Model of Timing and Prediction
575
ability of the relevant CF training information. An example eligibility trace for one PF-to-PC synapse is shown in the bottom plot of Figure 3a. This eligibility trace spans the interval from the time of the presynaptic PF’s activity until later CF discharges occur (plot CF in the figure), when this synapse’s weight is modified. To define the learning rule, we have to specify how synapses become eligible for modification and how CF activity alters the synaptic weights based on the eligibility of synapses. We first describe the eligibility process. The idea is that a synapse becomes eligible for modification if its presynaptic PF was active in the recent past at the same time that the synapse’s DZ was in state 1. Eligibility then persists as a graded quantity—a trace—that reflects both how frequently and how long in the past this eligibility-triggering condition was satisfied for that synapse. Although learning is not sensitive to the exact time course of eligibility traces, a synapse should reach peak eligibility at roughly the time at which a relevant CF discharge would reach the PC. By a relevant CF discharge, we mean one produced by a correction to a movement that was influenced by the eligibility-triggering activity at the given synapse. One of the simplest methods for computing eligibility is to simulate a second-order linear filter whose input is 1 whenever the triggering condition is satisfied and 0 otherwise. The filter’s parameters were set so that its impulse response rises to a peak about 255 ms after the triggering event, and then decays asymptotically to zero with a time constant of approximately 600 ms. A synapse is therefore maximally eligible 255 ms after the triggering event and becomes effectively ineligible approximately 2 sec later, assuming no additional triggering events occur (see the bottom plot of Figure 3a). This time course is appropriate for the movement durations and conduction delays in this model. An intracellular signal transduction mechanism for producing this kind of eligibility trace was proposed in Kettner et al. (1997). We also found it useful to limit the magnitude of eligibility so that prolonged periods during which the triggering condition is satisfied do not lead to excessively high eligibility, and hence to large weight changes. In the discussion, we comment on the biological realism of the eligibility idea. Letting ei (t) denote the eligibility of synapse i at time t, the model generates the eligibility trace for each synapse i by the following difference equations involving the intermediate variables e¯i and eˆi : e¯i (t) = .98¯ei (t − 1) + .02y(t)φi (t), eˆi (t) = .98ˆei (t − 1) + .02¯ei (t − 1), ei (t) = min{ˆei (t), 0.1}, where y(t) is the binary activity state of the synapse’s DZ at time step t, and φi (t) is the activity of the presynaptic PF. Each time step in the model represents 5 ms of real time.
576
s f EC Xeq (cm)
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
Thigh 1 0 right 0 left 10
x (cm) . x (cm/s)
CF
0
τ2
5 0 25 0 1 0 1
PF, eligibility 0
0
0.5
1
1.5
2
0
0.5
t (sec)
t (sec)
(a)
(b)
Figure 3: Single DZ behavior. The target end point, xT , was switched from 0 to 5 cm at time 0. Shown are the time courses of the DZ’s summed input, s; activation state, f ; extracerebellar corrective command, EC; motor command, xeq (after the 100 ms efferent delay τ2 ); and the position, x, and velocity, x˙ , of the mass for a movement that started at initial position x0 = 0. Plot CF shows climbing fiber activity, and the bottom plot shows the binary activity of an arbitrarily selected PF together with the eligibility trace of its synapse onto the DZ. (The eligibility trace’s amplitude is scaled up to make it easily visible; peak eligibility here is 0.029.) Tlow = 0.8 and Thigh = 1. (a) Early in learning (four trials). DZ state switched to 1 too soon, which caused the mass to undershoot the target. A sequence of six rightward corrective movements was generated by the extracerebellar system (EC) because all but the last failed to bring the mass close enough to the target. Each corrective movement caused a CF discharge. Each discharge of the selected PF contributed to the eligibility trace because the DZ was in state 1 at these times. The weight of this PF’s synapse (not shown) decreased when the CF discharge coincided with nonzero eligibility. (b) Late in learning (1000 trials). The model consistently produced accurate reaching with fast, smooth movements requiring no corrections (and hence with no CF discharges). To accomplish this, the DZ learned to switch to 1 well before (about 300 ms) the end point was reached.
The remainder of the model’s learning mechanism is a rule determining how the weights of eligible synapses are altered by CF activity. The logic of this learning rule is a result of the following reasoning. When the weights of a DZ’s eligible synapses decrease, that DZ becomes less likely to switch
A Cerebellar Model of Timing and Prediction
577
to state 1 in the future when a situation (represented by a pattern of PF activity) is encountered that is similar to the one that was present when the eligibility trace was initiated. This tends to prolong the pulse phase of the motor command by delaying the DZ’s contribution to PC inhibition, which increases movement duration and moves the initial movement’s end point to the right. Thus, the weights of eligible synapses should decrease as a result of each rightward corrective movement. Since the CF produces a discharge on each rightward corrective movement, CF discharge should cause depression of the eligible synapses. On the other hand, increasing the weights of a DZ’s eligible synapses makes that DZ more likely to switch to state 1 under similar circumstances in the future, which tends to shorten the pulse phase of the motor command, thus decreasing movement duration and moving the end point leftward. Therefore, leftward corrective movements should cause potentiation of the eligible synapses. In the model, this is accomplished by letting the CF signal drop below its background rate during leftward corrective movements. The following learning rule implements this logic: 1wi (t) = −αei (t)[c(t − τ3 ) − β], wi (t) = max{wi (t − 1) + 1wi (t), 0},
(4.1)
where α > 0 is a parameter influencing the rate of learning that was set to 2 × 10−3 in the simulations described below.4 The term β implies that weights do not change during background CF activity and that eligible weights increase during leftward corrective movements when CF activity drops below its background rate. Note that since β ¿ 1, weight increases are much smaller than weight decreases. The term c(t − τ3 ), where τ3 = 20 ms is the CF conduction delay, is the CF signal that reaches the synapse at time t. Since eligibility, ei (t), is a multiplicative factor, weights change in proportion to their degree of eligibility. All the DZs comprising the model’s PC learn independently according to this rule. To summarize the model’s learning mechanism, training information is supplied by CF responses to corrective movements. The CF for the single module described here discharges reliably in response to rightward corrective movements. This follows from the specificity of the CF system and the assumption that this module controls an agonist for rightward movement. Rightward corrective movements therefore raise the CF’s activity above its background rate. For leftward corrective movements, the CF’s activity decreases slightly below its background rate. The weights of the synapses from PFs to the DZs comprising the model’s PC change in response to CF activity so that the duration of the pulse phase of the motor command is increased 4 This was chosen to be such a small value because the resultant change in PC activation due to each learning step could be 80 times larger since 80 of the PF inputs are 1 at each time step.
578
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
in the case of rightward corrective movements and decreased in the case of leftward corrective movements. The model uses eligibility traces to bridge the time interval between the activity of the DZs and the relevant later CF activity. A synapse becomes eligible for modification when presynaptic activity coincides with the postsynaptic DZ being in state 1. Eligibility is realized as a synaptically local trace that persists for several seconds after the coincidence of pre- and postsynaptic activity. When CF activity rises above its background level, the weights of the synapses are depressed in proportion to their current degree of eligibility, which tends to lengthen the pulse phase of the command. When CF activity falls below its background level, synapses are facilitated in proportion to their eligibility, which tends to shorten the pulse phase of the command. 5 Simulations 5.1 Single Dendritic Zone. We performed a number of simulations of a single DZ learning to control the nonlinear spring-mass system. We trained the DZ to move the mass from initial positions selected randomly from the interval [0, 2 cm] to a target position randomly set to 3, 4, or 5 cm. DZ state 0 corresponded to the pulse phase of a motor command, which set a “far” equilibrium position of 10 cm; DZ state 1 corresponded to the step phase, which set a “near” equilibrium position of 4 cm (see section 2). Each simulation consisted of a series of trial movements. At the beginning of the first trial movement, we randomly initialized all 40,000 weights so that the weighted sum, s, fell uniformly between 0.68 and 1.48 for any initial pattern of PF activity. Each trial began when the state of the DZ was set to 0. We initialized each eligibility trace, ei (t), to 0 (¯ei (t) and eˆi (t) were also set to 0). We also set the pattern of MF activity to be consistent with the initial state of the spring-mass system. To study the influence of loop delay on learning and performance, we conducted simulations in which the loop delay was varied by setting the efferent delay (τ2 in Figure 2a) to 75, 100, or 125 ms. Figure 4a shows how the end point error decreased with trials for the various efferent delays (with Tlow = 0.8 and Thigh = 1). The DZ’s behavior is largely insensitive to this range of delays. In each case, the average absolute error rapidly dropped below 0.1 cm (see the dotted line in Figure 4a), the trigger criterion for the extracerebellar corrective movement. In all the simulations reported below, we set τ2 = 100 ms. However, the model’s behavior is sensitive to the amount of DZ hysteresis. Figure 4B shows how end point error decreased over trials for several different values of Tlow , with Thigh fixed at 1. Learning was seriously disrupted when there was no hysteresis (Tlow = Thigh = 1). In all simulations reported below, Tlow = 0.8 and Thigh = 1, unless noted otherwise. Figure 3 shows time courses of relevant variables at different stages in learning to move to target end point xT = 5 cm from initial position x0 = 0.
A Cerebellar Model of Timing and Prediction 2
0.4 0.35
75 ms 100 ms 125 ms
0.3
Average Absolute Error (cm)
Average Absolute Error (cm)
579
0.25 0.2 0.15 0.1
1.5
Tlow = 0.8 Tlow = 0.9 T = 1 (no hysteresis)
1
low
0.5
0.05 0 0
1000
2000
3000 Trial
(a)
4000
5000
0 0
1000
2000
3000
4000
5000
Trial
(b)
Figure 4: End point error (|xe − xT |) as a function of trial number for single DZ learning. Each plotted point is an average over a bin of 50 trials of 10 learning runs. The dotted horizontal line shows the minimum threshold above which corrective movements were generated. (a) Effect of loop delay. Plots for efferent delays (τ2 ) of 75, 100, and 125 ms. Here, Tlow = 0.8 and Thigh = 1. (b) Effect of hysteresis. Plots for Thigh = 1 and Tlow = .8, .9, and 1 (no hysteresis). Efferent delay (τ2 ) was 100 ms.
Early in learning (four trials, panel A), the DZ switched back to state 1 too soon (plot f ), which caused the mass to undershoot the target. Because of this undershoot, the extracerebellar system (EC) generated a corrective movement. In fact, a sequence of six corrective movements was generated because all but the last failed to bring the mass close enough to the target. Each corrective movement caused a CF discharge. The resulting movement accurately reached the target, but along a slow and irregular trajectory. Plotted at the bottom of Figure 3 is the binary activity of an arbitrarily selected PF and the eligibility trace of its synapse onto the PC. Note that each discharge of the PF contributed to the trace because the DZ was in state 1 at these times. The weight of this PF’s synapse (not shown) decreased when the CF discharge coincided with nonzero eligibility. The decrease of this weight, along with decreases of many others, tended to prolong the pulse phase of the motor command by delaying the DZ’s switch to state 1. None of the synaptic weights increased during this trial because there was no leftward corrective movement (see Figure 7a for an example of a trial with a leftward corrective movement). Later in learning (after 1000 trials, Figure 3b), the model consistently produced accurate reaching with fast, smooth movements requiring no corrections (and hence causing no CF discharges). To accomplish this, the DZ learned to switch to state 1 well before (about 300 ms) the end point was reached. Figure 5a shows the paths of a number of movements controlled by a well-trained DZ. The initial position of the mass for each movement is indicated by the circle at the left end of each line, and the target end points are indicated by the vertical dashed lines. The asterisk on each path marks
580
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk 30 A
Velocity (cm/s)
25 20 15
B
10 5
0
1
2 3 4 Position (cm)
(a)
5
6
0 0
2 4 Position (cm)
6
(b)
Figure 5: (a) Paths of a number of movements controlled by a single well-trained DZ. The path of each movement is shown by a horizontal line. The initial position of the mass for each movement is indicated by the circle at the left end of each line, and the target end points are indicated by the vertical dashed lines. The asterisk on each path marks the position of the mass when the PC state switched from 0 to 1. The actual end point of the movement is indicated by the × at the right end of each line. The DZ switches state well before a movement ends. The model used a motor efference delay of 100 ms. (b) Switching curves. Phase-plane portraits of switching curves for target xT = 5 cm learned by the model. Two switching curves and three example movement trajectories are shown. See the text for an explanation.
the position of the mass when the DZ switched state from 0 to 1. The end point of each movement is indicated by the × at the right end of each line. One can see that the movements were accurate across a range of initial positions and target end points. It is apparent that the DZ switched state well before the end of each movement. Figure 5b shows two representations (the dashed lines labeled A and B) of the switching curve learned by the DZ for target xT = 5 cm, together with three sample phase-plane trajectories. Switching curve A is the switching curve as it appears after the efferent delay τ2 , that is, as seen from the point marked A in Figure 2a. When the spring-mass system’s state crosses this curve, the command input to the spring switches from pulse to step. Clearly, it is positioned correctly to cause the mass to stick close to the desired end point for a range of initial conditions. Switching curve B, on the other hand, is the switching curve as it appears before the efferent delay, that is, as seen from the point marked B in Figure 2a. This curve is crossed 100 ms before switching curve A is crossed due to the 100 ms efferent delay. When the system state crosses this curve, the DZ switches state. One can see that the DZ learned to switch 100 ms before the motor command must switch at
A Cerebellar Model of Timing and Prediction
581
the spring itself, appropriately compensating for the 100 ms latency of the efferent pathway. To do this, the DZ effectively learned to “recognize” the patterns of PF activity that were present at its synapses when the system state crossed switching curve B. It is important to note that due to the various delays in the MF pathways, the recognized PF patterns actually encoded information about the spring-mass state as it was between 15 and 100 ms earlier. 5.2 Multiple Dendritic Zones. We simulated two versions of the model in which the PC consists of eight DZs. In each case, the PC’s activity level at any time is the fraction of its DZs that are in state 1 at that time (see section 3). In one version, each DZ receives input from all of the PFs (uniform model); in the other, each DZ receives input from a separate subfield of 5000 PFs (subfield model). Figure 6 shows how the end point error decreased with trials for these two variations, as well as for the single DZ model. The uniform model learned significantly faster than the others and reached a smaller final error. We believe this is due to the fact that the uniform model has many more adjustable parameters than the others so that there are many different potential solutions: the algorithm is constrained only to reduce the end point error, which can be accomplished in many ways. To save computer time, we restricted further simulations to the uniform model, but it is likely that the subfield model would have produced similar results. Figure 7 illustrates some details of the behavior of the uniform model. Panel a is analogous to Figure 4a except that it shows a trial in which there was a single leftward corrective movement instead of multiple rightward corrective movements. Note that the leftward corrective movement did not generate CF discharges but instead slightly depressed CF background rate. Unlike the single DZ case, here the motor command was graded due to the varying contributions of the eight DZs. This variety was due to the differing initial weights of the DZs. Later in learning (1500 trials, panel b), fast, accurate, and smooth movements were accomplished, although the motor command was not a pure pulse. We also investigated the effects of different levels of hysteresis on the uniform model by fixing Thigh to 1 and varying Tlow , as we did for the single DZ system (Figure 4b). Unlike the single DZ case, hysteresis had no significant effect on the learning rate of the uniform model. However, we did note that without hysteresis (Tlow = 1; see Figure 8), the final motor command was more irregular than it was with hysteresis. This was the result of multiple switching by approximately half of the DZs. 6 Discussion By including realistic conduction delays in afferent and efferent pathways, the model described here allowed the investigation of timing and predictive processes relevant to cerebellar involvement in the control of movement.
582
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
Average Absolute Error (cm)
0.4 0.35
1 DZ 8 DZs 8 DZs (subfields)
0.3 0.25 0.2 0.15 0.1 0.05 0 0
1000
2000
3000
4000
5000
Trial Figure 6: End point error (|xe − xT |) as a function of trial number for multi-DZ learning. In the uniform model, each DZ receives input from all of the PFs; in the subfield model, each DZ receives input from a separate subfield of 5000 PFs. Also shown is a plot for the single DZ model. The uniform model learned significantly faster than the others and reached a smaller final error. Each plotted point is an average over a bin of 50 trials of 10 learning runs. The dotted horizontal line represents the minimum threshold above which corrective movements were generated. Tlow = 0.8, Thigh = 1, and τ2 = 100 ms.
Moreover, the nonlinearity of the simple motor plant, which is based on muscle mechanical and spinal reflex properties, makes the control problem reflect properties of skeletomotor control better than would a simpler linear plant. While making the control problem more difficult from a conventional control perspective, the nonlinear damping has the advantage of allowing fast movements to be made with little or no oscillation, effectively solving the stability problem, at least for the one-degree-of-freedom positioning task studied here. Key to the model’s ability to perform accurate end point positioning is its ability to learn predictive control. This is illustrated most clearly in the case of a single DZ, for which clear switching curves could be derived and related to plant dynamics (see Figure 5b). The model’s relative insensitivity to loop delay is due to its predictive use of a rich array of afferent and efference-copy
A Cerebellar Model of Timing and Prediction
s f EC
583
Thigh 1 0 right 0 left 10
Xeq (cm) 0
x (cm) . x (cm/s)
CF
5 0 25 0 1 0 0
0.5 t (sec) (a)
0
0.5 t (sec) (b)
Figure 7: Multiple DZ behavior. This figure is analogous to Figure 3 but for a PC consisting of eight DZs, each receiving input from all the PFs (uniform model). The target end point, xT , was switched from 0 to 5 cm at time 0. Shown are the time courses of the eight DZs’ summed inputs, s; the PC’s activation state, f ; extracerebellar corrective command, EC; motor command, xeq (after the 100 ms efferent delay τ2 ); and the position, x, and velocity, x˙ , of the mass for a movement that started at initial position x0 = 0. The bottom plot shows CF activity. Tlow = 0.8 and Thigh = 1. (a) Early in learning (250 trials). One of the DZs switched back to 1 too late, which caused the mass to overshoot the target slightly. The extracerebellar (EC) system generated a leftward corrective movement, which decreased CF activity below its low background level. (b) Late in learning (1500 trials). The model consistently produced accurate reaching with fast, smooth movements requiring no corrections. Note that the command is still basically a pulse-step, although it is no longer binary.
signals. The model does not explicitly predict the motor plant’s behavior; that is, it does not use a forward model of the motor plant, a role suggested for the cerebellum by several researchers (Ito, 1984; Keeler, 1990; Miall, Weir, Wolpert, & Stein, 1993). In fact, the model makes no explicit predictions of any kind, if this is taken to mean the creation of representations of future events. Instead, it learns to generate motor commands in a manner that
584
s f EC
Xeq (cm) x (cm) . x (cm/s)
CF
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
T high 1 0 right 0 left 10 0 5 0 25 0 1 0 0
0.5
1
t (sec) (a)
0
0.5
t (sec) (b)
Figure 8: Multiple DZ behavior without hysteresis. This figure is analogous to Figure 7 except that there was no hysteresis (Tlow = 1). (a) Early in learning (175 trials). (b) Late in learning (1,000 trials). The motor command, xeq , was more irregular than it was with hysteresis. This was the result of multiple switching by approximately half of the DZs.
causes desired future behavior. The model is a kind of direct adaptive controller (e.g., Goodwin & Sin, 1984), where the term direct refers to the lack of a model of the controlled system. Our model adopts the hypothesis of Marr (1969) and Albus (1971) that the granular layer provides a sparse expansive encoding that increases the ease with which a large number of associations can be formed (Buckingham & Willshaw, 1992; Tyrrell & Willshaw, 1992). We combined this hypothesis with a more realistic representation of movement-related MF signals (Van Kan et al., 1993a). Although the model’s use of such a large number of PFs, and hence adjustable parameters (the PF-to-PC synaptic weights), for such a simple task is a defect from a purely engineering perspective, it is a result
A Cerebellar Model of Timing and Prediction
585
of our attempt to represent faithfully what is known about how information is encoded in the MF signals, coupled with our use of a random selection of MF inputs to granule units. Careful design of the latter connection pattern would decrease the number of PFs required. However, simulations show that the current model’s behavior degrades if the number of PFs is significantly decreased. We did not investigate the generalization capabilities of the model, which would also be influenced by the input encoding and the number of adjustable parameters. Experimental data on the effects of CF discharge on PF-to-PC synapses suggest an instructive role for CF signals, as adopted by the model. There now seems to be good, though not universal, agreement that CF activity, when coupled with other factors, produces long-term depression (LTD) of the action of PF-to-PC synapses (e.g., Crepel et al., 1996), as postulated by Albus (1971). Less is known about possible long-term potentiation (LTP) at these synapses, which the model also uses, although LTP has been induced in brain slices by stimulating PFs in the absence of CF activity (Sakurai, 1987), which is consistent with our model’s learning rule. An essential feature of the model’s learning rule is its use of synaptically local eligibility traces for learning with delayed training information. Eligibility traces are key components of many reinforcement learning systems (e.g., Sutton & Barto, 1998) as well as models of classical conditioning (Sutton & Barto, 1981, 1990; Klopf, 1988), where they address the sensitivity of conditioning to the time interval between the conditioned and the unconditioned stimuli and the anticipatory nature of the conditioned response. Eligibility traces play the same role in this model, whose learning mechanism is much like classical conditioning, with corrective movements playing the role of unconditioned responses.5 Our model is therefore in accord with the view that general principles of cerebellar-dependent learning may be involved in adaptation of the vestibulo-ocular reflex, classical conditioning of the eyelid response, as well as learning in saccadic eye movements and limb movements (Houk et al., 1996; Raymond, Lisberger, & Mauk, 1996). We hypothesize that for reaching, the role of the cerebellum is to eliminate corrective movements by suitably tuning the initial movement. Only a few studies of cerebellar plasticity have attempted to manipulate the relative timing of the experimental variables used to elicit LTD. In several studies, LTD occurred only if CF stimulation preceded PF stimulation (Ekerot & Kano, 1989; Schreurs & Alkon, 1993). Recently, however, Chen and Thompson (1995) demonstrated that delaying CF activation by 250 ms after a PF volley facilitates the appearance of LTD, suggesting that there may be a cellular mechanism that compensates for the time interval. Schreurs, Oh,
5 The present model lacks the ability to produce an analog of higher-order conditioning, one of the key features of the classical conditioning models. We know of no studies of the cerebellum’s involvement in higher-order classical conditioning.
586
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
and Alkon (1996) showed that a form of LTD, which they call pairing-specific LTD, results only when PF stimulation precedes CF stimulation. Although these studies were motivated by the timing parameters required for classical conditioning of the rabbit nictitating membrane response, their results are relevant to other aspects of motor learning as well. Houk and Alford (1996) presented a model suggesting how intracellular signal transduction mechanisms that mediate LTD could give rise to an eligibility trace. Recent results in which the timing of intracellular signals was controlled photolytically appear to suggest that CF activity should precede PF activity in order to produce LTD (Lev-Ram, Jiang, Wood, Lawrence, & Tsien, 1997). However, this conclusion depends critically on the interpretation given to the various intracellular signals. We hope that the computational importance of a trace mechanism will stimulate additional cellular studies to explore this critical issue. The nature of the training information provided by climbing fibers is incompletely understood. In oculomotor regions of the cerebellum, CFs are sensitive to retinal slip and thus are well suited to detect errors in the stabilization of visual input. By analogy, one presumes that the somatosensory sensitivity of CFs in limb regions has an analogous error-detection function, although this has been difficult to specify in detail (Fu, Mason, Flament, Coltz, & Ebner, 1997; Houk et al., 1996; Kitazawa, Kimura, & Yin, 1998; Simpson, Wylie, & de Zeeuw, 1996). In this model, we adopted our earlier working hypothesis that CFs detect hypometria by responding to corrective movements in the same direction as the primary movement (Berthier et al., 1993). This was rationalized from the finding that CFs with directional sensitivity to passive limb movements (units located in the rostral medial accessory olive) are inhibited during self-generated movements (Gellman et al., 1985) but fire when perturbations occur during or at the end of the movement (Andersson & Armstrong, 1987; Horn, Van Kan, & Gibson, 1996). We assume that corrective movements occur near the end of inaccurate movements and that they function like perturbations to fire CFs in a directionally selective manner. Reaching movements are known to consist of a primary movement, which is often succeeded by one or more secondary movements, the latter being corrective in nature (Prablanc & Martin, 1992). Lesion studies have demonstrated the involvement of several neural pathways in the generation of both the primary movements and the corrections (Pettersson, Lundberg, Alstermark, Isa, & Tantisira, 1997). Small corrections do not require vision of the arm and are often made without subject awareness and at shorter latencies than the primary movements (Goodale, P´elisson, & Prablanc, 1986). These findings suggest the involvement of a simple, automatic mechanism such as the propriospinal network (Alstermark, Eide, Gorska, ´ Lundberg, & Pettersson, 1984; Alstermark, Gorska, ´ Pettersson, & Walkowska, 1987b). Major corrections, such as reversals in direction, engage the corticospinal system (Georgopoulos, Kalaska, Caminiti, & Massey, 1983). In this model,
A Cerebellar Model of Timing and Prediction
587
we assumed that all corrections following primary movements are made by a simple, extracerebellar process presumed to be mediated by the propriospinal system. This was meant to be a minimalistic assumption; the model could have made use of training information derived from more accurate corrective movements generated, for example, by the corticospinal system. In fact, since training information is derived from the proprioceptive consequences of corrective movements, the model is capable of learning from corrections generated by any system or combination of systems. We also used the model to experiment with possible computational roles for plateau potentials in PC dendrites (Llin´as & Sugimori, 1980; Ekerot & Oscarsson, 1981; Campbell et al., 1983; Andersson et al., 1984). Our representation of DZs as linear threshold elements with hysteresis allows them to produce abstract analogs of plateau potentials. Hysteresis is sometimes used in two-action control systems to reduce “chattering” caused by repeated crossing of the switching curve. It has the same effect here in making the DZs switch state less frequently, which makes the model’s motor commands less erratic. Hysteresis greatly facilitated learning in the singleDZ case, presumably because it prevented chattering in motor commands, thereby making them closer to the pulse-step form and reducing the amount of learning required. In the multiple-DZ case, hysteresis had little influence on learning, perhaps because the motor commands were relatively smooth without hysteresis since they resulted from the activity of multiple DZs. We did, however, observe increased chatter in the pulse-step command when hysteresis was removed (see Figure 8), suggesting that hysteresis could have a role in facilitating the generation of well-formed motor commands. More study is needed to explore possible computational roles of nonlinear properties of PC dendrites. Several previous cerebellar models dealing with eye movement are closely related to the model of limb control presented here. Like our model, the model of adaptive control of saccades due to Schweighofer, Arbib, and Dominey (1996a, b) follows Berthier et al. (1993) and Houk et al. (1990) in making use of corrective movements as sources of training information. Schweighofer et al. also use eligibility traces following the classical conditioning models of Sutton and Barto (1981, 1990). Unlike the monotonically decaying traces in these models, however, the eligibility traces of Schweighofer et al. reach peaks sometime after being initiated. This is in accord with Klopf’s (1972) original conception that peak eligibility occurs at the optimal interstimulus interval for learning (see also Klopf, 1988). Our model also adopts this type of eligibility trace. The key differences between our model and that of Schweighofer et al. are due to differences in the dynamics of the motor plant and the degree of attention paid to system delays and afferent encoding. Because we are concerned with limb movement, our motor plant has significant inertia, which, together with nontrivial delays in various conduction channels, requires significant anticipatory control as illustrated by our simulations. The eye plant of the Schweighofer et al.
588
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
model lacks significant dynamics (the plant is essentially inertia-less), and it is not apparent that conduction delays are included. The ramp encodings we use for most of the MF signals are also more faithful representations of experimentally observed MF encodings. Our model also shares features with the model of predictive smoothpursuit eye movements due to Kettner et al. (1997). Like ours, this model includes MF inputs with diverse response properties and delays, a granular layer that expansively recodes this input, and a similar learning rule using eligibility traces generated by a second-order linear system. The PCs of that model, however, are continuous elements as opposed to the multistable ones used in our model (although as the number of DZs is increased, our model more closely approximates a continuous system). Additionally, training information in the Kettner model is provided by CFs that detect failures of image stabilization (retinal slip) instead of corrective eye movements. A model of limb movement related to ours is the feedback-error learning model of Kawato and Gomi (1992, 1993) in which the cerebellum learns to act as an inverse dynamic model of the motor plant, being trained by feedback generated from movement caused by an extracerebellar system. This is similar to what we have done in the our model, with two exceptions. First, our training information is intermittent feedback from discrete corrective movements instead of a continuous feedback signal. Second, unlike feedback-error learning models, as well as the limb control model of Schweighofer (1995), we do not assume that reference trajectories specifying the complete kinematic details of the desired movement are supplied to the cerebellum by another brain region. Therefore, we do not hypothesize that the cerebellum becomes an inverse dynamic model of the plant in the sense of associating a reference trajectory to appropriate control signals. Target signals in our model do not convey this kind of detailed information about the desired trajectory. Instead, through learning, target signals become associated with movements whose kinematic details are determined by the properties of the motor plant. Our model therefore has elements in common with the equilibrium-point hypothesis (Bizzi et al., 1992; Feldman, 1966, 1974) in that muscles and spinal reflexes play essential roles in trajectory formation. Unlike that hypothesis, however, movement end points are generally not equilibrium positions. The model presented in this article has a number of limitations. It lacks representations of many of the components of the full APG model on which it is based. In that model, movement would be the result of the combined effects of the elemental commands of a number of cerebellar APG modules that operate simultaneously. Here, we described only a single module consisting of a single PC and included no explicit representation of premotor circuits. Because the model presented here consists of a single PC controlling a single agonist actuator, it does not illustrate critical features of the full model. It does not show, for example, that during a movement, most PCs would have to increase activity to inhibit muscle synergies that should
A Cerebellar Model of Timing and Prediction
589
not fully participate in the movement. In the model presented here, the single PC always has to decrease activity to generate a motor command. Our model also suggests that after learning, the extracerebellar source of corrective movements no longer plays a role in limb movement. This is consistent with the feedback-error learning model but at variance with models of saccade generation in which cerebellar control augments, rather than replaces, the control provided by the brain stem burst generator (Dean, 1995; Arai et al., 1994; Optican, 1995). Our model does not adopt this approach because much less is known about the propriospinal network than is known about the brain stem pulse generator. However, this would be worthwhile to pursue in future research. Finally, nothing in this article suggests how the model presented here might extend to more complex control problems involving multi-degreeof-freedom limbs. One of the objectives of the full APG model is to explore how the collective behavior of multiple APG modules can accomplish pulsestep control of a more complex motor plant without resorting to preplanned reference trajectories. Our research is continuing in this direction (Fagg, Sitkoff, Barto, & Houk, 1997a, b). Acknowledgments This work was supported by NIH 1-50 MH 48185. We thank Jay Buckingham for his contributions to an earlier version of this model and Sascha Engelbrecht for helpful comments. References Albus, J. S. (1971). A theory of cerebellar function. Mathematical Biosciences, 10, 25–61. Alstermark, B., Eide, E., Gorska, ´ T., Lundberg, A., & Pettersson, L. G. (1984). Visually guided switching of forelimb target reaching in cats. Acta. Physiol. Scand., 120, 151–153. Alstermark, B., T. Gorska, ´ A. L., Pettersson, L. G., & Walkowska, M. (1987b). Effect of different spinal cord lesions on visually guided switching of targetreaching in cats. Neuroscience Research, 5, 63–67. Alstermark, B., Lundberg, A., Pinter, M. J., & Sadaki, S. (1987a). Long C3-C5 propriospinal neurones in the cat. Brain Research, 404, 382–388. Andersson, G., & Armstrong, D. M. (1987). Complex spikes in Purkinje cells in the lateral vermis (b zone) of the cat cerebellum during locomotion. Journal of Physiology, London, 385, 107–134. Andersson, G., Campbell, N. C., Ekerot, C. F., Hesslow, G., & Oscarsson, O. (1984). Integration of mossy fiber and climbing fiber inputs to Purkinje cells. Experimental Brain Research (Suppl), 9, 145–150. Arai, K., Keller, E. L., & Edelman, J. A. (1994). Two-dimensional neural network model of the primate saccadic system. Neural Networks, 7, 1115–1135.
590
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
Barto, A. G., Buckingham, J. T., & Houk, J. C. (1996). A predictive switching model of cerebellar movement control. In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo (Eds.), Advances in neural information processing systems: Proceedings of the 1995 Conference (pp. 138–144). Cambridge, MA: MIT Press. Benson, M. W., Bree, G. M., Kinahan, P. E., & Hoffmann, G. W. (1987). A teachable neural network based on an unorthodox neuron. Physica D, 22, 233–246. Berthier, N. E., Singh, S. P., Barto, A. G., & Houk, J. C. (1993). Distributed representations of limb motor programs in arrays of adjustable pattern generators. Journal of Cognitive Neuroscience, 5, 56–78. Bizzi, E., Hogan, N., Mussa-Ivaldi, F. A., & Gister, S. (1992). Does the nervous system use equilibrium-point control to guide single and multiple joint movements? Behavioral and Brain Sciences, 15, 603–613. Buckingham, J. T., Barto, A. G., & Houk, J. C. (1995). Adaptive predictive control with a cerebellar model. In Proceedings of the 1995 World Congress on Neural Networks (pp. 373–380). Mahwah, NJ: Erlbaum. Buckingham, J. T., & Willshaw, D. (1992). A note on the storage capacity of the associative net. Network: Computation in Neural Systems, 3(4), 404–414. Campbell, N. C., Ekerot, C. F., Hesslow, G., & Oscarsson, O. (1983). Dendritic plateau potentials evoked in Purkinje cells by parallel fibre volleys in the cat. Journal of Physiology (Lond), 340, 209–223. Chen, C., & Thompson, R. F. (1995). Temporal specificity of long–term depression in parallel fiber—Purkinje synapses in rat cerebellar slice. Learning and Memory, 2, 185–198. Crepel, F., Hemart, N., Jaillard, D., & Daniel, H. (1996). Cellular mechanisms of long-term depression in the cerebellum. Behavioral and Brain Sciences, 19(3), 347. Dean, P. (1995). Modelling the role of the cerebellar fastigial nuclei in producing accurate saccades: The importance of burst timing. Neuroscience, 68, 1059– 1077. Ekerot, C. F., & Kano, M. (1989). Stimulation parameters influencing climbing fiber induced long-term depression of parallel fiber synapses. Neuroscience Research, 6, 264–268. Ekerot, C. F., & Oscarsson, O. (1981). Prolonged depolarization elicited in Purkinje cell dendrites by climbing fiber impulses in the cat. Journal of Physiology (Lond), 318, 207–221. Fagg, A. H., Sitkoff, N., Barto, A. G., & Houk, J. C. (1997a). Cerebellar learning for control of a two-link arm in muscle space. In Proceedings of the IEEE Conference on Robotics and Automation (pp. 2638–2644). Fagg, A. H., Sitkoff, N., Barto, A. G., & Houk, J. C. (1997b). A model of cerebellar learning for control of arm movements using muscle synergies. In Proceedings of the IEEE Symposium on Computational Intelligence in Robotics and Automation (pp. 6–12). Feldman, A. (1966). Functional tuning of the nervous system with control of movement or maintenance of a steady posture. II Controllable parameters of the muscle. Biophysics, 11, 565–578. Feldman, A. (1974). Change in length of the muscle as a consequence of the shift in equilibrium in the muscle-load system. Biophysics, 19, 544–548.
A Cerebellar Model of Timing and Prediction
591
Fu, Q.-G., Mason, C. R., Flament, D., Coltz, J. D., & Ebner, T. J. (1997). Movement kinematics encoded in complex spike discharge of primate cerebellar Purkinje cells. NeuroReport, 8, 523–529. Gellman, R., Gibson, A. R., & Houk, J. C. (1983). Somatosensory properties of the inferior olive of the cat. Journal of Comparative Neurology, 215, 228–243. Gellman, R., Gibson, A. R., & Houk, J. C. (1985). Inferior olivary neurons in the awake cat: Detection of contact and passive body displacement. Journal of Neurophysiology, 54, 40–60. Georgopoulos, A. P., Kalaska, J. F., Caminiti, R., & Massey, J. T. (1983). Interruption of motor cortical discharge subserving aimed arm movements. Experimental Brain Research, 49, 327–340. Ghez, C. (1979). Contributions of central programs to rapid limb movement in the cat. In H. Asanuma and V. J. Wilson (Eds.), Integration in the nervous system (pp. 305–320). Tokyo: Igaku-Shoin. Ghez, C., Gordon, J., Ghilardi, M. F., Christakos, C. N., & Cooper, S. E. (1990). Roles of proprioceptive input in the programming of arm trajectories. In The Brain: Cold Spring Harbor Symp. Quant. Biol. (Vol. 55, pp. 837–847). Cold Spring Harbor, NY: Cold Springs Harbor Laboratory Press. Ghez, C., & Martin, J. H. (1982). The control of rapid limb movement in the cat. III. Agonist-antagonist coupling. Experimental Brain Research, 45, 115–125. Gielen, C. C. A. M., & Houk, J. C. (1984). Nonlinear viscosity of human wrist. Journal of Neurophysiology, 52, 553–569. Gielen, C. C. A. M., & Houk, J. C. (1987). A model of the motor servo: Incorporating nonlinear spindle receptor and muscle mechanical properties. Biological Cybernetics, 57, 217–231. Goodale, M. A., P´elisson, D., & Prablanc, C. (1986). Large adjustments in visually guided reaching do not depend on vision of the hand or perception of target displacement. Nature, 320, 748–750. Goodwin, G. C., & Sin, K. S. (1984). Adaptive filtering prediction and control. Englewood Cliffs, NJ: Prentice Hall. Gutman, A. M. (1991). Bistability of dendrites. International Journal of Neural Systems, 1, 291–304. Gutman, A. M. (1994). Gelfand-Tsetlin principle of minimal afferentation and bistability of dendrites. International Journal of Neural Systems, 5, 83–86. Hoffman, G. W. (1986). A neural network model based on the analogy with the immune system. Journal of Theoretical Biology, 122, 33–67. Horn, K. M., Van Kan, P. L. D., & Gibson, A. R. (1996). Reduction of rostral dorsal acessory olive responses during reaching. Journal of Neurophysiology, 76, 4140–4151. Houk, J. C. (1989). Cooperative control of limb movements by the motor cortex. In R. M. J. Cotterill (Ed.), Models of brain function (pp. 309–325). Cambridge: Cambridge University Press. Houk, J. C., & Alford, S. (1996). Computational significance of the cellular mechanisms for synaptic plasticity in Purkinje cells. Behavioral and Brain Sciences, 19(3), 457.
592
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
Houk, J. C., & Barto, A. G. (1992). Distributed sensorimotor learning. In G. E. Stelmach and J. Requin (Eds.), Tutorials in motor behavior II (pp. 71–100). Amsterdam: Elsevier Science Publishers. Houk, J. C., Buckingham, J. T., & Barto, A. G. (1996). Models of the cerebellum and motor learning. Behavioral and Brain Sciences, 19, 368–383. Houk, J. C., Galiana, H. L., & Guitton, D. (1992). Cooperative control of gaze by the superior colliculus, brainstem and cerebellum. In G. E. Stelmach and J. Requin (Eds.), Tutorials in motor behavior II (pp. 443–474). Amsterdam: Elsevier Science Publishers. Houk, J. C., Keifer, J., & Barto, A. G. (1993). Distributed motor commands in the limb premotor network. Trends in Neuroscience, 16, 27–33. Houk, J. C., & Rymer, W. Z. (1981). Neural control of muscle length and tension. In V. B. Brooks (Ed.), Handbook of physiology: Sec. 1: Vol. 2. Motor control (pp. 247–323). Bethesda, MD: American Physiological Society. Houk, J. C., Singh, S. P., Fisher, C., & Barto, A. G. (1990). An adaptive network inspired by the anatomy and physiology of the cerebellum. In T. Miller, R. S. Sutton, and P. J. Werbos (Eds.), Neural networks for control (pp. 301–348). Cambridge, MA: MIT Press. Houk, J. C., & Wise, S. P. (1995). Distributed modular architectures linking basal ganglia, cerebellum and cerebral cortex: Their role in planning and controlling action. Cerebral Cortex, 5, 95–110. Ito, M. (1984). The cerebellum and neural control. New York: Raven Press. Kawato,M., & Gomi, H. (1992). A computational model of four regions of the cerebellum based on feedback-error learning. Biological Cybernetics, 68, 95– 103. Kawato, M., & Gomi, H. (1993). Feedback-error-learning model of cerebellar motor control. In N. Mano, I. Hamada, and M. R. DeLong (Eds.), Role of the cerebellum and basal ganglia in voluntary movement (pp. 51–61). Amsterdam: Elsevier Science. Keeler, J. D. (1990). A dynamical system view of cerebellar function. Physica D, 42, 396–410. Keller, E. L., & Robinson, D. A. (1971). Absence of stretch reflex in extraocular muscles of the monkey. Journal of Neurophysiology, 34, 908–919. Kettner, R. E., Mahamud, S., Leung, H.-C., Barto, A. G., Houk, J. C., Peterson, B. W., & Sitkoff, N. (1997). Prediction of complex two-dimensional trajectories by the eye and by a cerebellar model of smooth eye movement. Journal of Neurophysiology, 77, 2115–2130. Kiehn, O. (1991). Plateau potentials and active integration in the “final common pathway” for motor behavior. Trends in Neuroscience, 14, 68–73. Kitazawa, S., Kimura, T., & Yin, P.-B. (1998). Cerebellar complex spikes encode both destinations and errors in arm movements. Nature, 392, 494–497. Klopf, A. H. (1972). Brain function and adaptive systems—A heterostatic theory (Tech. Rep. No. AFCRL-72-0164). Bedford, MA: Air Force Cambridge Research Laboratories. Klopf, A. H. (1982). The hedonistic neuron: A theory of memory, learning, and intelligence. Washington, DC: Hemisphere.
A Cerebellar Model of Timing and Prediction
593
Klopf, A. H. (1988). A neuronal model of classical conditioning. Psychobiology, 16, 85–125. Lev-Ram, V., Jiang, T., Wood, J., Lawrence, D. S., & Tsien, R. Y. (1997). Synergies and coincidence requirements between NO, cGMP, and Ca2+ in the induction of cerebellar long-term depression. Neuron, 18, 1025–1038. Llin´as, R., & Sugimori, M. (1980). Electrophysiological properties of in vitro Purkinje cell dendrites in mammalian cerebellar slices. Journal of Physiology (London), 305, 197–213. Marr, D. (1969). A theory of cerebellar cortex. Journal of Physiology (London), 202, 437–470. Miall, R. C., Weir, D. J., Wolpert, D. M., & Stein, J. F. (1993). Is the cerebellum a Smith predictor? Journal of Motor Behavior, 25, 203–216. Optican, L. M. (1995). A field theory of saccade generation: Temporal-to-spatial transform in the superior colliculus. Vision Research, 35, 3313–3320. Optican, L. M., & Robinson, D. A. (1980). Cerebellar-dependent adaptive control of primate saccadic system. Journal of Neurophysiology, 44, 1058–1076. Pettersson, L.-G., Lundberg, A., Alstermark, B., Isa, T., & Tantisira, B. (1997). Effect of spinal cord lesions on forelimb target-reaching and on visually guided switching of target-reaching in the cat. Neuroscience Research, 29, 241–256. Prablanc, C., & Martin, O. (1992). Automatic control during hand reaching at undetected two-dimensional target displacements. Journal of Neurophysiology, 67, 455–469. Raymond, J. L., Lisberger, S. G., & Mauk, M. D. (1996). The cerebellum: A neuronal learning machine? Science, 272(5265), 1126–1131. Robinson, D. A. (1975). Oculomotor control signals. In G. Lennerstrand & P. Bach-y-rita (Eds.), Basic mechanisms of ocular motility and their clinical implications (pp. 337–374). Oxford: Pergamon Press. Sakurai, M. (1987). Synaptic modification of parallel fibre–Purkinje cell transmission in in vitro guinea-pig cerebellar slices. Journal of Physiology (London), 394, 463–480. Schreurs, B. G., & Alkon, D. L. (1993). Rabbit cerebellar slice analysis of longterm depression and its role in classical conditioning. Brain Research, 631, 235–240. Schreurs, B. G., Oh, M. M., & Alkon, D. L. (1996). Pairing-specific long-term depression of Purkinje cell excitatory postsynaptic potentials results from a classical conditioning procedure in the rabbit cerebellar slice. Journal of Physiology, 75, 1051–1060. Schweighofer, N. (1995). Computational models of the cerebellum in the adaptive control of movements. Unpublished doctoral dissertation, University of Southern California, Los Angeles. Schweighofer, N., Arbib, M. A., & Dominey, P. F. (1996a). A model of the cerebellum in adaptive control of saccadic gain I. The model and its biological substrate. Biological Cybernetics, 75, 19–28. Schweighofer, N., Arbib, M. A., & Dominey, P. F. (1996b). A model of the cerebelllum in adaptive control of saccadic gain II. Simulation results. Biological Cybernetics, 75, 29–36.
594
A. G. Barto, A. H. Fagg, N. Sitkoff, & J. C. Houk
Simpson, J. I., Wylie, D. R., & de Zeeuw, C. I. (1996). On climbing fiber signals and their consequence(s). Behavioral and Brain Sciences, 19, 384–398. Sinkjaer, T., Wu, C. H., Barto, A. G., & Houk, J. C. (1990). Cerebellar control of endpoint position—A simulation model. In Proceedings of the 1990 International Joint Conference on Neural Networks (pp. II705–710). San Diego, CA. Sutton, R. S., & Barto, A. G. (1981). Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88, 135–170. Sutton, R. S., & Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel and J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks (pp. 497–537). Cambridge, MA: MIT Press. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press. Tyrrell, T., & Willshaw, D. J. (1992). Cerebellar cortex: Its simulation and the relevance of Marr’s theory. Proceedings of the Royal Society of London B, 336, 239–257. Van Kan, P. L. E., Gibson, A. R., & Houk, J. C. (1993a). Movement-related inputs to intermediate cerebellum of the monkey. Journal of Neurophysiology, 69, 74– 94. Van Kan, P. L. E., Houk, J. C., & Gibson, A. R. (1993b). Output organization of intermediate cerebellum of the monkey. Journal of Neurophysiology, 69, 57–73. Wang, L., & Ross, J. (1990). Synchronous neural networks of nonlinear threshold elements with hysteresis. Proceedings of the National Academy of Science USA, 87, 988–992. Wu, C. H., Houk, J. C., Young, K. Y., & Miller, L. E. (1990). Nonlinear damping of limb motion. In J. M. Winters and S. L.-Y. Woo (Eds.), Multiple muscle systems: Biomechanics and movement organization (pp. 214–235). New York: SpringerVerlag. Yuen, G. L., Hockberger, P. E., & Houk, J. C. (1995). Bistability in cerebellar Purkinje cell dendrites modelled with high-threshold calcium and delayedrectifier potassium channels. Biological Cybernetics, 73, 375–388. Received April 5, 1996; accepted July 2, 1998.
NOTE
Communicated by Geoffrey Goodhill
Improved Multidimensional Scaling Analysis Using Neural Networks with Distance-Error Backpropagation Llu´ıs Garrido Departament d’Estructura i Constituents de la Mat`eria/IFAE, Universitat de Barcelona, E-08028 Barcelona, Spain
Sergio Gomez ´ Departament d’Enginyeria Inform`atica, Universitat Rovira i Virgili, E-43006 Tarragona, Spain
Jaume Roca Departament d’Estructura i Constituents de la Mat`eria/IFAE, Universitat de Barcelona, E-08028 Barcelona, Spain
We show that neural networks, with a suitable error function for backpropagation, can be successfully used for metric multidimensional scaling (MDS) (i.e., dimensional reduction while trying to preserve the original distances between patterns) and are in fact able to outdo the standard algebraic approach to MDS, known as classical scaling. 1 Introduction A standard problem in multidimensional scaling analysis is to map a collection of patterns, represented as points in an n-dimensional space {xa ∈ Rn ; a = 1, . . . , p}, to a lower-dimensional space in such a way that the distances between the projected points resemble as closely as possible the distances between the original ones. More precisely, given the collection {xa }, with Euclidean distances between pairs (a, b) of patterns: d(n) ab =
p (xa − xb )2 ,
one has to find a map, ϕ : Rn → Rm , with m < n, such that it minimizes the quadratic distance-error function Eϕ =
´2 1 X ³ (n) , dab − d(m) ab 2 a,b
Neural Computation 11, 595–600 (1999)
c 1999 Massachusetts Institute of Technology °
596
Llu´ıs Garrido, Sergio Gomez, ´ & Jaume Roca
where d(m) ab are the Euclidean distances computed in the projected space d(m) ab =
q (ϕ(xa ) − ϕ(xb ))2 .
Typically, m is chosen to be two or three in order to make available a graphical representation of the projected configuration. This can help visualize an underlying structure that might be obscured by cluttered data in the original space. It is not known in general how to find the exact expression of the best map ϕ. Yet there is a standard method to approximate it, known as classical scaling (CLS), which involves the diagonalization of the symmetric matrix S of scalar products Sab = xa · xb , by means P of an orthogonal matrix C. Taking {xa } to be centered at the origin, that is, a xa = 0, and assuming that p > n, it is easy to show that S can have at most n nonzero eigenvalues. Each of these eigenvalues can be regarded as the typical scale of a principal direction. If we denote by 31 , . . . , 3m the m largest eigenvalues, the resultant mapping to Rm is given by α (xa ) = 31/2 ϕCLS α Caα
α = 1, . . . , m.
(See Cox & Cox, 1994, for a detailed description of this method.) CLS can be used in a broader context, when only a matrix of dissimilarities δab is known, as a tool to assign coordinates to the patterns. Once coordinates are already known for patterns, as in our case, CLS reduces to principal component analysis (PCA). 2 Multidimensional Scaling with Neural Networks In this article, we provide an alternative solution to this problem, which involves the use of neural networks. The main idea consists of building a net with n input units and a number of hidden layers, containing a bottleneck layer with only m units and an output layer with n units. A modified version of the standard backpropagation algorithm is then invoked (Rumelhart, Hinton, & Williams, 1986). In addition to the quadratic error term between input and output, it contains a new term that is introduced to minimize the difference between the distances of pairs in the input and neck layers. When enough iterations have been performed, the projected configuration is read out from the neck layer. In order to use the net in the most efficient way, it is convenient to perform a translation and a global scaling of the initial data, in xa −→ ξ in a = λ (xa − a),
Improved Multidimensional Scaling Analysis
597
in n so as to make ξ in a ∈ [0, 1] . Then one can use ξ a as the input to the net. nk The outcome of the neck layer, ξ a , lives in the region [0, 1]m since √ we are ≤ m while using sigmoid activation functions. This implies that 0 ≤ dnk ab √ in in nk and din stand ≤ n for any pair of input points ( ξ , ξ ), where d 0 ≤ din a b ab ab ab for the distances between patterns a and b in the neck and initial layers, respectively. The error function that we have considered in the backpropagation method is given by
E = α E1 + (1 − α) E2 , where E1 =
X³
ξ out a
−
ξ in a
´2
and
E2 =
a
X a,b
Ã
dnk din √ab − √ab m n
!2 ,
and α ∈ [0, 1] controls the relative contribution of each part. The term E1 favors those maps for which the representation in the bottleneck layer can be best accurately inverted to recover the original configuration. The second term, E2 , is the most important one since it forces this representation in the bottleneck to inherit, as closely as possible, the metric structure of the nk original configuration. The different scalings for din ab and dab in this term are introduced in order to have both numbers in the same range. In this way, we can guarantee that all possible configurations can still be covered with the use of sigmoids. The various scalings involved in this process make the outcome of the neck layer not to be directly interpretable as the final answer; we can bring it back to the original scale by setting ϕNN (xa ) = λout ξ nk a , with λout = instead
√ n/m λin . A slightly better solution can be obtained by choosing P out
λ
(n) nk a,b dab dab ¡ nk ¢2 a,b dab
= P
,
P since this is the value of λ that minimizes the function E(λ) = 12 a,b (d(n) ab − nk 2 λdab ) for the given neck configuration, which is what we are ultimately trying to achieve with the procedure. In the practical use of the neural network, we have noticed that the best results are obtained by letting the parameter α fall to zero as the learning grows so that the error function E reduces to E2 after a certain number of iterations. Actually, a nonzero value of α is useful only in the early stages of the learning, in order to speed up convergence. In this situation, with E = E2 ,
598
Llu´ıs Garrido, Sergio Gomez, ´ & Jaume Roca
it is easy to prove analytically that P the configuration minimizing E differs √ from the one minimizing directly (din−dnk )2 only by a global scaling n/m of all coordinates. Thus, the (otherwise technically convenient) scalings that we have introduced are completely harmless for the purpose of searching for the best mapped configuration. It is commonly known that a network with just the input, output, and neck layers, with linear activation functions and subject to self-supervised backpropagation, is equivalent to PCA (Sanger, 1989). Our approach goes beyond PCA, not only because of the use of sigmoid (nonlinear) activation functions and the addition of a number of hidden layers, but essentially for the presence of this new distance-term contribution, E2 , which favors those configurations in the neck layer that approximate the original distances better. One may wonder how our method compares to nonlinear PCA (NLPCA) (Kramer, 1991; DeMers & Cottrell, 1994; Kambhatla & Leen, 1995; Garrido, Gait´an, Serra-Ricart, & Calbet, 1995; Garrido, Gomez, ´ Gait´an, & Serra-Ricart, 1996). Actually, NLPCA can be recovered as a particular case of our approach by setting α = 1 in the error function (i.e., with E = E1 ). NLPCA will generally do better than ordinary PCA in the minimization of the term E1 because of the ability to model nonlinear configurations. However, NLPCA does not care at all about the distances between patterns in the bottleneck representation: any two neck configuration are equally good for NLPCA if both provide the same result in the output layer. Hence, the comparison of NLPCA with our approach is inappropriate because both methods are in fact designed for different purposes (minimizing E1 and E2 , respectively). On the contrary, the projected configuration of standard PCA still retains part of the metric structure of the initial configuration since it is just a linear orthogonal projection onto the largest-variance axes, and hence it produces better results for E2 than NLPCA. This is why we will compare the performance of our method with CLS (i.e., PCA) and not with NLPCA. A comparative analysis of both approaches over several types of configurations shows that our method produces better results in the tougher situations, when some of the discarded directions in the CLS method still have relatively large associated eigenvalues. Finally, it is worth stressing that CLS provides only a linear orthogonal projection, whereas the neural net is able to produce more general (nonlinear) mappings. Example. As an illustration of both procedures, we have considered a data set1 consisting of different animal species, characterized by n = 17 attributes each (15 boolean + 2 numerical). The coordinates xa and distances d(n) ab have been obtained after scaling the numerical attributes to the range 1 Extracted from the Zoo Database created by Richard S. Forsyth (1990) (ftp://ftp.ics.uci.edu: ˜ /pub/machine-learning-databases/zoo).
Improved Multidimensional Scaling Analysis
599
−2.5
2.5
CLS
1.5
76 75
69 70 65 68 51 46 45 48 511 30 41
49
55 50
56 29 18 6223 33
NN
71 66 32 7
30 69
75
28
76
64
0.5
56 29 23 18 62 10
49
36
85 37
55 50
67 20
67 20
66 71 32 7 33
65 68 51 46 45 48 5 11 41 70
−1.5
36 10
8537
28
−0.5 64
−0.5
61 19 87 83 74 35
62 39 313 9
61 19 87 77
81 63
77
8
53 26 27
90
59
42 88
−1.5
79 80 34
54
42
83 74 35
53 26
52 4140 58 21 12 84 60 44
38 17 89 25 82 22 43
62 39 3 913
1.5
31
−2.5 −2.5
−1.5
54
47 16 86 15
−0.5
58 21 12 84 60 44 38 17 22
52 41 40
73
14 78
80 79 34
90
27
24
88
59
8
24
73
57
72
81 63
0.5
57
72
14
82
8943 25
31
86 78 15 47 16
0.5
1.5
2.5
2.5 −2.5
−1.5
−0.5
0.5
1.5
2.5
Figure 1: Two-dimensional mapped configurations obtained with classical scaling (CLS) and with a neural network (NN).
[0, 1] in order to assign an equal weight to all attributes (implying in this case that we simply have ξ in a = xa ). The best scaling for the two-dimensional neck representation when using the neural net is given by λout = 2.946, which is in less than 1.1% disagree√ ment with the expected value of λout = 17/2. The projected configurations obtained with each method are drawn in Figure 1. Patterns are represented by their label. As the plot shows, both approaches produce a fairly similar configuration. However, the computation of the overall relative error, P
ε=
³
(n) (m) a,b dab − dab P ³ (n) ´2 a,b dab
´2 12 ,
shows for each method that the neural network gives a slightly better result,
ε
CLS
= 0.2728,
ε
NN
= 0.2346,
which amounts to a 14.00% improvement over the CLS method. Acknowledgments This work was supported in part by CICYT contract AEN95-0590 and a URV project, URV96-GNI-13. J.R. also thanks the Ministerio de Educacion ´ y Cultura of Spain for financial support.
600
Llu´ıs Garrido, Sergio Gomez, ´ & Jaume Roca
References Cox, T. F., & Cox, M. A. A. (1994). Multidimensional scaling. London: Chapman & Hall. DeMers, D., & Cottrell, G. (1994). Non-linear dimensionality reduction. In C. L. Giles, S. J. Hanson, & J. D. Cowan (Eds.), Advances in neural information processing systems, 5. San Mateo, CA: Morgan Kauffman. Forsyth, R. S. (1990). Zoo Database. Available online at ftp: //ftp.ics.uci.edu/ pub/machine-learning-databases/200. Garrido, Ll., Gait´an, V., Serra-Ricart, M., & Calbet, X. (1995). Use of multilayer feedforward neural nets as a display method for multidimensional distributions. Int. J. Neural Systems, 6, 273. Garrido, Ll., Gomez, ´ S., Gait´an, V., & Serra-Ricart, M. (1996). A regularization term to avoid the saturation of the sigmoids in multilayer neural networks. Int. J. Neural Systems, 7, 257. Kambhatla, N., & Leen, T. K. (1995). Fast non-linear dimension reduction. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6. San Mateo, CA: Morgan Kauffman. Kramer, M. A. (1991). Non-linear principal component analysis using autoassociative neural networks. AICHE Journal, 37, 233. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by backpropagating errors. Nature, 323, 533. Sanger, T. D. (1989). Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks, 2, 459. Received October 29, 1997; accepted July 10, 1998.
LETTER
Communicated by Dan Ruderman
Firing Rate Distributions and Efficiency of Information Transmission of Inferior Temporal Cortex Neurons to Natural Visual Stimuli Alessandro Treves SISSA, Programme in Neuroscience, 34013 Trieste, Italy
Stefano Panzeri Edmund T. Rolls Michael Booth Edward A. Wakeman University of Oxford, Department of Experimental Psychology, Oxford OX1 3UD, United Kingdom
The distribution of responses of sensory neurons to ecological stimulation has been proposed to be designed to maximize information transmission, which according to a simple model would imply an exponential distribution of spike counts in a given time window. We have used recordings from inferior temporal cortex neurons responding to quasi-natural visual stimulation (presented using a video of everyday lab scenes and a large number of static images of faces and natural scenes) to assess the validity of this exponential model and to develop an alternative simple model of spike count distributions. We find that the exponential model has to be rejected in 84% of cases (at the p < 0.01 level). A new model, which accounts for the firing rate distribution found in terms of slow and fast variability in the inputs that produce neuronal activation, is rejected statistically in only 16% of cases. Finally, we show that the neurons are moderately efficient at transmitting information but not optimally efficient. 1 Introduction The firing rates of single neurons from the inferior temporal visual cortex, in response to a large, natural set of stimuli, typically have a distribution that is graded (continuous from zero response to the maximum), unimodal (a single peak often close to the spontaneous firing rate, or to zero), and an approximately exponential tail. Such firing rate distributions, usually expressed in terms of the number of spikes in a window of fixed length, are a common observation in many parts of the brain, including the frontal cortex (Abeles, Vaadia, & Bergman, 1990) and the hippocampus and related structures (Barnes, McNaughton, Mizumori, Leonard, & Lin, 1990). Indeed, Neural Computation 11, 601–631 (1999)
c 1999 Massachusetts Institute of Technology °
602
A. Treves, S. Panzeri, E. T. Rolls, M. Booth, & E. A. Wakeman
exponential distributions have been used in formal neural network analyses as a first, simple model of realistic distributions of graded firing rates (Treves & Rolls, 1991). Part of the special interest of this observation in the case of sensory (e.g., visual) cortices is in the fact that an exponential distribution of spike counts would maximize their entropy (and thus the information a cell would transmit, for a noiseless discrete code), under the constraint of a fixed average count (Shannon, 1948).1 It has therefore been suggested (Levy & Baxter, 1996; Baddeley, 1996; Baddeley et al., 1997) that neurons might tend to use an exponential distribution of firing rates because it would be the most efficient in metabolic terms; that is, it would be the most informative once a given metabolic “budget” has been set in terms of the average rate. It remains dubious, though, whether the distribution that maximizes information transmission in the absence of noise would be of any particular value in the presence of noise. Moreover, other distributions would be optimal under somewhat different assumptions. For example, a very different distribution, a binary distribution, would instead maximize the instantaneous rate at which information about slowly varying stimuli is transmitted by a noisy spiking code, of the type used by neurons in the brain (Panzeri, Biella, Rolls, Skaggs, & Treves, 1996a). Other approaches to understanding the observed spike count distributions are not based on the notion that such distributions would reflect the optimization of information transmission, but rather on the idea that they would simply result from the intrinsic variability of the underlying process. Spikes are produced in a neuron when a sufficient amount of current enters the soma. (The current rather than the voltage is the relevant activation variable when considering the emission of several spikes; Treves & Rolls, 1991; Koch, Bernander, & Douglas, 1995.) The current is the sum of many synaptic inputs, and if these are weakly correlated, over a large set of natural stimuli, the distribution of the current can be expected by the central limit theorem to be approximately normal, that is, gaussian. Since going from current to spike count involves what is essentially a rectification (a threshold-linear transform), a possible model for the spike count distribution would be a truncated gaussian, with an additional peak at zero. The truncated gaussian model considers only the mean current over a time window and a fixed, deterministic, conversion of this value into a spike count; thus, it neglects sources of rapid variability. A simple and widely used model that instead emphasizes fast variability is that of Poisson firing. In the Poisson model, spike emission is a stochastic event, with a probability dependent on the instantaneous value of the activation. If this source of fast variability dom-
1 What is discussed here is the entropy of the spike count distribution. The entropy of the spike train, that is, of the collection of spike emission times, is maximized by Poisson spiking processes.
Firing Rate Distributions and Information Transmission
603
inates over the slower fluctuations of the mean current, the shape of the resulting spike count distribution is close to a simple Poisson distribution. We use face-selective cells (Rolls, 1984) recorded from the macaque inferior temporal visual cortex to show that none of these simple models, though attractive in their simplicity, satisfactorily captures the statistics of firing in response to large sets of natural stimuli. On the other hand, we find that a model, the S+F random model, which includes both slow and fast variability in the activation as generators of the spike count distribution and takes them both to be roughly normally distributed, comes much closer to an accurate description of the observed statistics. We discuss why the agreement with this still rather simple model is not, and should not be expected to be, perfect. Finally, we measure the efficiency with which the observed distribution transmits information. Although the model we propose as an explanation has nothing to do with optimizing efficiency, we do find that the efficiency is moderately high. Some of the results have been previously published in abstract form (Panzeri, Booth, Wakeman, Rolls, & Treves, 1996b.) 2 Methods 2.1 Collection of Firing Rate Distributions. We constructed firing rate probability distributions of 15 visual neurons shown to have face-selective responses recorded in the cortex in the anterior part of the superior temporal sulcus of two rhesus macaques. All the cells had responses that were face selective in that they responded at least twice as much to the most effective face stimulus as to the most effective nonface stimulus in initial screening (for criteria, see Rolls & Tov´ee, 1995). Spike arrival times were recorded with a Datawave spike acquisition system. During the recording session, the monkey was looking at a 5-minute video showing natural scenes familiar to the monkey. The video included views of laboratory rooms and equipment, as well as people and monkeys typically seen daily by these animals, and was continuously recorded with no breaks or cuts. For approximately 30% of the video time, at least one face was included in the view. During the video, the monkey was not required to maintain any static visual fixation, in an attempt to emulate natural viewing behavior. As the video was shown more than once when recording some of the cells, a total of 22 data sets (from monkey ay 20 data sets, and from monkey ba 2 data sets) were available for the analysis. We emphasize that the behavior of the monkey during consecutive recording sessions from the same cell was not constrained to be the same, and in particular eye movements during the video could vary from one presentation to the next. The neurophysiological methods have been described previously (Rolls & Tov´ee, 1995). Histograms of single-cell response probability distributions for the video data were created by dividing the recording time into nonoverlapping time bins of a given length and counting the number of spikes within each bin.
604
A. Treves, S. Panzeri, E. T. Rolls, M. Booth, & E. A. Wakeman
The error bars for each histogram are drawn according to the following standard procedure. The fluctuations in the observed frequency in each bin can be thought of as coming from the stochastic process of making random selections among a finite numbers of items, therefore following a distribution close to Poisson. The standard deviation of the observed frequency for each particular spike count is thus estimated as the square root of the observed frequency for that count, independent of the underlying distribution of the count frequencies across different spike counts. We used time windows of length L = 50, 100, 200, 400, and 800 ms. As a further comparison, we analyzed the responses of a second set of 14 face-selective inferior temporal cortex (IT) neurons recorded from another monkey (am), when a set of 65 static visual stimuli (natural images; 23 faces and 42 nonface images) were repeatedly presented to the animal during a visual fixation task. This large set of different stimuli was used to provide evidence on how the neurons would distribute their responses to a wide range of stimuli similar to those that they might see naturally. The stimulus was presented for 500 ms on each trial. This latter set of data has been previously studied using information theoretical techniques (Rolls & Tov´ee, 1995; Rolls, Treves, Tov´ee, & Panzeri, 1997b). In this article, we analyze these data and find that with such static images, the spike count distribution and other measures are very similar to those obtained with the video data. The fact that similar results on the distribution of spike counts were obtained with two different cell samples and with very different ways of presenting the stimuli (continuous video versus a large set of static images) corroborates the findings and analyses described here. Histograms of single-cell firing rate probability distributions for the static stimuli data were created by counting the number of spikes emitted by the cell in a time window L ms long, starting 100 ms after stimulus onset, in response to each of the 65 static stimuli in the sample, for each stimulus presentation. The firing rate probability histograms were calculated on the basis of 380 to 600 trials for each cell. Error bars for the histograms were calculated as for the video data. We used time windows of length L = 50, 100, 200, and 400 ms. 2.2 Models of the Firing Rate Distribution of Single Cells. 2.2.1 Exponential Distribution. This is the one-parameter distribution for the spike count n, P(n) =
1 exp (−λn) , 1 + (¯rL)
where λ is determined by the the single parameter r¯ as follows: ¶ µ 1 . λ = ln 1 + r¯L
(2.1)
(2.2)
Firing Rate Distributions and Information Transmission
605
The parameter r¯ is calculated from the data as the total spike count divided by the total duration of the video. Notice that the exponential distribution, equation 2.1, maximizes the entropy of the spike count under the constraint of fixed average firing rate (Rieke, Warland, de Ruyter van Steveninck, & Bialek, 1996). The significance of the agreement between the observed distribution for each data set and this model distribution is evaluated from the χ 2 statistics. χ 2 (L) is measured for each time window as χ2 =
X [N|Pobs (n) − Pmodel (n)| − 1/2]2 n
NPmodel (n)
,
(2.3)
where subtracting 1/2 is a procedure known as the Yates correction (or correction for continuity; Fisher & Yates, 1963). This correction properly takes into account the fact that NPobs (n), unlike NPmodel (n), can take only integer values, and is useful to ensure that histogram bins with very few expected events NPmodel (n) do not weigh disproportionately on χ 2 . In addition, bins with zero observed events, such as those at the large-n tail of the distribution, are all grouped together with the nearest preceding bin with at least one event. This is again a standard procedure to enable the use of the simple form of χ 2 distribution. From χ 2 (L) we derive, using the relevant probability distribution of χ 2 , the likelihood P(L) of the fit for time window L. Since the only free parameter of the probability distribution, r¯, is determined not by minimizing χ 2 but directly from the data, and it is common to all five window lengths L, we use as the number of degrees of freedom of the fit df = nbins − 1 − (1/5), with nbins the final number of histogram bins. 2.2.2 Binary Distribution. A binary distribution of firing rates is clearly not what is found,2 even in other experiments in which static stimuli are used, so there is no point trying to fit it to the observed spike count. 2.2.3 Poisson Distribution. P(n) = exp(−¯rL)
(¯rL)n , n!
This is expressed as (2.4)
and again the only parameter is r¯. Since r¯ is just the average of the rate ri = ni /L, which is different in each time bin i, the distribution of ni across bins will remain approximately Poisson only to the extent that the variability in ri is minor compared to the fast variability (“within-bin” variability) described by the model. With respect to fitting it to the observed distribution, the same considerations and procedures apply as with the exponential model. 2 Bursting cells might intuitively be taken to behave as binary units, but in any case we are not aware of any similar experiment reporting binary spike counts, with a cell emitting, say, either 0 or 3 spikes in a given window.
606
A. Treves, S. Panzeri, E. T. Rolls, M. Booth, & E. A. Wakeman
2.2.4 Truncated Gaussian Distribution. This model assumes that the spike count reflects an underlying activation variable, the current h(t) flowing into the cell body, which is approximately normally distributed around a mean ¯ The fluctuations in h, of width σS (L), are taken to occur on time scales slow h. with respect to the window length L, so that h(t) ' hi at any time t between iL and (i + 1)L. The current is taken to translate into a firing frequency through simple rectification, that is, the firing rate in a time bin, ri , is zero if the activation hi is below a threshold, T, and is linear above the threshold: r=0 r = g(h − T)
if if
h
T.
Such an input-output function is a simple but often reasonable model of current-to-frequency transduction in pyramidal cells (Lanthorn, Storm, & Andersen, 1984), and at the same time it lends itself easily to analytical treatment (Treves & Rolls, 1991, 1992). We note also that integrate-and-fire models of neuronal firing operate effectively with an activation function that is close to threshold linear (Amit & Tsodyks, 1991; Treves, 1993). Note that saturation effects in the firing rate are not represented (they could be included, if they turn out to be important, at the price of analytical simplicity, but are shown not to be important in section 4). The parameter g is the gain of the threshold-linear transform, and here it is fixed, g = 1, so that the activation h is measured, like r, in Hz. The probability distribution for r is then continuous and identical to the one for h above threshold, 1 [ri − (h¯ − T)]2 exp − dri p(ri )dri = √ 2σS2 2πσS
(2.5)
¯ S ], P(ri = 0) = φ[(T − h)/σ
(2.6)
and
where we have defined Z φ(x) ≡
x −∞
µ 2¶ x dx . √ exp − 2 2π
(2.7)
The spike count distribution is a discrete histogram, whereas this model has been derived, above threshold, in the form of a continuous distribution of real-valued frequencies. To convert the continuous probability distribution p(r) of the model into a histogram P(n), we make the rough assumption that if the mean firing rate across the bin, ri , exactly equals the integer ni /L, the number of spikes observed will be ni , whereas if it is slightly above or
Firing Rate Distributions and Information Transmission
607
below it, the number of spikes, depending on when the first spike occurs, may also be observed to be, respectively, ni + 1 or ni − 1. Specifically, Z P(n) =
n/L (n−1)/L
Z dr[Lr − n + 1]p(r) +
(n+1)/L
dr[n − Lr + 1]p(r).
(2.8)
n/L
p(r) is then discretized, but in much finer size than 1/L, to evaluate P(n) numerically. To fit this model to the observed distributions, we now have to determine two parameters, σS and h0 ≡ h¯ − T. These are not measured directly from the data, but obtained by minimizing the discrepancy in the fit. 2.2.5 Adding Fast Fluctuations: The S+F Model. The new model we consider can be seen as an extension of the one generating the truncated gaussian distribution, in the sense of also taking into account fast fluctuations in the activation variable (conceptually similar to the variability in the response to a given stimulus), assumed to be roughly normally distributed, as the slow fluctuations are (which conceptually may be roughly compared with the different responses produced to each stimulus, which can be thought of as changing in the video and in the world with a time scale usually—but not always—longer than the time window considered). “Fast” here means fluctuations occurring over time scales shorter than the window L used in counting spikes; it does not imply any definite assumption as to what is stimulus, for any given cell, and what is noise. In practice, it leads to the distribution for the instantaneous current, (h(t) − hi )2 1 exp − . p(h(t)) = √ 2σF2 2πσF
(2.9)
The standard deviation σF , like that of the slow fluctuations denoted as σS , is measured in Hz and will be a function of the window length L, as a short window will make moderately fast fluctuations appear slow, and vice versa. The model takes slow and fast fluctuations to be uncorrelated, and we expect their total squared amplitude to be roughly constant across time windows, σS2 (L) + σF2 (L) ' constant (see sections 3 and 4). It is again assumed that the current produces deterministically a firing frequency, through a simple threshold-linear activation function. Now, however, the input-output transform is taken to hold at any moment in time, as if an instantaneous firing frequency r(t) could be defined, instead of the discrete events represented by individual spikes: r(t) = 0 r(t) = g(h(t) − T)
h(t) < T
if if
h(t) > T.
In the model, the firing rate distribution is then obtained by averaging over the fast noise. The mean response ri in a given time window is calcu-
608
A. Treves, S. Panzeri, E. T. Rolls, M. Booth, & E. A. Wakeman instantaneous and effective activation functions 60
rate r (Hz)
50 40 30 20 10 0 -60
-40
-20 0 20 activation h (Hz)
40
60
Figure 1: Activation functions in the model. The solid line is the instantaneous threshold linear activation function (with g = 1). The dashed line is an example of an “effective” activation function obtained integrating over fast noise, using equation 2.10, with σF = 50 Hz. The dotted and dashed line is another example of effective activation function, equation 2.10, obtained with σF = 10 Hz.
lated by averaging r(t) over fast fluctuations, and expressing the result as a function of the mean activation hi during the window: " µ ¶# (hi − T)2 hi − T hi − T 1 + φ . (2.10) ri = σF √ exp − σF σF 2σF2 2π From this equation, and taking into account the normal distribution of slow fluctuations, we obtain the probability distributions of firing rates: ¯ 2 · µ hi (ri ) − T ¶¸−1 1 (hi (ri ) − h) exp − , φ p(ri ) = √ σF 2σS2 2πσS
(2.11)
where hi (ri ) is determined by inverting equation 2.10. The averaging has the effect of smoothing over the fluctuations that are faster than the time window, and as a result the effective input-output transform, equation 2.10, is as illustrated in Figure 1. Again, the discrete spike count distribution is derived from the continuous firing rate distribution using the procedure described above for the truncated gaussian model (see equation 2.8). The fit of this model to the observed distributions involves three free parameters, h0 (again one can see that it is only the difference h¯ − T that counts), σF , and σS , which, as for the truncated gaussian model, must be obtained by minimization.
Firing Rate Distributions and Information Transmission
609
Table 1: Percentage of Rejections (p < 0.01) of the Fits to the Various Models. Probability Model
Rejections (p < 0.01) (video data)
Rejections (p < 0.01) (static stimuli)
Exponential Poisson Gaussian S+F model
83.6% (92/110) 100% (110/110) 96.3% (106/110) 15.4% (17/110)
75% (42/56) 100% (56/56) 89.2% (50/56) 1.8% (1/56)
2.3 Procedures and Parameters for the Fit. We fit the models to the five observed distributions of spike counts in response to the video, simultaneously across the five time windows, for each data set. For the exponential and Poisson models, this is straightforward, since the only free parameter can be read off the data directly. For the last two models, there are more free parameters, and they must be adjusted so as to optimize the fit. One parameter that we did not allow to vary for different time windows is the difference between the mean input and the threshold, that is, h0 . σS (L) and σF (L) (for the truncated gaussian model there is only σS (L)) are instead left free to vary independently for each time window. Thus, we have 1 + 5 free parameters for the truncated gaussian model and 1 + 5 × 2 for the S+F model, including fast variability. These are chosen by maximizing a pseudo–maximum likelihood cost function, which we construct as C=−
X
log(P(L))
(2.12)
L
from the probabilities P(L) that, at each time scale, the observed distribution could indeed be generated from the model. P(L) is derived from the χ 2 (L) value calculated as for the exponential model, now using as the number of degrees of freedom df = (nbins ) − 1 − 1 − (1/5) for the truncated gaussian model and df = (nbins ) − 1 − 2 − (1/5) for the S+F model, where again we take into account with the (1/5) that h0 is in common to five time windows. Note that the same data are used with all windows L, so the five fits are not independent, and a single fit with df = 11 would not yield a meaningful P level. When optimizing the fit of the S+F and truncated gaussian models to the data, we also tried allowing h0 to take different values for the different time windows. As expected, the fits are better than those shown in Figures 2 and 3 and Table 1. However, we prefer to rely on the condition in which the parameter h0 is the same for a given data set for the different time windows, because h0 is meant to correspond to the same physical variable across window lengths (i.e., the mean input current over the recording session). The maximization is performed using standard routines from Press, Teukolsky, Vetterling, and Flannery (1992). In particular, within a loop over
610
A. Treves, S. Panzeri, E. T. Rolls, M. Booth, & E. A. Wakeman 0.5 0.5
0.3
0.4
0.4
0.2
0.3 0.3 0.2
0.2
0.1
0.1
0.1 0
0 0
4
8
12
0 0
2
ay080-05
6
8 10 12
0
2
4
6
8
10
ay102-02
0.6
0.3
Probability
4
ay087-01
0.3 0.5 0.4
0.2
0.2
0.3 0.2
0.1
0.1
0.1 0
0 0
4
8
12
16
0 0
ay144-02
4
8
12
0
5
ay156-02
0.4
0.25
0.3
0.2
10
15
20
ay158-02 0.3
0.2
0.15 0.2 0.1 0.1
0.1
0.05
0 0
2
4
6
8 10 12
ay180-02
0
0 0
5
10
15
20
ba001-01
0
4
8
12
16
ba003-03
Number of Spikes Figure 2: Probability histograms for nine data sets (video data). The time window is 100 ms long. Each graph plots the probability on the y-axis of the number of spikes, in a 100 ms time bin, given in the x-axis. The histogram is the neurophysiological data from the neuron, with the standard deviation shown; the dashed line shows an exponential distribution fitted to the mean firing rate of the cell; and the solid line shows the fit of the S+F model described in the text.
time scales, for each L either σS (L) alone or σS (L) and σF (L) are first optimized for any given value of h0 by routine amoeba, and then the common h0 value is optimized by (unidimensional maximization) routines golden and mnbrak. At the end, the individual levels of significance P(L) are extracted. Note that for visual inspection, the observed histograms are plotted in their original form, without any grouping.
Firing Rate Distributions and Information Transmission
0.1
0.16
0.16 0.12
0.12
0.08
0.08
0.04
0.04
611
0.08 0.06 0.04 0.02 0
0
10
20
30
0
5 10 15 20 25
ay080-05
0
10
ay087-01
20
30
ay102-02
Probability
0.1 0.08
0.08
0.2
0.06
0.15
0.04
0.1
0.04
0.02
0.05
0.02
0.06
0
0 0
15
30
0
45
ay144-02
5
10 15 20 25
0
ay156-02
10
20
30
40
ay158-02
0.12
0.08
0.06
0.06
0.08
0.04 0.04
0.04
0.02
0 0
10
20
ay180-02
30
0.02 0
0 0
20
40
60
ba001-01
0
10
20
30
ba003-03
Number of Spikes Figure 3: Probability histograms for nine data sets (video data). The time window is 400 ms long. The conventions are as in Figure 2.
Exactly the same analysis is applied to the probability distributions of the responses to static stimuli, the only difference being that in this case, the single time windows are fitted separately. This is because the time window is locked to the stimulus onset, and therefore the mean activation h0 was not expected in this case to be constant across different time windows. 2.3.1 Error Estimates of the Parameters. This can be done, for the S+F model, by calculating the region in parameter space within which C increases by no more than a set amount. We do not report error bars on individual parameters because they would misleadingly indicate that each
612
A. Treves, S. Panzeri, E. T. Rolls, M. Booth, & E. A. Wakeman
parameter may be undetermined to that extent. In fact, it turns out that both σS (L) and σF (L) largely covary with h0 . h0 itself has large error bars when it is large negative, whereas it is much better determined if the distribution has a clear peak above zero, which then roughly corresponds to h0 . 2.4 Power Spectra of the Firing Rate Distributions. Power spectra of the firing rates are computed using standard routines from the train of spikes recorded during each video presentation. We calculated, for every cell, the power spectra using a sampling frequency of either 100, 500, or 1000 Hz, dividing the data into nonoverlapping segments each 256 data points long and windowing with a Bartlett window (Press et al., 1992). The nonoverlapping segments used for the spectral analysis covered all 5 minutes of the video presentation. We also produced a normalized and averaged power spectrum across the data sets, first normalizing the power spectra of individual cells (to give a total power of 1) and then averaging over the data set. 2.5 Efficiency Measure. For each data set and each window L, we extract from the observed spike count distribution the so-called information per spike,3 χ=
X n
P(n)
n n log2 , r¯L r¯L
(2.13)
a quantity introduced by Rieke, de Ruyter van Steveninck, and Warland (1991) and Skaggs, McNaughton, Gothard, and Markus (1993), and which we argue below to be relevant. As shown by Panzeri et al. (1996a), this quantity can range between zero and a maximum, 0 ≤ χ ≤ log2 (1/a),
(2.14)
where a is the sparseness (Treves & Rolls, 1991) of the distribution, defined as P ( i ri /N)2 1 P = P (2.15) a= 2 /N (r ) P(n)(n/¯ rL)2 i i n where (N is the number of events, that is time bins). Note that the information per spike, χ, reaches its maximal value of log2 (1/a) for a binary distribution. The quantity χ is, in general, the time derivative of the information (about any correlated variable; here, the time bin of each count) conveyed by the spike count, and divided by the mean firing rate. For a Poisson process with independent spikes, χ acquires the additional and intuitive meaning of average information conveyed by each spike. We use the notation χ for consistency with previous work (Panzeri et al., 1996a), even at the risk of confusion with the—unrelated—quantity χ 2 appearing in this article. 3
Firing Rate Distributions and Information Transmission
613
We then define our efficiency measure, at each scale L, as %(L) ≡
χ , log2 (1/a)
(2.16)
so that % varies between 0 and 1. Further aspects of this measure are considered in sections 3 and 4. A related efficiency measure, used by Bialek et al. (1991), compares the instantaneous rate of information transmission to the entropy rate. Its expression in the short time limit is, %B (L) ≡
χ , ¯ log2 (e/n)
(2.17)
and can be applied with reasonable accuracy only to very short time windows, so that the mean number of spikes in the window n¯ ¿ 1. %B also varies between 0 and 1. For very short windows, as the distribution of spike counts in individual bins becomes binary, 0 or 1, χ → log2 (1/a) and % → 1. %B also tends to 1 but much more slowly (only for very short windows), ¯ → log2 (1/a) + log2 (e) because log2 (e/n) 3 Results 3.1 Spike Count Distributions and How They Fit Simple Models. To emulate naturally occurring firing activity, we recorded from face-selective cells in the inferior temporal cortex of monkeys who were viewing a video of everyday lab scenes and whose eye movements, level of attention, and motivation were not constrained. The firing rate distributions of the 22 data sets collected in these conditions show the general trend discussed in Section 1: they appear graded, unimodal (with the single peak close to either zero or the spontaneous activity), and with an exponential-like tail. (These data sets form the majority of those also analyzed by Baddeley et al., 1997. They also included four data sets from nonface-selective neurons, which we exclude here because the effective stimuli for these data sets were not known.) We now present the single-cell quantitative analysis for the video data, showing that the S+F model accounts for most of the data, while the exponential model does not describe the data satisfactorily, especially at low rates, and the truncated gaussian and Poisson models do not fit at all. Table 1 summarizes how well the four models considered fit the observed distributions of spike counts. Considering 22 data sets (from 15 different face cells, some of which were recorded over multiple presentations of the video) and five window lengths, there are 110 possible fits. Setting a confidence level of p = 0.01, we would expect that if a true underlying distribution existed and were known to us, it would fit the data at this confidence level always except
614
A. Treves, S. Panzeri, E. T. Rolls, M. Booth, & E. A. Wakeman
about once. The four simple models are not expected to get this close to the data. In fact, they differ considerably in the extent to which they can explain the observed distributions. The Poisson model is always rejected, and the truncated gaussian almost always (for 96.3% of the cases). The exponential model is rejected for 83.6% of the cases. The S+F model is rejected for only 15.4% of the cases: of the 22 data sets at five time scales, 13 give acceptable fits at all time scales, 3 are rejected for one window length, 4 are rejected at two window lengths, 2 at three window lengths, and for no data set is the fit to be rejected at four or all five window lengths. The rejections are more concentrated at shorter window lengths: five at 50 ms, four at 100 ms, six at 200 ms, two at 400 ms, and none at 800 ms. We should remember that the sharp separation between slow and fast fluctuations implied in the S+F model is clearly a simplification, particularly when considering multiple time scales, in each of which the separation is made at a different value. Nevertheless, for 13 data sets, the fit is acceptable at all window lengths, and for all data sets it is acceptable for at least two time window lengths. We can conclude that the agreement between the data and the S+F model is not just qualitative but is quantitatively a good fit. We show the response probability distributions for nine examples of the 22 available data sets, in Figures 2 (for a time window of 100 ms) and 3 (for a time window of 400 ms). These 9 data sets and two windows are representative of the quality of the results in all 22 data sets and five windows, as indicated by the similar percentage of model rejections when considering only these 18 cases: S + F = 16.6%; exp = 83.3%; Poisson = 100%; gaussian = 100%. An exponential curve fitted to the mean rate of each data set is shown by the dashed line in Figures 2 and 3. It is clear, particularly for the longer window, that many of the cells have a poor fit, with too few spikes at very low rates and too many spikes at a slightly higher rate. This is confirmed by the statistical analyses using the χ 2 goodness-of-fit test, as shown in Table 1. In Figures 2 and 3 we also show the fit of the S+F model to the observed rate distributions (solid line). The fits look much more acceptable. Again, this is confirmed by the statistical analyses using the χ 2 goodness-of-fit test, as quantified in Table 1. The Poisson and truncated gaussian models, which give much worse fits than the exponential model, are not shown. In particular, these last two models tend to tail off at high spike counts much faster than the real data. In addition, the Poisson model constrains the distribution to have a peak value at a nonzero count (in fact, at the mean count), and this is much worse than constraining it to have the peak always at zero, as in the case with the exponential model. To check that the general shape of the distribution is not especially dependent on either watching a video or on the particular video, we performed a similar analysis on the spike count distributions of the second population of 14 neurons responding to 65 natural static visual stimuli, presented during a visual fixation task. We present in Figure 4 three representative firing rate distributions of three different cells responding to static stimuli. (The time
Firing Rate Distributions and Information Transmission
615
window is 100 ms in Figure 4a and 400 ms in Figure 4b.) It is evident that the main features of the distribution (the gradedness, the unimodality, and the exponential-like tail) are found also in this case. Table 1 summarizes, for the experiment with static stimuli, the results of the statistical analysis using the χ 2 test for the fits of the observed distributions of spike counts to the various models. Considering 14 cells and four time windows, there are 56 possible fits. As before, the Poisson model was always rejected, and the truncated gaussian almost always (for 89.2% of cases). (The level of statistical significance used throughout was p < 0.01). The exponential model was rejected for 75% of cases. The S+F model was rejected in only one case (corresponding to 1.8%, reasonably within the acceptable range of rejections). We note that as the number of trials was smaller than in the video case, the different models were rejected less often for the experiment with static stimuli, but the number of trials was nevertheless high enough (380 to 600) to test adequately the fit of the firing rate distributions (see Figure 4) and to rule out all of the models analyzed apart from the S+F one (see Table 1). A further appreciation of the fact that the exponential distribution is not a very good fit to the observed firing rate distribution, especially at low rates, is provided by showing the average of the rate distribution across cells (possible by normalizing the mean rate of each cell to 1). This graph is shown as Figure 8a of Rolls et al. (1997b). Further, the graph for the video data averaged across cells and plotted in the same way appeared very similar to that figure and for that reason is not reproduced here. We conclude that despite its shortcomings, which we ascribe mainly to its simplicity, our hypothesis—that is, the S+F model—accounts for most of the data satisfactorily, and in any case better than the hypothesis that these distributions of firing rates tend to be maximum-entropy exponential distributions. The Poisson model has little to do with the data. The truncated gaussian model, whose sole difference with the S+F model is that it does not take fast variability into account, also gives very poor fits. 3.2 Parameters of the S+F Model from the Fits and Power Spectra. We present the results for the parameters of the best fits to the S+F model only for the video data. The reason for this is that the most interesting point that can be made from studying the parameters—the dependence of the parameters σF and σS on the duration of the time window—can be studied only for the video data, where the mean activation h0 was constant across the time windows and therefore the different time windows can be fitted simultaneously.4 For the video data, the parameters extracted from the fits for the S+F model take values in the range from about 10 to 100 Hz. The mean 4 For the static image data, either multiple nonoverlapping short windows are taken for each trial (e.g., eight consecutive 100 ms windows), but then they would refer to different phases in the responses; or the data analyzed are not the same across windows, and therefore h0 is not common to different window lengths.
616
A. Treves, S. Panzeri, E. T. Rolls, M. Booth, & E. A. Wakeman
A:
100 ms
0.3
0.3
0.4 0.3
0.2
0.2
0.2
Probability
0.1
0.1
0.1 0
0 0
4
8
0
12
4
am164
B:
8
12
0 0
am231
2
5
8 10
am235
400 ms
0.12
0.2
0.12
0.15
0.08
0.08 0.1
0.04
0.04
0.05 0
0
0 0
8
16
am164
24
0
8
16
24
am231
0
8
16
24
am235
Number of Spikes Figure 4: Probability histograms for three cells (responding to static stimuli). (A) The time window is 100 ms long. (B) The time window is 400 ms long. The conventions are as in Figure 2.
activation h0 (expressed in units of firing rate) varies from about −60 to +30 Hz across different data sets. Significant relationships were found between the parameters σF and σS describing, at different time scales, the amplitudes of the fast and slow fluctuations underlying each particular data set. In particular, the proportion of fast fluctuations increased logarithmically with the length of the time window (see Figure 6). This probably reflects that fact that for natural scenes, the power in the temporal spectrum of the intensity of a pixel decreases approximately as the inverse of the temporal frequency (Dong & Atick, 1995). If the fluctuations can in fact be divided to a good approximation between fast and slow, as the simple model posits, and if the parameters from the fit were to quantify their amplitudes precisely, one would expect the total power to remain constant, irrespective of where the division between slow and fast falls, 2 = constant. σS2 (L) + σF2 (L) = σTOT
(3.1)
Firing Rate Distributions and Information Transmission
617
90
st. dev. of fast noise (Hz)
80 70 60 50 40 30 20 10 10
20
30 40 50 60 70 st. dev. of slow noise (Hz)
80
90
Figure 5: (Ordinate) The standard deviation of the fast noise, σF (L). (Abscissa) The standard deviation of the slow noise, σS (L). Units in Hz (video data).
To check to what extent this occurs for each data set, Figure 5 shows the relation between the slow and fast variability parameters, with the points pertaining to the same data set linked by lines. If equation 3.1 were to be satisfied, all curves would describe arcs of a circle. For most data sets, the expected relation is not far from the observed one, considering that each of the five points on the curve comes from a different χ 2 minimization. This is a useful consistency check of the S+F model and indicates that the parameters σS and σF may indeed be associated with slow and fast fluctuations. Given this association, the observation that in Figure 5 most data sets lie between approximately 30 degrees and 60 degrees has further interesting implications. These are made clearer in Figure 6, where the proportion of the power assigned by the fit to fast fluctuations, F/(S + F) =
σF2 (L) σS2 (L) + σF2 (L)
,
(3.2)
is plotted against the time scale of the window used (on a log scale; each dashed line represents a different data set). If each of the four octaves included between our five time scales were to contribute an equal power to the fluctuations, the curves joining points from the same data set would be straight upward lines. This indeed appears to be the general trend, as emphasized by the bold line in Figure 6, which gives the average over the 22 data sets. Moreover, most of the curves start at values of F/(S + F) around 0.2 and end at values around 0.6–0.7. From these observations, we conclude that as the length of the time window increases, the contribution of the fast fluctuations increases approximately logarithmically. Given that the total
618
A. Treves, S. Panzeri, E. T. Rolls, M. Booth, & E. A. Wakeman
1
0.8
F/(S+F)
0.6
0.4
0.2
0
50
100
200 Time Bin Duration (ms)
400
800
Figure 6: (Ordinate) The fraction of the variability that is due to fast noise, F/(S + F). (Abscissa) The time window on a log scale. Dashed lines represent single data sets; the bold line is the average across data sets (video data).
variance of the slow and fast fluctuations is approximately constant, this implies that the contribution of the slow fluctuations decreases logarithmically. This is what would be expected if the sources of the slow and fast fluctuations were evenly distributed across time scales over a wide range that extends from below our shortest window of 50 ms to above the longest one of 800 ms. In statistics, this trend is often referred to as the 1/f law, and it is theorized to underlie many random processes. Note that if the power of fluctuations were distributed as p( f )df ∝ df/f , then the fraction of power between two frequencies f1 and f2 = f1 /2 differing by an octave would be proportional to log( f1 ) − log( f2 ) = log 2, that is, constant, as we do approximately find. An underlying basis of this may simply be that no intrinsic time scale is characteristic of the real images being seen in the video, in which some images, or elements of images, last for a long time and others for a shorter time (see Dong & Atick, 1995, for a quantitative analysis of the statistics of natural images). It is interesting in this respect to analyze the standard direct measure of the variability at different time scales, that is, the power spectrum of the spike trains of the cells responding to the video. The power spectra for our set of cells do show an approximate 1/f behavior at low frequencies,
Firing Rate Distributions and Information Transmission
619
0.055 0.05 0.045
Power
0.04 0.035 0.03 0.025 0.02 0.015 0.01 0.005 0.78
6.25
12.5
25 Frequency (Hz)
Figure 7: The average across data sets of the normalized power spectrum of the spike train for each recording session. The sampling frequency is 200 Hz (video data).
typically up to 4–8 Hz (see Figure 7). This is completely consistent with the S+F model, in which the slow fluctuations of the activation are found to decrease in the same way, but on a much wider frequency range. The power spectra tend to level off to a constant value instead of tailing away. In fact this is just an artifact intrinsic to extracting power spectra from spike trains (in which each isolated spike is essentially a delta function with a flat Fourier transform) rather than from a variable changing continuously in time, as discussed by Bair, Koch, Newsome, and Britten (1994), for example. Note that both the initial 1/f trend and the ensuing flatness of the power spectra at high frequencies are still visible when sampling at 1000 Hz, in which case all spikes, recorded at 1 ms resolution, indeed appear as delta functions. The high-frequency, almost flat portion of the spectra is therefore mainly telling us that we are looking at spikes—discrete events. The most informative part is the low-frequency end, where the shape we find is interestingly different from that observed in other (and very different) experiments, for example, by Bair et al. (1994). The parameters extracted from the S+F model are not affected by the fact that neuronal output is in the form of spikes. Their values are consistent with the shape of power spectra at low frequencies, but provide informative evidence over a wider frequency range. 3.3 Quantifying the Efficiency of the Distributions. We measured the information efficiency % introduced in section 2 and discussed below for our 22 video data sets and also for the data obtained with static images in a visual fixation task (Rolls & Tov´ee, 1995; Rolls, Treves, & Tov´ee, 1997a; Rolls et al., 1997b). The important difference with the video experiment is that each time
620
A. Treves, S. Panzeri, E. T. Rolls, M. Booth, & E. A. Wakeman
window (for the static image data, we choose windows of 12, 25, 50, 100, 200, and 400 ms, starting at 100 ms after presentation of the image) corresponds to a trial with a single stimulus, and trials with the same stimulus are repeated between four and ten times to obtain reliable estimates of the firing rate. In this situation, the part of the activation that is constant across trials with the same stimulus can be taken to be the signal, while the part that varies can be considered noise. The firing statistics of each cell can therefore be related to the transmission of the signal, 