Psychophysiology, 48 (2011), 293–302. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01065.x
Feedback processing in hypothesis testing: A Laplacian ERP study
LAURENCE CARBONNELL,a JEAN-PAUL CAVERNI,b and FRANCK VIDALa,c a
Institut de Recherche Biome´dicale des Arme´es, Antenne Toulon, Toulon, France Laboratoire de Psychologie Cognitive, Aix-Marseille Universite´/CNRS, Marseille, France c Laboratoire de Neurobiologie de la Cognition, Aix-Marseille Universite´/CNRS, Marseille, France b
Abstract The capacity to adjust behavior and evaluate performance in a changing environment is highly related to taking feedback into account. Using feedback is, hence, crucial for different cognitive capacities. The present study focused on feedback processing in hypothetico-deductive reasoning. Electroencephalographical (EEG) activity was recorded while participants performed a hypothesis-testing task in which a feedback signal allowing the evaluation of performance was delivered at each trial. To ameliorate the spatial resolution of EEG signals, the surface Laplacian was estimated. Results showed three distinct patterns of activation. At central and centro-parietal electrodes, the activity was sensitive to participants’ expectancies in a binary and a gradual way, respectively. At the parieto-occipital electrode, the activity was sensitive to the evaluation of performance itself: correct versus false. In conclusion, the processing of feedback signal depends on at least three separate processes. Descriptors: Feedback processing, Hypothetico-deductive reasoning, Surface Laplacian, Separated sources
terpreted as reasoning biases. However, in some cases, the expression of these biases is not sufficient to confirm a reasoning bias. For example, taking into account participants’ expectancies relative to feedback is crucial. However, asking participants about their expectancies might change the initial strategy used to resolve the task (Caverni & Rossi, 1997; Wetherick, 1962). That is why, in a previous study, we elaborated an hypotheticodeductive reasoning task in which the measure of the electrodermal activity (EDA), sensitive to participants’ expectancies (Lim et al., 1997, 1999; Siddle, 1985; Simons et al., 1987), demonstrated the existence of a reasoning bias (Carbonnell, Vidal, Sequeira, & Caverni, 2006). At the level of the electroencephalogram (EEG), several patterns of activations were evidenced after feedback. In a time estimation task, Miltner, Braun, and Coles (1997) evidenced a fronto-central negativity that reaches maximum amplitude about 250 ms following the onset of negative feedback. This negativity shares functional and topographical similarities with the error negativity (Falkenstein, Hohnsbein, Hoormann, & Blanke, 1991) also called error-related negativity (Gehring, Goss, Coles, Meyer, & Donchin, 1993), observed at erroneous motor response onset. This is why the authors proposed that these negativities were the manifestation of a ‘‘generic’’ neural system involved in error detection. Also, this negative wave after the feedback signal is also called feedback error-related negativity (fERN; e.g., Holroyd, Pakzad-Vaezi, & Krigolson, 2008). Apparently, presentation of positive feedback did not elicit an fERN, but, instead, a large positivity called feedback correct-related positivity (fCRP; Holroyd et al., 2008), suggesting that the system that produces this component was differentially sensitive
The capacity to adjust behavior in a changing environment is essential. This involves that the individual must be able to take into account feedback signals from the environment. Hence, to take feedback into account is essential to evaluate performance. Every day, people make hypotheses or apply rules that can be directly tested in interaction with the environment. They receive feedback that allows them to verify or to correct their hypotheses and find the appropriate behavior. When people get positive feedback, then their hypotheses can be kept in mind and used appropriately. When they get negative feedback, they have to revise or leave their hypotheses. Feedback is, hence, crucial for reasoning. The question of this study is: How is feedback used and processed by the central nervous system when people reason? In hypothetico-deductive reasoning, the way in which feedback is used depends on the phase in reasoning. In the inductive phase, feedback is used to elaborate hypotheses. In the deductive phase, feedback is used to verify the most probable hypothesis: If the hypothesis is correct, feedback confirms it, but if participants make an error of logic, feedback refutes it. In many tasks, participants show response biases, a systematic behavior apparently incompatible with the application of formal logical rules. These response biases have often been inWe thank Jean-Luc Peris for his computer science contribution and Sonia Allain and Dominique Reybaud for their technical assistance. The first author was supported by a grant from the French Ministry of Research and Education. Address correspondence to: Dr. Laurence Carbonnell, Institut de Recherche Biome´dicale des Arme´es, Antenne Toulon, Boulevard Sainte Anne, BP 20548, 83041 Toulon cedex 9, France. E-mail: laurence.
[email protected] 293
294 to positive and negative feedback (Miltner et al., 1997). Holroyd and Coles (2002) proposed that the fERN ‘‘is produced when the system first detects that the consequences of an action are worse than expected’’ (p. 694). However, it has been shown more recently that an uninformative feedback signal also elicited a negativity almost as large and sometimes larger than that elicited by negative feedback (Mu¨ller, Mo¨ller, Rodriguez-Fornells, & Mu¨nte, 2005). Moreover, Oliveira, McDonald, and Goodman (2007) showed that correct feedback signals evoked a negativity as large as the negativity elicited by error feedback when participants had misestimated their performance or when the signal corresponding to correct feedback was delivered but participants’ performance was not correct. Therefore, the authors concluded that the negativity elicited after error feedback signals does not seem specific to the negative content of the feedback signal, but may be ‘‘related directly to the recognition of violations in expectancy’’ (p. 2001), when the consequences of an action are simply unexpected. Finally, Holroyd et al. (2008) proposed that the fERN would be, in fact, an N200 occurring also on correct trials but masked by the fCRP (see also Simons, 2010, for a review about the fERN). At variance with these results, some studies did not evidence any clear negativity after feedback (Campbell, Courchesne, & Squires, 1979; Chwilla & Brunia, 1991; Stuss & Picton, 1978) but showed a large positivity (called P300). For example, in a time estimation task, Chwilla and Brunia showed a centroparietal positivity peaking about 400 ms after feedback onset sensitive to the informative value of feedback: larger for true compared to false feedback signal. They showed, in addition, that a similar but more frontocentral P300 was larger for incorrect than for correct feedback. However, as the authors noted, the P300 was shown to be also modulated by the signal probability (Campbell et al., 1979). Because in Chwilla and Brunia’s study the incorrect feedback was rarer than the correct feedback, it was, hence, not possible to determine if this modulation was due only to the nature of the feedback signal itself or to its occurrence probability. Interestingly, one can note a little negative rebound at about 250 ms embedded in the large positivity. In the same way, one can notice that the negativity observed after feedback is often followed by or even embedded in a large positivity. In fact, it seems that there are two components related to a feedback signal: a negativity and a positivity, which may be modulated independently one from each other (Holroyd et al., 2008). However, because of volume conduction effects, these two components overlap. Indeed, when cerebral electrical activity goes through the resistive skull bones, it diffuses out at the surface of the scalp. Hence, if different sources are coactivated, their activities may overlap in space and in time. This would explain why the positivity and the negativity elicited by feedback do not always appear clearly. In the present study, we focused on feedback processing when people reason or, more precisely, when they test their hypotheses to discover a hidden rule determined by the experimenter. Only a few event-related potential (ERP) studies have focused on this topic. Moreover, results may vary. For example, Papo, Baudonnie`re, Hugueville, and Caverni (2003) evidenced a negativity only after negative feedback, whereas Zanolie et al. (2008) evidenced a negativity also after positive feedback. However, a pioneering study in this domain (Stuss & Picton, 1978) did not show any modulation of negativity after feedback signal whether positive or negative, but rather large late positivities were evident. These positivities were greater after negative feedback, denoting a change in criterion and the need to formulate new hypotheses.
L. Carbonnell et al. Moreover, they were of large amplitude in the research stage called the ‘‘preinsight’’ stage. In this stage, all the selected trials did contain a negative feedback signal, but it is noteworthy that these negative feedback signals were much rarer than the positive (Stuss & Picton, 1978). Here again, this effect may be confounded with signal probability or novelty because negative feedback signals were rarer than positive. This discrepancy in the results could be explained by overlapping effects between positivity and negativity. Indeed, as emphasized by Arbel and Donchin (2009), because of current diffusion at the scalp surface (volume conduction effects), using an appropriate source separation method ‘‘is critical in the study of ERPs, as it is evident that some ERP components overlap in space and in time’’ (p. 1181). Therefore, after having recorded brain electrical activity while participants performed an hypothetico-deductive reasoning task (Carbonnell et al., 2006), we estimated the surface Laplacian to minimize volume conduction and, therefore, overlapping effects in EEG. Acting as a high-pass spatial filter, the surface Laplacian removes the blurring effects of resistive skull bones (Katznelson, 1981). As such, it provides a good approximation of the corticogram (Gevins, 1989). Moreover, the temporal resolution of EEG data is improved, because the surface Laplacian allows the observance of the time course of individual sources separately (Law, Rohrbaugh, Adams, & Eckardt, 1993). In this way, the overlapping effects between the feedback negativity and positivity, if any, should be removed, or at least minimized, and should allow the independent analysis of the variations of these two waves. Chwilla and Brunia (1991) evidenced a P300 that was enhanced for incorrect feedback signal. Stuss and Picton (1978) showed increased positivities after negative feedback, denoting a change in criterion and the need to formulate new hypotheses. However, it was not clear if this modulation was due to incorrect feedback or to the probability of the occurrence of this signal. In the present study, it was necessary to make the occurrence of the feedback signals equiprobable; that is why two feedback signals directly indicated the to-be-given response, so that participants had to compare their own responses with the feedback signal and confirm the correctness or incorrectness of their response. Hence, if any cerebral activity is modulated after feedback, it will not be attributable to the occurrence probability of the feedback signal itself. If the fERN is actually modulated by the expected or unexpected character of the feedback signal, as suggested by Oliveira et al. (2007), we predict the occurrence of such a negativity when participants make an error of reasoning and get negative feedback. This negativity should not appear during the inductive and deductive phases. In the present reasoning task, there are four reasoning phases in which the status of feedback is different. (1) In the inductive phase, the rule cannot be discovered yet, and participants should have no specific expectation about the confirmative or infirmative nature of feedback signals. (2) Conversely, during the deductive phase, the rule being discovered, participants should expect their responses to be confirmed. (3) At the end of the deductive phase, participants get a disconfirmation feedback signal that indicates rule switching; a new rule has to be discovered. (4) At the turning point between the inductive and the deductive phase, there is a key trial at the level of which we have already evidenced a reasoning bias (Carbonnell et al., 2006); participants were prematurely sure they had discovered the correct rule and they expected a confirmation. However, as they did
Feedback processing in hypothesis testing
295
not take all the elements into account in most of the cases, they got a disconfirmation which corresponds to an error signal. The effect of confirmative versus disconfirmative nature of feedback signals and the effect of feedback signals as a function of the reasoning phases (or participants’ expectancies) were analyzed on the evoked potentials time-locked to the feedback signals on monopolar recordings and after Laplacian transformation.
Materials and Methods Participants Ten participants volunteered (7 women, 1 left-handed, mean age: 25, range: 22–29). All had normal or corrected-to-normal vision. They were seated in a Faraday cage in front of a faradized computer screen on which stimuli were displayed. They were asked to respond to stimuli by pressing alternatively with the thumbs two mice, one in each hand.1 Informed consent was obtained. The rights of participants were protected according to the Declaration of Helsinki. Design and Stimuli The task of the participants was to discover a hidden rule determined by the experimenter. To this aim, pairs of cards displaying geometrical figures were sequentially presented to participants (Figure 1). These pairs of cards did or did not represent examples of the rule. These geometrical figures varied according to three attributes: shape, color, and number. Each attribute had three modalities: shape (square, circle, or star), color (green, red, or blue), and number (one, two, or three identical figures on the same card). Each pair of cards could be either congruent with the rule to discover (the relevant modalities were displayed on each card of the pair) or incongruent (the relevant modalities were not displayed at all or not displayed on each card of the pair). The rule was either based on one modality of the geometrical figures (‘‘circle,’’ ‘‘two,’’ etc.) or corresponded to a conjunction of two modalities (‘‘two and circle,’’ ‘‘green and square,’’ etc.), that is, a subset of the preceding kind of rule (‘‘two and circle’’ is included in ‘‘circle’’). To simplify, we termed these rules, respectively, general rules and specific rules. Participants were explicitly informed that only those two kinds of rules were possible. Their task was to successively discover a set of specific and general rules. Each rule had to be discovered within a finite number of trials (8 to 12). Participants were asked to respond as accurately as possible after the presentation of each pair of cards by pressing with the thumb the right or left mouse (Figure 2), the right if they considered that the example fit the rule to discover or the left if they considered that the example did not fit the rule. Then, they were asked to produce a second key press to indicate whether they were confident in the correctness of their response: They had to press the same key again if they were sure they were right or press the opposite key if they were not sure. After both key presses, participants received feedback. There were two feedback signals: the letter x in closed square brackets [x] or in open square brackets ]x[. To avoid effects related to signal probability on 1 Both buttons of one mouse were programmed to trigger the same response when pressed. Participants chose to press the button that was the more comfortable for them.
Figure 1. The figure is to be read line by line from ‘‘Trials’’ to ‘‘Possible rules after feedback (FB).’’ For set n, the rule to discover is ‘‘circle.’’ At the first trial, a pair of cards displaying two green circles is presented (white figures correspond to green figures in the real task). Feedback indicates that this pair of cards fits the rule. Therefore after feedback, six rules remain possible. Trial by trial, some possible rules are eliminated until one rule remains possible (i.e., after the key trial). The switch trial occurring just after the sixth trial of set n is the first trial of set n11: indeed, after feedback, the rule ‘‘circle’’ is no more valid and ‘‘red and star’’ is the new rule to discover. The fourth trial of the set n is a key trial at the level of which one specific (‘‘two and circle’’) and one general rule (‘‘circle’’) remain possible. The inductive (research) phase is before the key trial. The deductive phase is after the key trial.
the ERPs, the occurrence of both feedback signals had to be equiprobable. To this aim, feedback signals did not directly inform participants about their performance, but indicated whether the pair of cards fit (indicated by [x]) or did not fit (indicated by ]x[) the rule to discover. Hence, both feedback signals could be either positive or negative, depending on participants’ response. After 8 to 12 trials, the rule to discover changed (Figure 1, switch trial). Participants were explicitly informed that the rule could change but they were not informed about the moment of this switching. They had to infer this switching and discover the new rule. There were 50 rules to discover: 25 specific and 25 general, presented in a nonpredictable order. For each 50 sets of trials, there was a key trial. At the key trial, two rules, one specific and one general, remained possible before feedback: (Figure 1, key trial). A reasoning bias was evidenced at the level of this trial (Carbonnell et al., 2006). Before the experimental session, written instructions were given to participants. Then they were asked to perform a test run aloud, so that they could also ask questions. The experimenter then explained orally the types of rules (specific vs. general) once more. When the participants were judged to be performing the
296
L. Carbonnell et al.
Figure 2. Let us imagine that, at the first trial of set n of Figure 1, a participant pressed the right mouse, testing the hypothesis that the example fits the rule. Then the participant performed a second key press on the opposite side showing that he/she was not sure that the first response was correct. Then, feedback indicated that the pair of cards did fit the rule. Now, at the fourth (key) trial, one specific and one general rule were still possible. Let us imagine, as it has been shown most often to be the case (Carbonnell et al., 2006), that the participant considered only the specific rule (‘‘two and circle’’). He/she pressed the left mouse, suggesting that the example does not fit the rule. Then the same key was pressed a second time indicating that he/she was sure about the correctness of the response. Error feedback was then displayed to indicate that the example fit the rule and so on.
task correctly, the experimental session began with their agreement. The two cards of each pair had at least one modality in common. A pair of cards was displayed for a maximum of 10 s, so that participants were given enough time to respond as accurately as possible. The response erased the stimulus, and 500 ms later the feedback signal was displayed during 1 s. Then a new pair of cards was displayed 500 ms later. Electrophysiology Recordings. The EEG was recorded continuously during the experimental blocks from 18 Ag/AgCl scalp electrodes. The reference and ground were, respectively, on the right and left mastoids. The electro-oculogram (EOG) was recorded bipolarly: One electrode was above the right eye and the other at its outer canthus (Figure 3). EEG and EOG signals were fed into Nicolet amplifiers, amplified 30,000 times, filtered, and digitized online (bandwidth: 0.016–100 Hz, 12 dB/octave, sampling rate: 256 Hz). No selective notch 50-Hz filter or additional digital filtering was used. Impedances were kept below 5 kO. To increase the spatial resolution of the signal, the time course of the surface Laplacian (Babiloni, Cincotti, Carducci, Rossini, & Babiloni, 2001) was estimated using the source derivation
Figure 3. Electrode position on the scalp. The eight nodal electrodes (in black) were on the medial line of the scalp. The nodal electrodes were equidistantly surrounded by three electrodes (gray and black) disposed in equilateral triangular arrays. The interelectrode distance was d (1/20th of the inion-nasion plus traggus-traggus distance: 3.6 cm on average). The surface Laplacian was estimated at the nodal electrodes by the method of Hjorth (1975) adapted by MacKay (1983). For example for Cz, the three surrounding electrodes were FCz, A, and B. Laplacian estimation at Cz is DVCz ! [4/3(VCz " VFCz1VCz " VA1VCz " VB)]/d2, where VCz is the potential recorded at the nodal electrode Cz and VA, VB, and VFCz are the potentials recorded at the surrounding electrodes.
method (Hjorth, 1975) modified by MacKay (1983).2 A specific electrode configuration that partly differs from the 10/20 system was used. Equilateral triangles were formed with one electrode at each apex. A ‘‘nodal’’ electrode was placed at the center of these triangles, and the surface Laplacian was estimated at these nodal electrodes. The distance between a nodal electrode and its three surrounding electrodes was 1/20th of the inion-nasion plus tragus-tragus distance (3.6 cm on average). As the negativity elicited by feedback is classically observed over central and frontocentral regions and as the positivity is observed either over frontocentral or centroparietal regions, nodal electrodes covered the medial line at AFz, Fz, FCz, Cz, CPz, Pz, POz, and Oz (Figure 3). Artifact rejection. Ocular artifacts were subtracted by a statistical method (Semlitsch, Anderer, Schuster, & Presslich, 1986). Trials for which the subtraction was not satisfactory as well as trials containing other artifacts were rejected on the basis of a trial-by-trial visual inspection of the monopolar recordings. Thereafter, the remaining monopolar recordings were averaged and the surface Laplacian was estimated at the nodal electrodes. Data Processing Behavior. To verify that participants correctly performed the task, the percentages of feedback signals congruent with the responses (termed positive feedback to simplify) in the deductive phase as well as for the switching trial were calculated. If par2 Tandonnet, Burle, Hasbroucq, and Vidal (2005) provided empirical evidence that the scalp current density (Perrin, Bertrand, & Pernier, 1987) and the source derivation methods of surface Laplacian estimation are equivalent to improve the spatial resolution of EEG traces.
Feedback processing in hypothesis testing ticipants performed the task well, then these percentages should not have been far from 100% and 0%, respectively. To show that participants presented a reasoning bias at the key trial, the percentage of responses compatible with the test of specific rules was calculated. As replication of the reasoning bias, this percentage was expected to be larger than 50%. Moreover, participants should have been sure they were right when they tested specific rules. Therefore, the percentage of specific rules tested with confidence was also calculated. It was expected to be larger than 50%. The percentage of general rules tested with confidence was also calculated and should not have been larger than 50%. For the reasoning bias, all percentages were compared with the one-tailed Student’s t test to the random value (50%). Arcsines transformation was applied on response rate squares (Winer, 1970). ERPs. Trials were sorted into five categories: (1) induction positive feedback (inductive phase, response given with uncertainty [second key press on the opposite side] and positive feedback), (2) induction negative feedback (inductive phase, response given with uncertainty [second key press on the opposite side] and negative feedback), (3) deduction feedback (deductive phase, response given with certainty [second key press on the same side] and positive feedback), (4) switching feedback (switch trial, response given with certainty [second key press on the same side] and negative feedback), and (5) error feedback (key trial, response given with certainty [second key press on the same side] and negative feedback). Feedback-related activities were averaged with a 100-ms prefeedback baseline. A visual inspection of the grand averages shows that the main differences were from 400 to 600 ms. Therefore, the mean amplitudes of the Laplacian traces were calculated and analyzed in this 400- to 600-ms time window. A repeated measures analysis of variance (ANOVA) was performed with the factor feedback (F5) as the within-participants factor at CPz (greater amplitude) for the monopolar recordings. Because, on the Laplacian curves, the patterns of activation are different from one nodal electrode to the other, the ANOVA was performed for each nodal electrode. Results Behavior Deductive phase and switch trial. The percentage of positive feedback in the deductive phase was 96%. The percentage of positive feedback was 3% at the switching trial. It can be concluded that participants correctly performed the task. Key trial (reasoning bias; Figure 4). The percentage of responses compatible with the test of specific rules was 72% and significantly superior to 50%, t(9) 5 3.69, p 5 .0025. Moreover, the percentage of specific rules tested with confidence was 77% and was also significantly superior to 50%, t(9) 5 3.63, p 5 .0027. Conversely, participants rarely tested general rules with confidence (30%). This percentage was not different from 50%, t(9) 5 1.72. ERPs (Figures 5 and 6) For the Laplacian traces (Figure 5), the ANOVA showed an effect of the factor F for the nodal electrodes Cz, F(4,36) 5 2.66, p 5 .048, CPz, F(4,36) 5 7.31, p 5 .0002, and POz, F(4,36) 5 2.74, p 5 .043. The effect of the factor F was not
297
Figure 4. Percentages of responses given with or without confidence at the key trial for the test of specific rules and general rules.
significant for the other nodal electrodes: AFz and FCz, Fs(4,36)o1; Fz, F(4,36) 5 2.04; Pz, F(4,36) 5 1.04; and Oz, F(4,36) 5 1.83. Cz. Visual inspection of the traces shows two distinct groups of curves. One corresponds to induction positive feedback, induction negative feedback, and deduction feedback. The other corresponds to switching and error feedback (the most positive group). Contrasts were performed inside each group. The curves were not different, all Fs(1,9)o1. A contrast was performed between both groups. The activities of each group were different, F(1,9) 5 9.53, p 5 .013. CPz. As for Cz, visual inspection of the traces shows that the curves corresponding to induction positive feedback, induction negative feedback, and deduction feedback do not differentiate. This group of curves differentiates from the switching feedback curve, which is more positive. This later differentiates from the error feedback curve, which is the most positive. Therefore, contrasts were performed between the curves inside the group, between the group and switching feedback, between the group and error feedback, and between switching feedback and error feedback. The three curves of the group are not different, Fs(1,9)o1. The switching feedback curve is more positive than the activity of the group, F(1,9) 5 7.47, p 5 .023, as well as the error feedback curve, F(1,9) 5 15.05, p 5 .0037. The error feedback curve is more positive than the switching feedback curve, F(1,9) 5 5.6, p 5 .042. On monopolar recordings (Figure 6), one can observe the same pattern of activity, which is more or less ample according to the electrode site. Hence, similar comparisons were performed at CPz, where the effect was maximal on monopolar recordings. The three curves of the group are not different, Fs(1,9)o1. The switching feedback curve and the error feedback curve are more positive than the activity of the group, respectively, F(1,9) 5 7.47, p 5 .023, and F(1,9) 5 15.05, p 5 .0037. The error feedback curve is more positive than the switching feedback curve, F(1,9) 5 5.6, p 5 .042. POz. Visual inspection of the traces showed two distinct groups of curves: positive feedback (deduction and induction positive feedback) versus negative feedback (the most positive group: switching, error, and induction negative feedback). Contrasts were performed inside and between both groups. Inside both groups, the curves are not different, Fs(1,9)o1. Between
298
L. Carbonnell et al. estimation and monopolar recordings. The correlation coefficient was also calculated between adjacent electrodes, and the independency between adjacent electrodes was tested for Laplacian estimation as well as for monopolar recordings. Results are presented in Table 1. As we can see on monopolar recordings, the activities from adjacent channels do not differentiate and are highly correlated. This result shows that the four adjacent channels are very dependent on monopolar recordings. Conversely, results are very different for Laplacian estimation traces. Although the activities between Cz and CPz and between POz and Oz are closely similar, no correlation between adjacent electrodes is shown. Moreover, between CPz and Pz, the activities are significantly different. This shows that, contrary to monopolar recordings, Laplacian estimation done by strongly reducing volume conduction effects allows separation of the sources. In conclusion, although at first sight, after Laplacian estimation, patterns of activation look alike, contrary to monopolar recordings, no correlation between adjacent electrodes are evident. Moreover, in the example, activities of two adjacent electrodes are significantly different. This means that there is a topographical independency between the traces after Laplacian estimation. Now the fact that adjacent electrodes present similar patterns of activity after Laplacian estimation is probably not a weakness of the method but reflects a common sensitivity to a generator close enough to the electrodes to affect them in a similar way. Discussion
Figure 5. Laplacian estimation amplitude in microvolts per centimeter squared as a function of time in milliseconds corresponding to the five feedback conditions. Time 0 corresponds to feedback onset.
both groups, the activities are different, F(1,9) 5 20.51, p 5 .0014. Methodological Note: Independency of Channels after Laplacian Estimation To show how independent the observations from different channels are after Laplacian estimation, we compared the activity recorded over four adjacent channels (Cz, CPz, Pz, and POz) in the induction negative feedback condition before and after Laplacian estimation (Figure 7). According to the criteria described in data processing, mean amplitudes of the four adjacent electrodes (calculated in the 400–600-ms time window with a 100-ms prefeedback baseline) were compared with the two-tailed Student’s t test for Laplacian
The behavioral results of the present study replicate those found previously (Carbonnell et al., 2006). In the inductive phase, participants responded at chance, which shows that they were actually searching for the rule. In the deductive phase, participants gave almost systematically the correct response, which indicated that they had found the correct rule; this fact is confirmed by the analysis of an additional trial (switch trial), for which almost all the responses were invalidated. Results also replicate the reasoning bias at the key trial, that is, the propensity of the participants to consider directly the specific rules, corresponding to the conjunction of two modalities (e.g., ‘‘two and circle’’) as correct without considering the general rules, corresponding to one modality (e.g., ‘‘circle’’). Moreover, the bias persists despite the fact that participants were explicitly informed about the two types of rules, first, with the written instructions and, second, orally during training. That means that this reasoning bias is actually well anchored. The theoretical implications of this reasoning bias have already been discussed (Carbonnell et al., 2006). It was concluded that the strategy used here could be explained in terms of the mental model theory (Johnson-Laird, 1983). Participants performed the task as expected, which meant that the feedback signals were actually used differently according to the reasoning phases. Moreover, as they were asked to estimate their performance (second key press), we could evaluate directly their expectancies and select properly the trials revealing inductive and deductive activities and especially errors of reasoning at the key trial. Participants’ expectancies were different according to the selected trials. There were, hence, different levels of violation in expectancies. In the selected trials for EEG analyses, for the key trial, participants made an error of reasoning; they got a
Feedback processing in hypothesis testing
299
Figure 6. Monopolar recordings at the 18 channels showing the activities recorded for the five feedback conditions. The amplitude of the potentials in microvolts is represented as a function of time in milliseconds. Time 0 corresponds to feedback onset. Channels are presented in the same configuration as in Figure 3, showing the position of the electrodes.
300
L. Carbonnell et al.
Figure 7. Activities of four adjacent electrodes (Cz, CPz, Pz, and POz) in the induction negative feedback condition. Top: Amplitude of Laplacian estimation in microvolts per centimeter squared as a function of time in milliseconds. Bottom: Amplitude of the potentials in microvolts of monopolar recordings as a function of time in milliseconds.
disconfirmation whereas they expected a confirmation. For the deductive phase, participants expected a confirmation and got a confirmation. For the inductive phase, participants had no specific expectations and got either a confirmation or a disconfirmation. Finally, for the switch trial, participants expected more a confirmation but knowing, however, that they could get a disconfirmation. As in Chwilla and Brunia (1991) and in Stuss and Picton (1978), electrophysiological results of the present study show no fERN on monopolar traces; only a slight negative-going wave about 250 ms after feedback embedded in a large positivity can Table 1. Paired Comparisons of the Activities of the Four Adjacent Electrodes (Cz, CPz, Pz, and POz) with the Two-Tailed Student’s t Test for Laplacian Estimation and Monopolar Recordings and Correlation Coefficients of Adjacent Electrodes Laplacian estimation Cz/CPz CPz/Pz Pz/POz t(9) p5 r F(1,8) p5
0.36 n.s. .51 2.81 n.s.
3.92 .0035 .40 1.49 n.s.
0.16 n.s. .37 1.26 n.s.
Monopolar recordings Cz/CPz
CPz/Pz
Pz/POz
0.82 0.38 2.16 n.s. n.s. n.s. .95 .96 .92 72.75 84.75 45.68 .000027 .000016 .00014
be observed. However, as explained in the introduction section, to minimize overlapping effects due to volume conduction and to observe independently, if possible, the feedback-related negativity and the feedback-related positivity, the Laplacian transformation was applied. Results showed that at FCz, where the feedback-related negativity is classically observed, no negativity was unmasked at 250 ms after Laplacian transformation despite the fact that the positivity was reduced. First, as Laplacians are quite insensitive to deep sources, because they ‘‘decrease with eccentricity more rapidly than potentials’’ (Pernier, Perrin, & Bertrand, 1988, p. 388), one explanation is that the negativity came from the depth. Second, it can be argued that feedback signals are processed differently in hypotheses testing compared to tasks involving more elementary operations, such as time estimation tasks. Third, to avoid effects due to the occurrence probability of the signal, we made the occurrence of the feedback signal equiprobable. The feedback signal, representing the correct response, did not directly deliver information regarding participants’ performance. Hence, participants had to infer themselves whether they were right or wrong by comparing their response with the feedback signal. Compared to a situation in which the feedback signal gives direct information on performance, there is an additional comparison process occurring before the actual evaluation of the performance. If so, the negativity should occur later. But no negativity sensitive either to the nature of feedback (positive vs. negative) or to participants’ expectancies occurred after 250 ms. As observed in a majority of feedback studies, feedbackrelated potentials of the present study revealed a large positivity on the monopolar traces. Compared to other sites, the positivity is maximal at CPz as classically described. At this location, the positivity is maximal for the key trial, when participants’ expectancies are the most violated. It almost disappears in the inductive phase, for which there is no violation because participants had no expectancies. It also almost disappears in the deductive phase, for which there is also no violation because participants expected a confirmation and got a confirmation. It is intermediate for the switch trial, for which there is a partial violation as expectancy is weaker; participants knew that the rule may change. Hence, the positivity increases with the degree of violation in expectancies. This positivity occurs later than in time estimation tasks. However, the fact that the positivity occurred later in our study in comparison with other studies on feedbackrelated potentials may be explained by the nature of the feedback, which imposes indirectly extracting information on the performance of participants. A late positivity (called P600), with the same latency, topography, and amplitude, has been described when an unexpected event occurrs. This positivity was observed when a syntactic violation occured in different languages: English (Hagoort, Brown, & Groothusen, 1993; Osterhout & Holcomb, 1992, 1993), Italian (Angrilli et al., 2002), German (Gunter, Friederici, & Schriefers, 2000), or Japanese (Nakagome et al., 2001). Other studies (Patel, Gibson, Ratner, Besson, & Holcomb, 1998) showed that this positivity was also elicited when acoustical dissonances occurred in a musical sequence. Finally, this wave also occurred when a word was visually incongruent with other words in a written sentence (written in big capital letters whereas the other words were written in little capital letters; Kutas & Hillyard, 1980). These results are compatible with the fact that, in the present study, a positivity occurred for unexpected feedback signals. This activity would seem to evidence a general
Feedback processing in hypothesis testing
301
process involved in violation in expectancies when an unexpected event occurs. It has to be noted that, in the present study, with the objective probability of each feedback signal being 50%, the expectancies were actually subjective and contextual. On classical monopolar recordings, the pattern of activation is quite the same at all electrodes with different amplitudes. In addition, we showed in the results section that there was a high correlation and no difference between the activities recorded at adjacent electrodes, at least between Cz, CPz, Pz, and POz. After Laplacian transformation, none of these correlations was significant anymore, and a difference appeared between adjacent electrodes (CPz and Pz). In fact, different patterns of activities evoked by feedback signals (which appear on monopolar traces as a weighted average due to volume conduction effects) are disentangled. The large positivities are differently modulated by information that can be extracted from the feedback signal at Cz, CPz, and POz in the same latency range. At CPz, the positivity, as in the monopolar traces, shows a gradual sensitivity to violation in expectancies. It is worth noticing that the Laplacian estimation in this case does not provide further information as compared to the results obtained with conventional monopolar recordings. Therefore results obtained on monopolar recordings as well as after Laplacian estimation index a processing relying on a subjective (internal) criteria corresponding to the degree of violation in participants’ expectancies. Conversely, at Cz, the positivity is no more gradually sensitive but binarily sensitive to violation in expectancies. Therefore, another component seems to extract that there is either a violation in expectancies or not, independent of the degree of this violation. Now, at POz, results evidenced a new pattern of modulation of brain activity that did not appear on monopolar recordings. The activity is here modulated according to the confirmative versus disconfirmative nature of feedback. This component is sensitive to the outcome of performance evaluation, and it is absolutely independent of participants’ expectancies. This indexes a processing relying on an objective (external) criteria making a sorting out between confirmed and infirmed responses whatever the reasoning phase in which the participants are. For example, this process is not sensitive to errors. In the present task, the feedback signal indicated whether the example belongs to the rule or not. From the feedback signal,
participants extracted the relevant information to adequately perform the task. The categorization of feedback in different classes was shown thanks to the Laplacian estimation effect, which allows a spatial separation of components. One has to note that other methods could have been proposed to overcome component overlapping effects. Principal component analysis (PCA; Spencer, Dien, & Donchin, 2001) and independent component analysis (ICA; Onton, Westerfield, Townsend, & Makeig, 2006) are among these methods. Although they are extremely different from Laplacian estimation from a mathematical point of view (ICA and PCA decomposition are mainly based on the statistical dependence between signals recorded across sensors whereas Laplacian is a sum of second spatial derivatives of the recorded signal), they aim at performing blind sources separation. For example, Roger, Be´nar, Hasbroucq, and Burle (2010) applied on the same data set both ICA (applied on monopolar data) and Laplacian transformation. These authors showed that (1) the components of interest were captured in a similar manner by ICA and Laplacian transformation and (2) the time courses of the components obtained after Laplacian transformation and ICA were strikingly similar. As these two methods rely on different mathematical principles, one can imagine that both methods adequately captured the essence of the components actually present in the data. In the present experiment, the midline configuration of electrodes used here was not very appropriate for applying ICA, whereas it allowed adequately estimation of the Laplacian using the source derivation method (Hjorth, 1975). Finally, although PCA and ICA are efficient source separation methods (Arbel & Donchin, 2009; Roger et al., 2010), the surface Laplacian method is mainly attractive because (1) one can easily understand its physical significanceFit acts as a high-pass spatial filter and its maxima and minima correspond to scalp current sources and sinks, respectivelyFand (2) Laplacian-transformed data are assumed to represent a good approximation of the corticogram (Gevins, 1989). In conclusion, thanks to Laplacian deblurring properties, it has been shown in the present study that different types of processing are performed by different structures. Among these, two are involved in subjective (internal) participants’ expectancies processing and one is involved in an objective (external) processing related to the outcome of performance evaluation. The extraction of specific characteristics of feedback for this reasoning task is performed at the level of distinct modules in the brain.
REFERENCES Angrilli, A., Penolazzi, B., Vespignani, F., De Vincenzi, M., Job, R., Ciccarelli, L., et al. (2002). Cortical brain responses to semantic incongruity and syntactic violation in Italian language: An event-related potential study. Neuroscience Letters, 322, 5–8. Arbel, Y., & Donchin, E. (2009). Parsing the componential structure of post-error ERPs: A principal component analysis of ERPs following errors. Psychophysiology, 46, 1179–1189. Babiloni, F., Cincotti, F., Carducci, F., Rossini, P. M., & Babiloni, C. (2001). Spatial enhancement of EEG data by surface Laplacien estimation: The use of magnetic resonance imaging-based head models. Clinical Neurophysiology, 112, 724–727. Campbell, K. B., Courchesne, E., & Squires, K. C. (1979). Evoked potential correlates of human information processing. Biological Psychology, 8, 45–68. Carbonnell, L., Vidal, F., Sequeira, H., & Caverni, J.-P. (2006). A reasoning bias revealed by electrodermal activity. Psychophysiology, 43, 387–393.
Caverni, J. P., & Rossi, S. (1997). A nice bit of scandal: About a disconfirmation bias in the Wason’s 2–4–6 problem. Swiss Journal of Psychology, 56, 239–242. Chwilla, D. J., & Brunia, C. H. M. (1991). Event-related potentials to different feedback stimuli. Psychophysiology, 28, 123–132. Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1991). Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. Electroencephalography and Clinical Neurophysiology, 78, 447–455. Gehring, W. J., Goss, B., Coles, M. G. H., Meyer, D. E., & Donchin, E. (1993). A neural system for error detection and compensation. Psychological Science, 4, 385–390. Gevins, A. S. (1989). Dynamic functionnal topography of cognitive tasks. Brain Topography, 2, 37–56. Gunter, T. C., Friederici, A. D., & Schriefers, H. (2000). Syntactic gender and semantic expectancy: ERPs reveal early autonomy and late interaction. Journal of Cognitive Neuroscience, 12, 556–568.
302 Hagoort, P., Brown, C., & Groothusen, J. (1993). The syntactic positive shift (SPS) as an ERP measure of syntactic processing. Language and Cognitive Processes, 8, 439–483. Hjorth, B. (1975). An on-line transformation of EEG scalp potentials into orthogonal source derivations. Electroencephalography and Clinical Neurophysiology, 39, 526–530. Holroyd, C. B., & Coles, M. G. H. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the errorrelated negativity. Psychological Review, 109, 679–709. Holroyd, C. B., Pakzad-Vaezi, K. L., & Krigolson, O. E. (2008). The feedback correct-related positivity: Sensitivity of the event-related potential to unexpected positive feedback. Psychophysiology, 45, 688– 697. Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference and consciousness. Cambridge, UK: Cambridge University Press. Katznelson, R. D. (1981). EEG recording electrode placement, and aspects of generator localization. In P. Nunez (Ed.), Electric fields of the brain (pp. 176–213). New York: Oxford University Press. Kutas, M., & Hillyard, S. A. (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207, 203–205. Law, S. K., Rohrbaugh, J. W., Adams, C. M., & Eckardt, M. J. (1993). Improving spatial and temporal resolution in evoked EEG responses using surface Laplacians. Electroencephalography and Clinical Neurophysiology, 88, 309–322. Lim, C. L., Gordon, E., Rennie, C., Wright, J. J., Bahramali, H., Li, W. M., et al. (1999). Dynamics of SCR, EEG, and ERP activity in an oddball paradigm with short interstimulus intervals. Psychophysiology, 36, 543–551. Lim, C. L., Rennie, C., Barry, R. J., Bahramali, H., Lazzaro, I., Manor, B., et al. (1997). Decomposing skin conductance into tonic and phasic components. International Journal of Psychophysiology, 25, 97–109. MacKay, D. M. (1983). On-line source density computation with a minimum of electrodes. Electroencephalography and Clinical Neurophysiology, 56, 696–698. Miltner, W. H. R., Braun, C. H., & Coles, M. G. H. (1997). Eventrelated brain potentials following incorrect feedback in a time-estimation task: Evidence for a ‘‘generic’’ neural system for error detection. Journal of Cognitive Neuroscience, 9, 788–798. Mu¨ller, S. V., Mo¨ller, J., Rodriguez-Fornells, A., & Mu¨nte, T. F. (2005). Brain potentials related to self-generated and external information used for performance monitoring. Clinical Neurophysiology, 116, 63– 74. Nakagome, K., Takazawa, S., Kanno, O., Hagiwara, H., Nakajima, H., Itoh, K., et al. (2001). A topographical study of ERP correlates of semantic and syntactic violations in the Japanese language using multichannel EEG system. Psychophysiology, 38, 304–315. Oliveira, F. T. P., McDonald, J. J., & Goodman, D. (2007). Performance monitoring in the anterior cingulate is not all error related: Expectancy deviation and the representation of action-outcome associations. Journal of Cognitive Neuroscience, 19, 1994–2004. Onton, J., Westerfield, M., Townsend, J., & Makeig, S. (2006). Imaging human EEG dynamics using independent component analysis. Neuroscience and Biobehavioral Reviews, 30, 808–822. Osterhout, L., & Holcomb, P. J. (1992). Event-related potentials elicited by syntactic anomaly. Journal of Memory and Language, 31, 785–806.
L. Carbonnell et al. Osterhout, L., & Holcomb, P. J. (1993). Event-related potential and syntactic anomaly: Evidence of anomaly detection during the perception of continuous speech. Language and Cognitive Processes, 8, 413–437. Papo, D., Baudonnie`re, P.-M., Hugueville, L., & Caverni, J.-P. (2003). Feedback in hypothesis testing: An ERP study. Journal of Cognitive Neuroscience, 15, 508–522. Patel, A. D., Gibson, E., Ratner, J., Besson, M., & Holcomb, P. J. (1998). Processing syntactic relations in language and music: An event-related potential study. Journal of Cognitive Neuroscience, 10, 717–733. Pernier, J., Perrin, F., & Bertrand, O. (1988). Scalp current density fields: Concept and properties. Electroencephalography and Clinical Neurophysiology, 69, 385–389. Perrin, F., Bertrand, O., & Pernier, J. (1987). Scalp current density mapping: Value and estimation from potential data. IEEE Transaction on Biomedical Engineering, 34, 283–288. Roger, C., Be´nar, C. G., Vidal, F., Hasbroucq, T., & Burle, B. (2010). Rostral Cingulate Zone and correct response monitoring: ICA and source localization evidences for the unicity of correct- and errornegativities. NeuroImage, 51, 391–403. Semlitsch, H. V., Anderer, P., Schuster, P., & Presslich, O. (1986). A solution for reliable and valid reduction of ocular artefacts applied to the P300 ERP. Psychophysiology, 23, 695–703. Siddle, D. A. (1985). Effects of stimulus omission and stimulus change on dishabituation of the skin conductance response. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11, 206–216. Simons, R. F. (2010). The way of our errors: Theme and variations. Psychophysiology, 47, 1–14. Simons, F., Rockstroh, B., Elbert, T., Fiorito, E., Lutzenber, W., & Birbaumer, N. (1987). Evocation and habituation of autonomic and event-related potential responses in a nonsignal environment. Journal of Psychophysiology, 1, 45–59. Spencer, K. M., Dien, J., & Donchin, E. (2001). Spatiotemporal analysis of the late ERP responses to deviant stimuli. Psychophysiology, 38, 343–358. Stuss, D. T., & Picton, T. W. (1978). Neurophysiological correlates of human concept formation. Behavioral Biology, 23, 135–162. Tandonnet, C., Burle, B., Hasbroucq, T., & Vidal, F. (2005). Spatial enhancement of EEG traces by surface Laplacian estimation: Comparison between local and global methods. Clinical Neurophysiology, 116, 18–24. Wetherick, N. E. (1962). Eliminative and enumerative behavior in a conceptual task. Quarterly Journal of Experimental Psychology, 14, 246–249. Winer, B. J. (1970). Statistical principles in experimental design. London: McGraw Hill. Zanolie, K., Teng, S., Donohue, S. E., van Duijvenvoorde, A. C. K., Band, G. P. H., Rombouts, S. A. R. B., et al. (2008). Switching between colors and shapes on the basis of positive and negative feedback: An fMRI and EEG study on feedback-based learning. Cortex, 44, 537–547.
(Received June 15, 2009; Accepted April 7, 2010)
Psychophysiology, 48 (2011), 303–311. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01077.x
Executive control and response expectancy: A Laplacian ERP study
CEDRIC MECKLER,a SONIA ALLAIN,a LAURENCE CARBONNELL,a THIERRY HASBROUCQ,b BORIS BURLE,b and FRANCK VIDALa,b a
Institut de Recherche Biome´dicale des Arme´esFAntenne de ToulonFIMNSSA, Toulon, France Laboratoire de Neurobiologie de la Cognition, Aix-Marseille Universite´, CNRS, Marseille, France
b
Abstract The event-related potential called ‘‘Error Negativity’’ (Ne, ERN), which appears when subjects commit errors in choice reaction time tasks, is a marker of response monitoring. By introducing a response probability bias, we show that the Ne is sensitive to response expectancy. We further show that the small negativity evoked by correct responses (Ne-like, CRN) is also sensitive to response expectancy: On unexpected responses, the former decreases while the latter increases to such an extent that the amplitudes of the two components are in the same range of magnitude. Although the sensitivity of the Ne to response expectancy is compatible with the current models accounting for the Ne, the common sensitivity of the Ne and the Ne-like supports the idea that they reflect functionally similar monitoring processes. Descriptors: Error negativity, ERN, CRN, Laplacian, Reaction time, Response probability
anterior cingulate cortex and/or supplementary motor area sources, also termed rostral cingulate zone (RCZ) by Ridderinkhof, Ullsperger, Crone, and Nieuwenhuis (2004), following a suggestion of Picard and Strick (1996). Most of current models of the Ne would predict that its amplitude, compared to a classical equiprobable RT situation, would be decreased when the required response was unexpected, i.e., when the expected response is given by error. Indeed, in the conflict model (Botvinick, Braver, Carter, Barch, & Cohen, 2001; Yeung, Botvinick, & Cohen, 2004), the Ne amplitude represents the level of conflict between alternative responses. Conflict is defined as the simultaneous activation of mutually inhibiting alternative responses. Because of the probability bias, the expected response is the most prepared one (Rosenbaum & Kornblum, 1982) and as such becomes prepotent. If the unexpected response is required, the erroneous (prepotent) response is often activated before response signal onset. When the erroneous response is produced, the prepotent response is produced first and the correct (unprepared) response is weakly activated. Therefore, the level of co-activation between the correct and the erroneous response is low. Thus, according to the conflict model, the Ne should be smaller when the required response is unexpected. In the mismatch model (Coles, Scheffers, & Holroyd, 2001; Falkenstein, Hoormann, Christ, & Hohnsbein, 2000), Ne amplitude reflects the level of discordance between the representation of the required response and that of the actually emitted one. Response expectancy strengthens the representation of the expected response while it weakens that of the unexpected one. When the unexpected response is required but the erroneous one is produced, the mismatch between the weak representation of
Executive control processes in general, and in particular response monitoring, are implemented to allow one to adapt to a given context. Such an adaptation builds on knowledge relative to this context. On the basis of this knowledge, it is possible to develop expectancies. The response monitoring system can take advantage of these expectancies to fulfill its executive function. Experimentally, in choice reaction time (RT) tasks, response expectancy can be manipulated via a probability bias, i.e., through the relative probability of occurrence of each possible response. In a between-hand choice, for example, if one motor response is more frequently required than the other one, the preparation of this most probable response will be favored by the subject (Rosenbaum & Kornblum, 1982). As a marker of response monitoring, the event-related potential called ‘‘Error Negativity’’ (Ne: Falkenstein, Hohnsbein, Hoormann, & Blanke, 1991) or ‘‘Error-Related Negativity’’ (ERN: Gehring, Goss, Coles, Meyer, & Donchin, 1993) should be sensitive to response expectancy. This negative wave appears when subjects commit errors in choice RT tasks. It begins shortly after electromyogram (EMG) onset, precedes the mechanical response (Allain, Carbonnell, Falkenstein, Burle, & Vidal, 2004), and peaks about 100 to 150 ms after EMG onset. Its topography takes the form of a narrow frontocentral focus compatible with
We thank Dominique Reybaud and Bruno Schmid for their helpful technical assistance. Address correspondence to: Ce´dric Meckler, Institut de Recherche Biome´dicale des Arme´esFAntenne de ToulonFIMNSSA, Boulevard Saint-Anne BP 548, 83041 Toulon Cedex 09, France. E-mail: cyrilelie@ gmail.com 303
304 the unexpected (required) response and the erroneous given response should be lower and thus the Ne smaller. In the reinforcement learning model (Holroyd & Coles, 2002), the Ne arises when an event is worse than expected. Giving an erroneous response when the required response is unexpected is probably not as bad as when the required response is more expected, and the Ne should be smaller. In accordance with these models, if the expected response is produced while the unexpected response is required (error commission), we predict that the Ne will be decreased, compared to an equiprobable RT situation. On correct trials, a negativity, in the same latency range, with a similar topography, and of equal amplitude as that of the Ne observed for errors, was evidenced in schizophrenic patients and called ‘‘Correct Response Negativity’’ or CRN (Ford, 1999). Thanks to Laplacian transformation, Vidal, Hasbroucq, Grapperon, and Bonnet (2000) showed that the CRN is also present in healthy subjects, but with lower amplitude than the Ne on errors, and referred to it as ‘‘Ne-like.’’ Indeed, this small Nelike in healthy subjects is often blurred in monopolar recordings because it is overlapped by other components, due to volume conduction (Nun˜ez, 1981). Acting as a high-pass spatial filter, the Laplacian transformation minimizes these effects (Babiloni, Cincotti, Carducci, Rossini, & Babiloni, 2001): It removes the blurring effect of the diffusion of the currents through the highly resistive skull (Katznelson, 1981) by reducing the contribution of remote sources to local recordings. Laplacian transformation is often considered as a good approximation of the corticogram (Gevins, 1989). Contrary to the Ne on errors, the sensitivity of Ne-like to response expectancy is hard to predict. Indeed, models cannot easily account for the development of an Ne-like because they have not been designed to this aim. While some neurological or psychiatric diseases affect the amplitude of the Ne-like (Ford, 1999; Gehring & Knight, 2000; Hajcak & Simons, 2002; Kim, Kang, Shin, Yoo, Kim, & Kwon, 2006; Mathalon, Bennett, Askari, Gray, Rosenbloom, & Ford, 2003; Mathalon, Fedor, Faustman, Gray, Askari, & Ford, 2002), few experimental factors have been reported to affect Ne-like amplitude. And yet, like the Ne, the Ne-like is sensitive to spontaneous variations in performance (Allain, Carbonnell, Falkenstein, et al., 2004), suggesting that both of these post-response negativities are involved in response monitoring (Allain, Carbonnell, Falkenstein, et al., 2004; Bartholow, Pearson, Dickter, Sher, Fabiani, & Gratton, 2005; Suchan, Jokisch, Skotara, & Daum, 2007). If, like the Ne, the Ne-like reflects response monitoring processes, it should be, like the Ne, sensitive to response expectancy. Therefore, it seems worth studying the sensitivity of the Ne-like to response expectancy as well as that of the Ne. In the present study, we introduced a probability bias in a between-hand choice RT task. Depending on the conditions, the two possible responses could be equiprobable, or one response could be required more often than the other one. We then examined the sensitivity to response frequency of the Ne on errors (ERN) and of the Ne-like on correct responses (CRN).
Materials and Method Subjects Twelve healthy subjects (5 women, 7 men; mean age 34.3, range: 26–49) volunteered for the experiment. All were right-handed
C. Meckler et al. with normal or corrected-to-normal vision. They gave written informed consent according to the Declaration of Helsinki. Task Subjects were comfortably seated in a Faraday cage, within a sound attenuated room. Subjects were performing a betweenhand choice RT task: They had to produce a left or right button press on a response pad (NeuroScan, Compumedics, Charlotte, NC), with their right or left thumb, respectively, as fast and accurately as possible after a response signal (RS). RSs were even or odd numbers; 2, 3, 8, 9 (Stim system of NeuroScan; total of visual angle: 1.61), displayed in the center of a faradized video monitor. The trial began with a white fixation cross in the center of the screen. This cross was turned off when the RS was presented. When subjects gave their response, the RS was turned off and the next one was presented 1300 ms after the previous RS. Between the response and the next RS, the fixation cross was presented again during the interval separating subjects’ response and the presentation of the next response signal. If subjects did not respond within 820 ms, the RS was turned off and the next one was presented. RTs longer than 800 ms (exactly 820 ms for hardware reasons) were discarded, and considered as omissions. Half of the subjects had to respond to even and odd numbers by a right or a left button press, respectively. The other half performed the opposite stimulus-response mapping. We used two types of blocks. In the first one (3 blocks of 240 trials), right and left responses were equiprobable. This condition was called ‘‘Equal expectation condition.’’ In the second type of blocks (6 blocks), subjects were informed that there was a probability bias. Three blocks were performed with an 80% probability of responding right (vs. 20% left), and three others with an 80% probability of responding left (vs. 20% right). Trials when subjects had to provide the most probable response were reassembled in a condition called ‘‘Expected condition,’’ and trials when they had to provide the less probable response in a condition called ‘‘Unexpected condition.’’ Therefore, each subject performed 9 blocks of 240 trials each: 3 in the Equal expectation condition, 6 in the biased conditions. Three training blocks (120 trials each) were presented. Order of blocks was counterbalanced across subjects. Between each block, subjects could take a break at their convenience. EMG and Behavioral Data Processing EMG was recorded from the flexor pollicis brevis of each hand, by paired surface Ag/AgCl electrodes, amplified (30,000 times), filtered (5 Hz–1500 Hz, 12 db/octave), full wave rectified and integrated (integration window 5 ms) and then digitized (sampling rate: 1024 Hz). A correct response was defined as a button press occurring first on the required side. An error was defined as a button press occurring first on the non-required side. Trials in which a subthreshold EMG activation on the incorrect side preceded the execution of the correct response were excluded from the analysis. Indeed, these trials which are not purely correct are interpreted as ‘‘partial errors’’ (Scheffers, Coles, Bernstein, Gehring, & Donchin, 1996; Burle, Possamaı¨ , Vidal, Bonnet, & Hasbroucq, 2002), and, as such, reflect a different processing compared to the pure correct trials. For example, mean RT of correct responses including partial errors are longer than that of pure correct responses, and are not affected in a same manner by Stimulus-Response compatibility manipulation, such as the
Executive control and response expectancy
305
so-called ‘‘Simon effect’’ (Hasbroucq, Possamaı¨ , Bonnet, & Vidal, 1999). These double muscular activation trials can represent incorrect activations which would be corrected in time, as a result of two successive processes (Burle et al., 2002). RT was measured from RS to mechanical response (button press). In order to separate central from peripheral processes durations, we split the RT into two components: The premotor time (PMT) from response signal to EMG onset, and the motor time (MT) from EMG onset to mechanical response. EMG onset represents the time-lock for electroencephalogram (EEG) averages. It is generally considered that PMT reflects the duration of central processes, whereas MT reflects the duration of peripheral motor execution. Electrophysiological Recordings and Data Processing EEG was recorded continuously from 28 Ag/AgCl scalp electrodes (AFz, Fz, FCz, Cz, CPz, Pz, C1, C2, C3, C4, C5, C6, F1, F2, F3, F4, FC1, FC2, FC3, FC4, FC5, FC6, CP1, CP2, CP3, CP4, CP5, CP6). Reference and ground were on the right and left mastoids, respectively. Vertical and horizontal electrooculograms (EOG) were recorded between two electrodes: Above and below the right eye, and at the outer canthi, respectively. Impedances were kept below 5 kO. EEG and EOG signals were fed into Nicolet amplifiers, amplified (30,000 times for EEG, 12,000 times for EOG), filtered (0.016–100 Hz, 12 db/octave), and digitized (sampling rate: 1024 Hz). Ocular artefacts were subtracted by a statistical method (Gratton, Coles, & Donchin, 1983). After visual inspection, if the subtraction was judged unsatisfactory, the corresponding trials were rejected. Other artefacts were also rejected by visual inspection. EEG was averaged time-locked to EMG onset, detected by visual inspection of each trial (see Hasbroucq et al., 1999). The Laplacian transformation was applied on the monopolar averages, after spherical spline interpolation, with 3 as the degree of spline and a maximum of 15 degrees for the Legendre polynomial (Perrin, Bertrand, & Pernier, 1987). Based on the Laplacian transformed grand averages obtained at FCz, we determined the latency of the negative peak of the Ne and that of the positive peak preceding it, which was taken as representing the onset of the Ne. We then defined two time windows of 40 ms, each centred on the mean latency of each peak (the positive and the negative ones). For each subject in each condition, we measured the peak-to-peak amplitude which is thus baseline-free (Allain, Carbonnell, Falkenstein, et al., 2004), defined as the difference between the surfaces under the curve for each time window. On correct trials, the following wave, known as N300 (Vidal et al., 2000), was measured with the same procedure.
Some values being based on few trials, normality cannot be warranted. Non-parametric tests (Wilcoxon and Friedman) were used. Topographic details were presented in maps, computed on the peak of each activity in each condition, each map at its own scale.
Results Behavioral Data Data are reported in Table 1. Mean RTs, mean MTs, and mean PMTs for pure correct responses were submitted to repeated measures analysis of variance (ANOVA) with Hand (Right, Left) and Condition (Expected, Unexpected, Equal expectation) as the within-subjects variables. For RTs, the analysis revealed a main effect of the hand (F(1,11) 5 7,5; po.05: Right hand 372 ms, Left hand 382 ms), a main effect of the condition (F(2,22) 5 67.9, po.001), no interaction between these two factors (Fo1). Contrasts showed that each condition differed from the two others; Equal expectation (397 ms) vs. Expected (318 ms) (F(1,11) 5 86.87, po.001), Equal expectation vs. Unexpected (416 ms) (F(1,11) 5 6.52, po.05). Similar results were obtained for PMTs, except that the effect of Hand was not significant (Fo1), but a main effect of Condition (F(2,22) 5 86.5, po.001), and no interaction between these two factors (Fo1) showed-up. Contrasts confirmed that each condition differed from the two others; between Equal expectation (310 ms) and Expected (232 ms) (F(1,11) 5 93.24, po.001), between Equal expectation and Unexpected (330 ms) (F(1,11) 5 15.19, po.01). For MTs, the analysis revealed a main effect of Hand (F(1,11) 5 16.89; po.01: Right hand 81 ms, Left hand 91 ms), no effect of Condition (Fo1), and no interaction between these two factors (F(2,22) 5 2.63, p 5 .09). In summary, response expectancy only affected PMTand the main effect of Hand on RT was only due to a difference on MT, likely due to a difference of stiffness of the response pad buttons. Because of the absence of errors in the Expected condition for two subjects, mean RTs, mean MTs, and mean PMTs for errors were compared only for the two remaining conditions (Equal expectation and Expected). The low number of errors in these two conditions did not allow separating the hands. Two-tailed Student’s t-test showed an effect of Condition on RT of errors: Equal expectation (351 ms) vs. Unexpected (269 ms) (t(11) 5 11.73, po.001), and on PMT of errors: Equal expectation (260 ms) vs. Unexpected (181 ms) (t(11) 5 12.69, po.001), and no effect on MT of errors (t(11) 5 .57, p 5 .58). Arcsine transformation was applied on error rate square roots (Winer, 1970). The results were then submitted to a repeated measures ANOVA with the Hand and Condition as variables.
Table 1. Table of Behavioral Data Equal expectation condition
Mean number of trials across subjects Range Mean PMTs (ms) Mean errors rates
Expected condition
Unexpected condition
Errors
Pure corrects
Errors
Pure corrects
Errors
Pure corrects
45.25 19–71
447 79–567 310
11.08 0–43
873 600–1008 232
76.7 3–144
113.75 41–153 330
8.6
1.3
35.8
Note: Mean number of trials across subjects and range, values for PMTs, and error percentages as a function of the three conditions.
306 There was no Hand effect (F(1,11) 5 2.87, p 5 .12), an effect of Condition (F(2,22) 5 190.97, po.001) and no Hand ! Condition interaction (Fo1). Contrasts showed that each condition differed from the two others; Equal expectation (8.6%) vs. Expected (1.3%) (F(1,11) 5 93.81, po.001), Equal expectation vs. Unexpected (35.8%) (F(1,11) 5 178, po.001). Electrophysiological Data In the Expected condition, two subjects did not produce any error, and they were very rare for several of the 10 others. Errors were therefore not analyzed in this condition. The surface under rectified EMG curves was measured from 30 to 100 ms (Allain Carbonnell, Burle, Hasbroucq, & Vidal, 2004) and submitted to a one-tailed Wilcoxon matched-pairs signed-ranked test. EMG activity was reduced on erroneous compared to correct trials in both the Unexpected (T(12) 5 5, po.005) and the Equal expectation (T(12) 5 5, po.005) conditions. Similarly, EMG bursts for pure correct responses were compared in all three conditions. A two-tailed Friedman test revealed an effect of Condition on EMG burst on correct responses (w2 5 6.16 (df 5 2), po.05). Two-tailed Wilcoxon matched-pairs tests showed that EMG activity was larger for Unexpected condition compared to Equal expectation condition (T(12) 5 14, po.05) and compared to Expected condition (T(12) 5 9, po.05). As expected, monopolar recordings (Figure 1A) show an Ne in Equal expectation and Unexpected conditions. In contrast, no Ne-like was clearly identifiable in all three conditions. This is especially clear in the Unexpected condition in which there was no ‘‘wave’’ but rather a negative slope followed by a plateau preceding a second negative slope. In the 25-ms time-window when each slope was the steepest on grand averages, the negative and positive slopes of Ne-like were compared to zero (Tandonnet, Burle, Vidal, & Hasbroucq, 2003; Vidal, Grapperon, Bonnet, & Hasbroucq, 2003) by one-tailed Wilcoxon tests. This 25ms period was chosen since in the Unexpected condition, the ‘‘plateau’’ evoked by pure correct responses lasted about 25 ms. These tests showed that there was no reliable positive slope in the Unexpected condition (not different from zero) (Table 2). Thus, in this condition, there was no reliable ‘‘wave.’’ As a consequence, it was not possible to identify a reliable negative peak, and, as a result, to measure the peak-to-peak amplitude: The Nelike could not be analyzed in the Unexpected condition. Therefore, we analyzed only the Ne in the (i) Equal expectation condition, (ii) Unexpected condition, and the Ne-like in the (iii) Equal expectation condition, (iv) Expected condition. Two-tailed Wilcoxon matched-pairs tests confirmed that the Ne-like was smaller than the Ne in Equal expectation condition (T(12) 5 0, po.005). The Ne was smaller in Expected condition compared to Equal expectation condition (T(12) 5 14, po.05). The Ne-like was not different between Equal expectation condition and Expected condition (T(12) 5 22, p 5 .18). Monopolar maps suggest that the Ne were rather focused on FCz electrode, whereas the Ne-like were more widely distributed over the frontal areas. In contrast, on Laplacian data, both the Ne and Ne-like were maximal at fronto-central electrode FCz, and peaked around 150 ms after EMG onset. Maps of Figure 1B show that topographies of Ne and Ne-like were similar across all conditions. As above, in the 25-ms time-window when each slope was the steepest on grand averages, the negative and positive slopes of
C. Meckler et al. Ne-like were compared to zero by one-tailed Wilcoxon tests: All the Ne and Ne-like were reliably identified in Laplacian transformations (Table 2). This allowed analyzing the Ne-like in the Unexpected condition. Figure 1B suggests that the effect of the nature (correct or error) of the response depends on the condition. We tested this interaction by comparing the difference (between conditions) of the differences (between correct and errors) in post-response negativities amplitudes, by a two-tailed Wilcoxon matched-pairs signed-ranked test. This interaction was significant: T(12) 5 7, po.01 and indicated that the Ne was larger than the Ne-like in Equal expectation condition T(12) 5 5, po.01, whereas the Ne and the Ne-like were not statistically different in the Unexpected condition (T(12) 5 19, ns). The Ne was smaller in Unexpected compared to Equal expectation condition: T(12) 5 14, po.05. For the Ne-like, the three conditions were available, and a two-tailed Friedman test was performed which revealed an effect of Condition (w2 5 6 (df 5 2), po.05). The Ne-like was larger in Unexpected condition compared to both Equal expectation (T(12) 5 10, po.02) and Expected conditions (T(12) 5 7, po.01) but not statistically different in Equal expectation and Expected conditions (T(12) 5 21, ns). Finally, on correct trials, a smaller negative peak (N300) was present after the Ne-like on correct responses only, which is coherent with previous reports (Bartholow et al., 2005, Vidal et al., 2000). This N300 is, as the Ne and the Ne-like, also interpreted as belonging to monitoring processes (Bartholow et al., 2005). As previously, we performed a two-tailed Friedman test, which shows no effect of Condition on the N300 amplitude (w2 5 2.16 (df 5 2), ns).
Discussion PMT but not MT was affected by response expectancy, indicating that it had an effect on the central processes: The more the response was expected, the shorter the PMT, and the lower the error rate, which confirms that subjects did expect and prepare the most probable response. As expected from the literature, the probability bias favored the preparation of the most probable response at the expense of the other one (Rosenbaum & Kornblum, 1982; Vidal, Bonnet, & Macar, 1995); Preparation thus induced a prepotent response activation and a stronger response representation. Before discussing further the effect of response expectancy on the Ne and the Ne-like in the present study, some methodological comments are in order. No clear Ne-like can be identified on monopolar data (Allain, Carbonnell, Falkenstein, et al., 2004; Ridderinkhof, Nieuwenhuis, & Bashore, 2003), which is likely due to mixing effects induced by volume conduction (Roger, Be´nar, Vidal, Hasbroucq, & Burle, 2010). As a matter of fact, Luu and Tucker (2001) noticed that, since because most RTs fall in the P300 latency range, response-related events are likely overlapped by the P300. Being of large amplitude, the Ne on error emerges nonetheless. However, the Ne-like being smaller, it is largely hidden by the superimposed P300, or weakly apparent at frontal sites only, where the P300 is minimal. If the overlapping components (e.g., the P300) are generated remotely from the RCZ, the spatial filtering properties of the Laplacian transformation will (at least) greatly attenuate their impact at the sites of interest
Executive control and response expectancy
307
Figure 1A. Top: Averaged EMG-locked monopolar EEG activities at FCz (ordinate in mV) as a function of time (abscissa in ms, zero of time at EMG onset), for errors (dotted lines) and pure correct responses (solid lines), for each condition. Bottom: Monopolar maps (mV) computed at the time of the peak of each Ne/Ne-like (ERN/CRN), each map at its own scale. Map for pure correct responses in the Unexpected condition is missing because of the absence of a reliable peak.
Figure 1B. Top: Averaged EMG-locked Laplacian transformed EEG activities at FCz (ordinate in mV/cm2), as a function of time (abscissa in ms, zero of time at EMG onset), for errors (dotted lines) and pure correct responses (solid lines), for each condition. Bottom: Laplacian maps (mV/cm2) computed at the time of the peak of each Ne/Ne-like (ERN/CRN), each map at its own scale.
308
C. Meckler et al.
Table 2. Negative and Positive Slopes of the Ne-like for Each Condition Negative slope
Laplacian
Positive slope
Condition
Equal
Expected
Unexpected
Equal
Expected
Unexpected
Window T p
75–100 0 0.001
70–95 0 0.001
80–105 0 0.001
155–180 5 0.004
145–170 0 0.001
155–180 13 0.02
Negative slope
Monopolar
Positive slope
Condition
Equal
Expected
Unexpected
Equal
Expected
Unexpected
Window T p
75–100 7 0.006
75–100 10 0.01
80–105 4 0.003
140–165 17 0.04
145–170 6 0.005
170–195 34 ns
Note: Negative and positive slopes corresponding to the 25-ms time-window when each grand averaged slope was the steepest. Each slope was compared to zero with a one-tailed Wilcoxon test. ns 5 not significant.
(FCz). This is exemplified in the present data by the comparison between monopolar (Figure 1A) and Laplacian (Figure 1B) maps. Regarding the Ne, the main difference between monopolar and Laplacian maps relates to the extent of their spatial distribution: the monopolar distribution is much broader than the Laplacian one; They are, however, qualitatively similar. Let us consider, now, the Ne-like. Its Laplacian distribution is tightly focused around FCz, as for the Ne. On the contrary, its broad monopolar distribution is clearly confined to the frontal sites giving the misleading impression that, because the Ne and the Ne-like present different scalp topographies, they correspond to different generators. By removing (filtering) the influence of a large posterior positive component, the Laplacian transformation unmasks an underlying Ne-like, whose actual distribution is, in fact, very similar to that of the Ne, reflecting the same generators (Roger et al., 2010). Although the Ne emerges more easily from overlapping components on monopolar data, this does not mean that the Ne is immune from overlapping effects. For example, Hajcak, Vidal, and Simons (2004) have shown that, depending on the RT of errors, the apparent amplitude of the Ne could be affected by stimulus-locked components. However, as argued earlier, because of the spatial high-pass filtering properties of the Laplacian operator, Laplacian transformed data are theoretically less affected by remote sources and hence less prone to overlapping effects. This has been empirically evaluated by Vidal, Burle Bonnet, Grapperon, and Hasbroucq (2003) who showed that, although the contamination of correct response-locked monopolar data by stimulus-locked activities could depend on RTs (Coles et al., 2001), Laplacian-transformed Ne-like was not significantly contamined by stimulus-locked activities. Moreover, after splitting the data into fast vs. slow RT, the authors showed that the amplitude of the Laplacian transformed Ne-like was unaffected by the RT. Simulation studies have suggested that the Laplacian transformation is less sensitive to deep sources (Pernier, Perrin, & Bertrand, 1988), which, in some cases, may constitute a disadvantage. Therefore, one could legitimately wonder whether Laplacian transformed data can capture the activity of the RCZ (fairly deep), which is assumed to be the source of both the Ne and the Ne-like (Roger et al., 2010). More generally, one may wonder whether the spatial filtering properties of the Laplacian transformation erase some important features of the Ne and/or Ne-like.
Recently, Roger et al. (2010) have shown that a decomposition using independent component analysis (ICA) (Onton, Westerfield, Townsend, & Makeig, 2006) captures both the Ne and the Ne-like in the same component, corresponding to a dipolar source in the RCZ. It is worth noticing that ICA was applied on monopolar data and, as such, it was not liable to the drawback described above about Laplacian. Interestingly, the ICA and Laplacian transformed data were remarkably similar, suggesting that both methods extracted the same activity (see Figure 4 in Roger et al., 2010). This indicates that Laplacian can, at least as efficiently as ICA, recover the activity of the RCZ. We now turn back to the effect of the manipulated factor. In the ‘‘Equal expectation’’ condition (both responses were equiprobable), negative waves were observed for both erroneous (Ne, ERN), and correct responses (Ne-like, CRN), with similar topographies (fronto-central), the Ne-like being smaller than the Ne (Figure 1A and 1B), which reproduces earlier results obtained in the literature on monopolar and Laplacian data (Allain, Carbonnell, Falkenstein, et al., 2004; Bartholow et al., 2005; Falkenstein, Hielscher, Dziobek, Schwarzenau, Hoormann, et al., 2001; Vidal et al., 2000). In our study on Laplacian data, Ne amplitude was decreased in the ‘‘Unexpected’’ condition as compared to the ‘‘Equal expectation’’ condition, which fits with predictions of the models briefly presented in the introductory section. This result in itself does not allow coming out for or against any particular model but remains noteworthy: So far, few experimental conditions have affected Ne amplitude in healthy subjects. One noticeable condition is the importance of the subjective value of errors since the Ne increases with the value of errors (Hajcak, Moser, Yeung, & Simons, 2005). When the Stimulus-Response (S-R) association is incompatible, the Ne amplitude decreases (Scheffers & Coles, 2000), but divergent data have been reported (Christ, Falkenstein, Heuer, & Hohnsbein, 2000; Leuthold & Sommer, 1999). Speed-accuracy trade-off constitutes a robust example of Ne amplitude sensitivity: The Ne is smaller in speeded conditions than in accurate ones (Falkenstein et al., 1991; Gehring et al., 1993). One may wonder, however, whether Ne amplitude is related to the increased error rate or to RT shortening. In the present experiment, these two factors were dissociated: Indeed, in the ‘‘Unexpected’’ condition, error rate was increased (as for the speeded condition), but RT was lengthened (contrary to the speeded condition). The Ne being increased in this condition
Executive control and response expectancy suggests that Ne amplitude is more related to the error rate than to response speed. Ne-like amplitude was similar in the ‘‘Expected’’ and ‘‘Equal expectation’’ conditions. This is in agreement with Vidal, Burle, et al. (2003, see Figure 5, p. 276) who showed that Ne-like amplitude was identical in a simple RT task (response highly expected) and a choice RT task (identical to our ‘‘Equal expectation’’ condition). In contrast, in the present study the Ne-like was larger in the ‘‘Unexpected’’ condition than in the ‘‘Expected’’ and ‘‘Equal expectation’’ conditions, in agreement with the results of Suchan and colleagues (Suchan et al., 2007; Suchan, Zoppelt, & Daum, 2003). To explain this increase, one could argue that, in the ‘‘Unexpected’’ condition, subjects misclassified some correct responses and considered them as errors. This interpretation is, however, unlikely and not supported by our data. First, an N300 is present on correct responses in all three conditions, and absent on errors1 (Bartholow et al., 2005; Vidal et al., 2000). Second, Allain, Carbonnell, Burle, et al. (2004) previously reported that EMG burst amplitude is reduced on errors as compared to correct responses. This reduction was interpreted as reflecting a desperate attempt to inhibit detected errors. In the present study, the EMG burst was also reduced on errors as compared to correct responses in the ‘‘Unexpected’’ condition (Figure 2). For the suppression process to be correctly activated, errors need to have been detected first. Therefore, on average in the ‘‘Unexpected’’ condition, subjects did not misclassify correct responses as errors. Actually, the EMG burst for pure-correct trials was even larger for the Unexpected condition as compared to the other conditions, which does not support the idea that a significant amount of false error detections have occurred in this condition. The Ne-like thus seems sensitive to the rarity of motor events. It is noteworthy that the ‘‘Unexpected’’ condition provides a so far unique circumstance in healthy subjects, where Ne-like and Ne amplitudes are so close that the usual statistical tests do not allow one to discriminate the two waves: Ne and Ne-like amplitudes are in the same order of magnitude. This suggests that they could be of the same nature and reflect functionally equivalent monitoring processes (see Simons, 2010, for a review). Besides this functional argument, Roger et al. (2010) recently showed that the Ne and the Ne-like share common generators, in line with the similar topographies observed in the present study. If the Ne and the Ne-like are of the same nature, it could seem surprising, at face value, that they are sensitive in a different way to the same experimental manipulation: When the required response was unexpected, the Ne decreased while the Ne-like increased (for the sake of simplicity, let us call them ‘‘post-response negativity’’ or PRN). This apparent paradox could be resolved if one imagines, for example, that the PRN could be sensitive to the likelihood of the association between the stimulus and the ongoing response, both for correct and erroneous responses. Such information is indeed coded in the central nervous system by neurons in the primary motor area, just after the issuance of the motor command (Zhang, Riehle, Requin, & Kornblum, 1997): For every trial, although the motor command has been sent, the motor network stores the actually executed S-R conjunction. This could account 1 Contrary to the Ne-like, the N300 was not modulated by the condition, which would suggest that, even if the N300 is related to monitoring processes (Bartholow et al., 2005), these two components reflect distinct processes.
309
Figure 2. Averaged EMG activities in arbitrary units (a.u.), time-locked to EMG onset (zero of time) as a function of time (abscissa in ms) on errors and pure correct responses, in the Unexpected condition.
for a decreased PRN on erroneous responses and an increased PRN on correct responses in the ‘‘Unexpected’’ condition as compared to the ‘‘Equal expectation’’ condition: In this situation, the correct S-R association is rarely actually produced (because of the probability bias), hence explaining the increased PRN on correct trials. On the contrary, the incorrect S-R association is often produced (because of the high error rate), hence reducing the PRN amplitude on errors. In such a framework, whatever the correctness of the response, the rarer the executed SR association, the larger the PRN. In that case, the PRN could serve as an alarm signal to alert that, up to now, the just-emitted response seldom has been associated to the stimulus. According to Zhang et al. (1997), the actually executed S-R conjunction would be useful ‘‘for the computation of reward expectancy [. . .]. It could result from a self-monitoring system that would be useful for detecting errors’’ (p. 2242). This assumption is reminiscent of the premise of the reinforcement learning models (Frank, Worotch, & Curran, 2005; Holroyd & Coles, 2002). According to this class of model, a negative (worse than expected) outcome induces a phasic increase of dopaminergic (DA) neuron activity, supposed to be at the origin of the Ne. On the contrary, a positive (better than expected) outcome is supposed to reduce the activity of DA-neurons. In their present form,2 however, these models do not seem reconcilable with the present results. Indeed, in this frame, the increased amplitude of the Ne-like in the Unexpected condition amounts to saying that the correct (unexpected) response evoked a phasic increase of DA neurons. Still in this context, producing the correct (unexpected) response would correspond to a negative outcome, which seems unlikely. The conflict model can predict the presence of a negativity on correct trials when two competing responses are co-activated. This model further predicts that the negativity is larger in the Unexpected than in the Expected condition, since the required 2 See Roger et al. (2010) for an alternative point of view on these models.
310
C. Meckler et al.
(unexpected) response is likely to compete with the (expected) prepotent response. However, according to the model, this negativity should occur before and not after the correct response (Yeung et al., 2004). As a consequence, this model cannot predict an enhancement of the Ne-like in the Unexpected condition. Finally, it is obvious that the mismatch model cannot predict the presence of an enhanced Ne-like on correct unexpected trials. Lastly, our data allow proposing an interpretation of the effects observed in some pathologies. Close amplitudes of the Ne and the Ne-like have previously been reported in prefrontal damaged patients (Gehring & Knight, 2000): ‘‘Correct-trial activity was equal to error-trial ERN activity’’ (p. 516). Analogous results have also been obtained in schizophrenic patients (Ford, 1999; Kim et al., 2006; Mathalon et al., 2002; see, however, Bates, Kiehl, Laurens, & Liddle, 2002) who are often thought to
present prefrontal impairment (Sullivan, Lim, Mathalon, Marsh, Harris, et al., 1998). In these pathologies, equality of amplitudes of the Ne and the Ne-like were obtained in equiprobable choice RT tasks (identical to our ‘‘Equal expectation’’ condition). It has been proposed that patients present a failure to distinguish correct responses from errors (Mathalon et al., 2002, p. 36). The present results suggest another kind of explanation: Even though schizophrenic and prefrontal patients were in an ‘‘Equal expectation’’ condition, their deficit made them unable to correctly handle expectations, making everything ‘‘unexpected.’’ This suggests that, contrary to healthy subjects, equiprobable responses are always unexpected for the patients or, in other words, that patients are unable to take contextual information into account. This opens new perspectives with respect to both the understanding of those pathologies and the functional interpretation of the Ne.
REFERENCES Allain, S., Carbonnell, L., Falkenstein, M., Burle, B., & Vidal, F. (2004). The modulation of the Ne-like wave on correct responses foreshadows errors. Neuroscience Letters, 372, 161–166. Allain, S., Carbonnell, L., Burle, B., Hasbroucq, T., & Vidal, F. (2004). On-line executive control: An electromyographic study. Psychophysiology, 41, 113–116. Babiloni, F., Cincotti, F., Carducci, F., Rossini, P. M., & Babiloni, C. (2001). Spatial enhancement of EEG data by surface Laplacien estimation: the use of magnetic resonance imaging-based head models. Clinical Neurophysiology, 112, 724–727. Bartholow, B. D., Pearson, M., Dickter, C., Sher, K. J., Fabiani, M., & Gratton, G. (2005). Strategic control and medial frontal negativity: Beyond errors and response conflict. Psychophysiology, 42, 33–42. Bates, A. T., Kiehl, K. A., Laurens, K. R., & Liddle, P. F. (2002). Errorrelated negativity and correct response negativity in schizophrenia. Clinical Neurophysiology, 113, 1454–1463. Botvinick, M. M., Braver, T. S., Carter, C. S., Barch, D. M., & Cohen, J. D. (2001). Conflict monitoring and cognitive control. Psychological Review, 108, 624–642. Burle, B., Possamaı¨ , C. A., Vidal, F., Bonnet, M., & Hasbroucq, T. (2002). Executive control in the Simon effect: An electromyographic and distributional analysis. Psychological Research, 66, 324–336. Christ, S., Falkenstein, M., Heuer, H., & Hohnsbein, J. (2000). Different error types and error processing in spatial stimulus-response-compatibility tasks: Behavioural and electrophysiological data. Biological Psychology, 51, 129–150. Coles, M. G., Scheffers, M. K., & Holroyd, C. B. (2001). Why is there an ERN/Ne on correct trials? Response representations, stimulusrelated components, and the theory of error-processing. Biological Psychology, 56, 173–189. Falkenstein, M., Hielscher, H., Dziobek, I., Schwarzenau, P., Hoormann, J., Sundermann, B., & Hohnsbein, J. (2001). Action monitoring, error detection, and the basal ganglia: An ERP study. NeuroReport, 12, 157–161. Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1991). Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. Electroencephalography and Clinical Neurophysiology, 78, 447–455. Falkenstein, M., Hoormann, J., Christ, S., & Hohnsbein, J. (2000). ERP components on reaction errors and their functional significance: A tutorial. Biological Psychology, 51, 87–107. Ford, J. M. (1999). Schizophrenia: The broken P300 and beyond. Psychophysiology, 36, 667–682. Frank, M. J., Worotch, B. S., & Curran, T. (2005). Error-related negativity predicts reinforcement learning and conflict biases. Neuron, 47, 495–501. Gehring, W. J., Goss, B., Coles, M. G. H., Meyer, D. E., & Donchin, E. (1993). A neural system for error detection and compensation. Psychological Science, 4, 385–390.
Gehring, W. J., & Knight, R. T. (2000). Prefrontal-cingulate interactions in action monitoring. Nature Neuroscience, 3, 516–520. Gevins, A. S. (1989). Dynamic functional topography of cognitive tasks. Brain Topography, 2, 37–56. Gratton, G., Coles, M., & Donchin, E. (1983). A new method for off-line removal of ocular artifact. Electroencephalography and Clinical Neurophysiology, 55, 468–484. Hajcak, G., Moser, J. S., Yeung, N., & Simons, R. F. (2005). On the ERN and the significance of errors. Psychophysiology, 42, 151–160. Hajcak, G., & Simons, R. F. (2002). Error-related brain activity in obsessive-compulsive undergraduates. Psychiatry Research, 110, 63–72. Hajcak, G., Vidal, F., & Simons, R. F. (2004). Difficulties with easy tasks: ERN/Ne and stimulus component overlap. In M. Ullsperger & M. Falkenstein (Eds.), Errors, conflicts, and the brain. Current opinions on performance monitoring (pp. 204–211). Leipzig: MPI of Cognitive Neuroscience. Hasbroucq, T., Possamaı¨ , C. A., Bonnet, M., & Vidal, F. (1999). Effect of the irrelevant location of the response signal on choice reaction time: An electromyographic study in humans. Psychophysiology, 36, 522–526. Holroyd, C., & Coles, M. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109, 679–709. Katznelson, R. D. (1981). EEG recording, electrode placement, and aspects of generator localization. In P. L. Nun˜ez (Ed.), Electric fields of the brain (pp. 177–213). New York: Oxford University Press. Kim, M. S., Kang, S. S., Shin, K. S., Yoo, S. Y., Kim, Y. Y., & Kwon, J. S. (2006). Neuropsychological correlates of error negativity and positivity in schizophrenia patients. Psychiatry and Clinical Neurosciences, 60, 303–311. Leuthold, H., & Sommer, W. (1999). ERP correlates of error processing in spatial S-R compatibility tasks. Clinical Neurophysiology, 110, 342–357. Luu, P., & Tucker, D. M. (2001). Regulating action: Alternative activation of midline frontal and motor cortical networks. Clinical Neurophysiology, 112, 1295–1306. Mathalon, D. H., Bennett, A., Askari, N., Gray, E. M., Rosenbloom, M. J., & Ford, J. M. (2003). Response-monitoring dysfunction in aging and Alzheimer’s disease: An event-related potential study. Neurobiology of Aging, 24, 675–685. Mathalon, D. H., Fedor, M., Faustman, W. O., Gray, M., Askari, N., & Ford, J. M. (2002). Response-monitoring dysfunction in schizophrenia: An event-related brain potential study. Journal of Abnormal Psychology, 111, 22–41. Nun˜ez, P. L. (1981). Electric fields of the brain. New York: Oxford University Press. Onton, J., Westerfield, M., Townsend, J., & Makeig, S. (2006). Imaging human EEG dynamics using independent component analysis. Neuroscience Biobehavioral Review, 30, 808–822.
Executive control and response expectancy Pernier, J., Perrin, F., & Bertrand, O. (1988). Scalp current density fields: Concepts and properties. Electroencephalography and Clinical Neurophysiology, 69, 385–389. Perrin, F., Bertrand, O., & Pernier, J. (1987). Scalp current density mapping: Value and estimation from potential data. IEEE Transaction on Biomedical Engineering, 34, 283–288. Picard, N., & Strick, P. L. (1996). Motor areas of the medial wall: A review of their location and functional activation. Cerebral Cortex, 6, 342–353. Ridderinkhof, K. R., Nieuwenhuis, S., & Bashore, T. (2003). Errors are foreshadowed in brain potentials associated with action monitoring in cingulate cortex in humans. Neuroscience Letters, 348, 1–4. Ridderinkhof, K. R., Ullsperger, M., Crone, E. A., & Nieuwenhuis, S. (2004). The role of the medial frontal cortex in cognitive control. Science, 306, 443–447. Roger, C., Be´nar, C. G., Vidal, F., Hasbroucq, T., & Burle, B. (2010). Rostral cingulate zone and correct response monitoring: ICA and source localization evidences for the unicity of correctFand errorFnegativities. NeuroImage, 51, 391–403. Rosenbaum, D. A., & Kornblum, S. (1982). A priming method for investigating the selection of motor responses. Actea Psychologica, 51, 223–243. Scheffers, M. K., Coles, M. G. H., Bernstein, P., Gehring, W. J., & Donchin, E. (1996). Event-related brain potential and error-related processing: An analysis of incorrect response to go and no-go stimuli. Psychophysiology, 33, 42–53. Scheffers, M. K., & Coles, M. G. H. (2000). Performance monitoring in a confusing world: Error-related brain activity, judgements of response accuracy, and types of errors. Journal of Experimental Psychology: Human Perception and Performance, 26, 141–151. Simons, R. F. (2010). The way of our errors: Theme and variations. Psychophysiology, 47, 1–14. Suchan, B., Jokisch, D., Skotara, N., & Daum, I. (2007). Evaluationrelated frontocentral negativity evoked by correct responses and errors. Behavioural Brain Research, 183, 206–212.
311 Suchan, B., Zoppelt, D., & Daum, I. (2003). Frontocentral negativity in electroencephalogram reflects motor response evaluation in humans on correct trials. Neuroscience Letters, 350, 101–104. Sullivan, E. V., Lim, K. O., Mathalon, D. H., Marsh, L. M. B. D., Harris, D., Hoff, A. L., et al. (1998). A profile of cortical gray matter volume deficits characteristic of schizophrenia. Cerebral Cortex, 8, 117–124. Tandonnet, C., Burle, B., Vidal, F., & Hasbroucq, T. (2003). The influence of time preparation on motor processes assessed by surface Laplacian estimation. Clinical Neurophysiology, 114, 2376–2384. Vidal, F., Bonnet, M., & Macar, M. (1995). Programming the duration of a motor sequence: Role of the primary and supplementary motor areas in man. Experimental Brain Research, 106, 339–350. Vidal, F., Burle, B., Bonnet, M., Grapperon, J., & Hasbroucq, T. (2003). Error negativity on correct trials: A reexamination of available data. Biological Psychology, 64, 265–282. Vidal, F., Grapperon, J., Bonnet, M., & Hasbroucq, T. (2003). The nature of unilateral motor commands in between-hand choice tasks as revealed by surface Laplacian estimation. Psychophysiology, 40, 796–805. Vidal, F., Hasbroucq, T., Grapperon, J., & Bonnet, M. (2000). Is the ‘error negativity’ specific to errors? Biological Psychology, 51, 109–128. Winer, B. J. (1970). Statistical principles in experimental design. London: McGraw Hill. Yeung, N., Botvinick, M. M., & Cohen, J. D. (2004). The neural basis of error detection: Conflict monitoring and the error-related negativity. Psychological Review, 111, 931–959. Zhang, J., Riehle, A., Requin, J., & Kornblum, S. (1997). Dynamics of single neuron activity in monkey primary motor cortex related to sensorimotor transformation. The Journal of Neuroscience, 17, 2227–2246. (Received December 18, 2009; Accepted May 11, 2010)
Psychophysiology, 48 (2011), 312–322. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01066.x
Shifting visual attention away from fixation is specifically associated with alpha band activity over ipsilateral parietal regions
DIEGO COSMELLI,a,b VLADIMIR LO´PEZ,a JEAN-PHILIPPE LACHAUX,c,d JAVIER LO´PEZ-CALDERO´N,e,f BERNARD RENAULT,g,h,i JACQUES MARTINERIE,g,h,i and FRANCISCO ABOITIZb,f a
Escuela de Psicologı´ a, Pontificia Universidad Cato´lica de Chile, Santiago, Chile Centro Interdisciplinario de Neurociencia, Pontifica Universidad Cato´lica de Chile, Santiago, Chile c Unite´ 280, INSERM 151, Lyon, France d Universite´ Lyon 1, Lyon, France e Center for Mind and Brain, University of California, Davis, Davis, California, USA f Laboratorio de Neurociencia Cognitiva, Departamento de Psiquiatrı´ a, Escuela de Medicina, Pontificia Universidad Cato´lica de Chile, Santiago, Chile g UPMC Universite´ Paris 06, UMR 7225, UMR-S 975, Centre de Recherche de l’Institut Cerveau-Moelle (CRICM), Paris, France h CNRS, UMR 7225, CRICM, Paris, France i INSERM, UMR-S 975, CRICM, Paris, France b
Abstract We studied brain activity during the displacement of attention in a modified visuo-spatial orienting paradigm. Using a behaviorally relevant no-shift condition as a control, we asked whether ipsi- or contralateral parietal alpha band activity is specifically related to covert shifts of attention. Cue-related event-related potentials revealed an attention directing anterior negativity (ADAN) contralateral to the shift of attention and P3 and contingent negative variation waveforms that were enhanced in both shift conditions as compared to the no-shift task. When attention was shifted away from fixation, alpha band activity over parietal regions ipsilateral to the attended hemifield was enhanced relative to the control condition, albeit with different dynamics in the upper and lower alpha subbands. Contralateral-toattended parietal alpha band activity was indistinguishable from the no-shift task. Descriptors: EEG, Alpha band, Parietal cortex, No-shift, Central cue
including both cortical and subcortical structures (Raz, 2004; Weiskrantz, Warrington, Sanders, & Marshall, 1974). At the cortical level, fronto-parietal networks are thought to play a critical role in attentional selection, and the right hemisphere is usually considered dominant (Corbetta, Miezin, Shulman, & Petersen, 1993; Corbetta & Shulman, 2002; Green & McDonald, 2008; Hopfinger, Camblin, & Parks, 2010). Traditionally, electrophysiological studies of attention orienting in humans have focused on the modulation of brain activity evoked by a stimulus that has or has not been attended (Hillyard & Kutas, 1983; Hillyard, Luck, & Mangun, 1994). When one is studying covert visuo-spatial attention shifts, the most common experimental manipulation is to instruct the subject to orient his or her attention to a cued locationFwithout moving the eyesFand then detect a relevant stimulus that may or may not appear at the cued location (Posner, 1980). However, more recently, the question about the brain mechanisms underlying the process of orienting or shifting attention, in contradistinction to its effects, has begun to be addressed in electroencephalogram (EEG) work. To do this, studies have focused on analyzing brain responses during the cue–stimulus interval in addition to that evoked by the subsequent stimulus.
Waking life is typically characterized by ongoing shifts of attention among a vast number of possible objects. Behaviorally, attending shortens reaction times, inhibits distracters, and facilitates recognition, discrimination, and recall (Kastner & Ungerleider, 2000). Neurophysiological studies have shown attention-related modulations at different levels, including enhanced baseline firing patterns (Nakamura & Colby, 2000), changes in temporal relations among neuronal populations (Fries, Reynolds, Rorie, & Desimone, 2001; Jensen, Kaiser, & Lachaux, 2007), and selective modulation of several components of the event-related potential (ERP; Hillyard & Anllo-Vento, 1998; Luck, Woodman, & Vogel, 2000). Neuroimaging and neuropsychological approaches have consistently shown that a distributed brain network is implicated in the control of attention,
The authors thank Steven Hillyard for insightful comments on a previous version of this work. This work was supported by FONDECYT projects 3060094, 1090612, and PBCT-CONICYT PSD72. Address correspondence to: Diego Cosmelli, Escuela de Psicologı´ a, Pontificia Universidad Cato´lica de Chile, Av. Vicun˜a Mackenna 4860, Macul, Santiago de Chile, Chile. E-mail:
[email protected] 312
Attention shifts and parietal alpha band activity Among these, studies of oscillatory activity during the orientation of visuo-spatial attention in the cue–stimulus interval have shown consistent changes in the alpha (8–12 Hz) band (Foxe, Simpson, & Ahlfors, 1998; Freunberger et al., 2008; Fu et al., 2001; Kelly, Lalor, Reilly, & Foxe, 2006; Rihs, Michel, & Thut, 2007, 2009; Sauseng et al., 2005; Thut, Nietzel, Brandt, & Pascual-Leone, 2006; Worden, Foxe, Wang, & Simpson, 2000; Yamagishi, Callan, Anderson, & Kawato, 2008). Because an increase in alpha rhythm has been considered indicative of reduced cortical activity (Pfurtscheller, 2001; but see Nunez, Wingeier, & Silberstein, 2001; Palva & Palva, 2007), occipito-parietal contralateral-to-cued decrease (Sauseng et al., 2005; Thut et al., 2006; Yamagishi et al., 2008) and/or ipsilateral-to-cued increase (Doesburg, Roggeveen, Kitajo, & Ward, 2008; Fu et al., 2001; Kelly et al., 2006; Rihs et al., 2007; Worden et al., 2000) in alpha power has been interpreted as reflecting facilitation of the cued (attended), or active suppression of the noncued (ignored) location, respectively. Whether an ipsilateral increase or a contralateral decrease in posterior alpha band power is specific to covert visuo-spatial shifts of attention remains an open question, although recent studies suggest that both can be elicited depending on task requirements such as the cue–stimulus interval duration (Rihs et al., 2009). Because of the way the cue–stimulus paradigm is constructed, being able to identify brain activity that is specific to the process of orienting attention and not merely related to sensory or interpretative aspects of the cue is necessary (Jongen, Smulders, & Van Der Heiden, 2007; Nobre, Sebestyen, & Miniussi, 2000; van Velzen & Eimer, 2003). Different approaches have been developed to address this question, such as using bilaterally symmetric cues (Nobre et al., 2000; van Velzen & Eimer, 2003), nonlateralized cues (Jongen et al., 2007), neutral noninformative cues (Jongen, Smulders, & van Breukelen, 2006; Luck et al., 1994; McDonald & Green, 2008; Posner, 1980; Slagter, Kok, Mol, Talsma, & Kenemans, 2005; Talsma, Slagter, Nieuwenhuis, Hage, & Kok, 2005), or including trials in which the subject has to attend to or interpret the cue but not perform any further task (Grent-’t-Jong & Woldorff, 2007). However, as used, neutral noninformative, nonlateralized, or interpret-cue controls do not target the main distinction that has to be made in order to specify brain activity engaged by displacing the focus of attention away from fixation. For nonlateralized, noninformative, or neutral cues, this is because the subsequent task is always required at the (otherwise cued) peripheral locations. As a number of studies show (de Gonzaga Gawryszewski, Riggio, Rizzolatti, & Umilta, 1987; Jongen et al., 2006; Luck et al., 1994; Yantis & Jonides, 1990), after a neutral cue, subjects will either expand their attentional focus or divide their expectations between possible target locations. As the attentional focus has been shown capable of adopting rather complicated spatial distribution patterns (Malinowski, Fuchs, & Muller, 2007; Mu¨ller, Malinowski, Gruber, & Hillyard, 2003), it seems unlikely that no attentional displacement (i.e., spatial modulation) will occur after noninformative cues with peripheral, behaviorally relevant stimuli. On the other hand, for the interpret-cue condition, a different but equally problematic situation arises: Because subjects know that no target will appear after the cue, there is no relevant task at fixation and theFcovertFdistribution of attention is left uncontrolled. In this study we took a different approach to this problem by using an informative no-shift cue to instruct the subject that a behaviorally relevant task would have to be done at fixation.
313 This way, we used a natural control as a contrast for covert shifts, namely, attending to something where one is looking. We applied this approach to ask whether alpha band activity over parietal regions ipsilateral or contralateral to the attended hemifield was specifically related to the covert displacement of visuo-spatial attention.
Methods Participants Twenty-five college student volunteers initially participated in this study. Because of excessive blinking or failure to comply with fixation during the task, 7 subjects were rejected for further analysis (see Data Preprocessing below for details). Thus, the results from 18 subjects (7 women) are reported here (mean age 5 24.7, range 5 20–31). All participants were right-handed, had normal or corrected-to-normal vision, and reported no color-vision deficiency. Participants gave informed consent and had no history of drug abuse or neurological or psychiatric conditions. The Ethics Committee of Pontificia Universidad Cato´lica de Chile approved the study, and all experiments were performed at the Cognitive Neuroscience Laboratory of the Department of Psychiatry of the University. Stimuli and Procedure A modified version of the spatial-cueing paradigm (Posner, 1980) was used in this study (see Figure 1). All stimuli were presented on a black background on a computer screen situated 57 cm away from the subject using Stim 1.0 software (Compumedics Neuroscan, Texas, USA). The fixation symbol was composed of two white small concentric circles flanked by white left- and rightward pointing arrowheads extending in total 1.51 horizontally and 0.41 vertically of visual angle. Centered at 81 to the left and right of the fixation symbol, two thin white vertical rectangles of 1 ! 21 were presented. These lateral rectangles were added to indicate the peripheral locations and avoid visual search in the time period between cue and stimulus. Both the central fixation symbol and the two lateral rectangular outlines were present throughout the entire task without interruption except between blocks (see below). Subjects maintained fixation on the central symbol throughout the entire block. Each trial began at t 5 0 (see Figure 1), and, after an intertrial interval of 600, 900, or 1200 ms, the central fixation symbol changed according to one of three possibilities: either the left- or the right-pointing arrowhead or the central concentric rings briefly changed from white to blue (150 ms), cueing the subject as to where to orient his attention covertly: left, right, or maintain center (no-shift), respectively. Each cueing direction occurred in 33% of trials in randomized order. Target or distracter stimuli (150 ms duration) were presented 1100 ms after cue onset in one of the three possible locations (center, left peripheral, or right peripheral). All stimuli (target and distracters) consisted of two small colored squares (0.61/side) presented in vertical arrangement separated by 0.61. Red (RGB: 255, 0, 0), magenta (RGB: 255, 0, 255), yellow (RGB: 235, 255, 0), and green (RGB: 155, 255, 0) colors were used, and all combinations excluding same-color arrangement were presented. In the central location, stimuli appeared above and below the fixation symbol with the same size and spacing between colored squares as the lateral stimuli. The trial ended 1100 ms after the presentation of the target/distracter stimulus,
314
D. Cosmelli et al. Behavioral Measurements Task performance was studied through reaction times (RT), hit rate (HR), and false alarm (FA) rate for all target trials. Any button press occurring prior to 200 ms or after 1100 ms posttarget was excluded from the analysis. The hit rate was defined as the percentage of correctly identified targets. The false alarm rate was defined as the percentage of distracters identified as targets (Stanislaw & Todorov, 1999).
Figure 1. Schematic depiction of experimental paradigm. Black rectangles represent screens. Horizontal arrangements represent successive views. Vertically arranged screens represent alternative views. Fixation and lateral outlines were present throughout the entire block. The fixation figure has been augmented in the first two vertical arrangements for presentation purposes.
although the central fixation symbol and the lateral rectangles remained present continuously so that the subject did not know when the next trial would commence. A total of 1,062 trials were presented to each subject in six blocks of 10.5 min each, with 5-min pauses between blocks and a longer pause halfway through the experiment. Target stimuli (11.8% of all trials) were 100% valid with respect to the cued location, and subjects were informed of this fact prior to the experiment, stressing that they should always comply with the cue direction in order to detect the corresponding target. Additionally, distracters (88.2% of all trials) were presented 50% valid and 50% invalid with respect to the cued location. When lateral shifts were cued, invalid distracters were presented at the opposite (peripheral) location and never at fixation. When no-shift was cued, invalid distracters appeared with equal probability at either left or right positions. Therefore, in order to keep a comparable number of trials across conditions, when attention was cued to stay at fixation, ERP waveforms to lateral stimuli were obtained, collapsing across all (left and right) invalid distracters. Subjects were instructed to ignore invalid distracters and maintain attention on the central fixation. Target stimuli were used only to control for task performance, and all further analysis of electrophysiological data was done on distracters. At the beginning of each block, the subject was informed about which of the four colors would be considered as the target and instructed to indicate its presence among the two colored squares by pressing a button as fast as possible. Subjects were asked to withhold response if the target color was not present at the cued position. Response hand was alternated halfway through the experiment, and the order was counterbalanced across subjects. Color combinations prescribed two possible types of blocks (i.e., difficulty levels) based on differences between the target color and the distracters: red and magenta were expected to be easier whereas yellow and green were expected to be more difficult to discriminate. Separate analysis of performance during the two types of blocks was used to rule out unspecific effects such as effort or alertness (see Discussion). The absence of differences in alpha band power between difficulty levels despite significant behavioral differences suggested that changes in oscillatory activity were not related to unspecific effects (see Supplementary Material), so we chose to collapse across levels in all further analysis.
Electrophysiological Recordings Continuous EEG recording was performed with an 80-electrode NuAmps EEG System (Compumedics Neuroscan). Subjects used a chin rest throughout the entire experiment except during pauses. All impedances were kept below 10 kO, or else the electrode was rejected. Electrode impedance was retested during pauses to assure stable values throughout the experiment. Because of hardware constraints of the setup, all electrodes were referenced to CPz during acquisition but off-line re-referenced to averaged mastoids. The signal was bandpassed filtered between 0.1 and 100 Hz prior to being digitized at 1 kHz. Diagonal electro-oculogram (EOG) was obtained with two electrodes positioned in the outer canthi of the eyes, displaced 1 cm upward on the left derivation and 1 cm downward on the right derivation (Jervis, Ifeachor, & Allen, 1988). All recordings were acquired using Scan 4.3 software (Compumedics Neuroscan) and stored for off-line treatment. Data Preprocessing Because of the importance of ensuring adequate fixation for covert visuo-spatial orienting, detection of eye movements was performed in two steps. First, an automated procedure implemented in the acquisition software was used, and any segment of the data presenting voltage deflections exceeding ! 60 mV was rejected. Full trials (" 1700 ms to 1100 ms around target/distracter presentation) were obtained from the remaining data, and only data from subjects with 60 or more trials per condition after this step were kept. This resulted in the rejection of data from 5 of the 25 initial subjects due mainly to excessive blinking. Second, the resulting set of trials were linearly detrended, using a " 1700 to " 1050-ms (" 600 to " 50 ms relative to cue presentation) baseline segment as reference for the linear trend and then visually inspected using EEGLAB v6.03b Toolbox (Delorme & Makeig, 2004). Any trials presenting further artifacts or evidences of horizontal or vertical eye movements were rejected and only data from subjects with 30 trials or more per condition after this second step were retained for further analyses, resulting in the rejection of data from 2 other subjects. After this procedure, the average number of clean trials was 97.3 for shift left (SD 5 25.4, range 46–138), 94.6 for shift right (SD 5 27.5, range 39–134), and 97.1 for no shift (SD 5 27.7, range 41–139). Statistical analysis showed that there were no significant differences in the number of kept trials between the three attentional conditions, F(2,34) 5 0.95, p 5 .4, as well as no significant pairwise differences (po.05, Bonferroni corrected comparisons, within MS 5 44.365, df 5 34). To further control for eye movements, we obtained average EOGs for all three conditions following the same processing for ERPs (see below). The average EOG waveforms for the remaining trials for all three conditions is presented in Figure 3D, below, as left–right difference and confirms that no systematic eye movements were associated with any of the three conditions.
Attention shifts and parietal alpha band activity Data Analysis All subsequent data analysis was done with in-house implemented Matlab 7.1 (The Mathworks Inc.) routines. Two sets of analysis were performed on the data: ERPs to cues and distracters and oscillatory activity in the alpha band during the cue–stimulus interval. ERPs to the cues and distracters were obtained by averaging all trials for each condition, electrode, and subject separately, and then filtering the resulting data to reject frequencies above 30 Hz. Average amplitude values for the different ERP peaks were obtained in prespecified windows for ERPs to the cue and around individual peak values for early potentials evoked by the distracters. A pilot study with 8 subjects and a passive task was run to assess the dependency of early (170–220 ms) ERPs to the cue’s physical properties. We presented the same cues as a distracter in a go/no-go task at fixation. Although the short intertrial interval we used to ensure a high number of trials precluded analysis of oscillatory activity, we observed that, despite the absence of instructions to orient attention, a contralateral occipital negativity to the lateral cues and a bilateral occipital negativity to the central cues was apparent (data not shown). This reproduces the early negative deflections seen in the present paradigm and is in line with the results of van Velzen and Eimer (2003). We therefore do not include further analysis of this time window here. Following Kiss, van Velzen, and Eimer (2008), we measured the average amplitude in a 350–650-ms postcue window to study
315 the attention-directing anterior negativity (ADAN) component, in bilateral fronto-central electrodes (see inset in Figure 3, below, for the corresponding electrode locations). Additionally, we measured a midline parieto-central broad P3-type positivity (250–400 ms postcue) and a midline fronto-central contingent negative variation (CNV; 400–50 ms prestimulus). During the poststimulus interval, we focused on the early P1 and N1 waves evoked by valid and invalid distracters. We first determined the two posterior electrodes (contralateral to attended stimulus) that showed the strongest voltage deflection for both P1 and N1 components in the average across subjects. Electrodes 22 (left occipital lateral) and 65 (right occipital lateral) were chosen (see inset in Figure 2), and a 20-ms window average was obtained from the voltage traces of these electrodes, centered on each peak and for each subject individually. Average peak latencies across subjects were 98.1 ms for P1 (SD 5 15.8 ms) and 168.1 ms for N1 (SD 5 30.2 ms). To study oscillatory activity, data were subsampled at 250 Hz prior to applying a seven-cycle Morlet wavelet transform (TallonBaudry & Bertrand, 1999) in 35 frequency steps from 4 to 80 Hz on single trials. Raw time-frequency charts for each trial were averaged across trials in order to obtain the induced power for each condition. Values were log-transformed and baseline-corrected with respect to a ! 400 to ! 50-ms window prior to cue presentation; all power values are therefore presented here as relative change with respect to this precue period. Because we were interested in discriminating posterior ipsilateral versus con-
A
B
Figure 2. P1/N1 effect. A: Scalp potentials evoked by distracters at contralateral posterior electrodes, according to the direction of attention. When attention is cued to stay at the center, waveforms are the result of averaging over left and right distracters. Inset indicates electrodes from which waveforms were obtained. B: Same as in A but indicating average amplitudes (error bars correspond to SEM) for the P1 and N1 peaks (see text for details) according to the relation between the attentional focus and the distracter’s appearance. Valid corresponds to attention and distracter at the same position (attended hemifield and distracter side coincide); invalid indicates attention at the oppositeFperipheralFside of distracter presentation (i.e., if attention is directed to the right hemifield, the distracter appears at the left hemifield). Center indicates attention at fixation and distracters appearing at lateral positions. It can be seen that maintaining attention at fixation avoids facilitation of lateral locations, thus behaving as if attention was actively shifted away.
316
D. Cosmelli et al.
A
B
C
D
Figure 3. ERP results in the cue–stimulus interval. Depicted electrodes are indicated in each panel’s inset. Gray horizontal segments in A, B, and C indicate time windows for statistical analysis. A: Attention directing anterior negativity (ADAN). B: P3-type waveform. C: Contingent negative variation. D: Average electro-oculogram (here presented as left–right electrode difference).
tralateral variations in alpha power, all statistical analysis of time-frequency data was performed on the regional average of two parietal electrodes for each hemisphere (see inset in Figure 4, below). We found important differences in the frequency location of the alpha peak for the different subjects’ spectra, so we decided to adjust for individual alpha frequency (see Supplementary Figure S1). Moreover, because upper and lower alpha subbands can react in different ways according to the task (Freunberger et al., 2008; Klimesch, 1999), we separated for each subject a lower (four frequency bins below and including the peak bin) and an upper (three frequency bins above the peak bin) alpha band. All further analyses were performed on the corresponding individually adjusted subbands separately. Statistical Design t test statistics for dependent samples were used to compare the behavioral performance between the different conditions. All statistical analyses of electrophysiological measures were performed using the individual values obtained from the electrodes/ latencies described above in a repeated measures analysis of variance (ANOVA) design. For the P1 and N1 components the factors Visual Field (two levels: left and right) and Attentional Condition (three levels: lateral valid, central invalid, and lateral invalid) were considered. For the cue-related ERP, the repeated measures ANOVA included the factors Hemisphere (two levels: right hemisphere and left hemisphere) and Cue relative direction (two levels: ipsilateral and contralateral). Oscillatory alpha-band activity during the cue–stimulus interval was statistically tested using repeated measures ANOVA for each frequency band. Within-subject factors were Cue (three levels: shift left, no shift and shift right), Hemisphere (two levels: left parietal and right parietal), and Cue-Stimulus Interval Latency (three levels: L1 through L3). In all cases Greenhouse–Geisser corrections were applied to compensate for violations of assuming sphericity, and
planned univariate comparisons were carried out when necessary (see Results section for further details). Scalp interpolation on a standard head (obtained from the EEGLAB toolbox; Delorme & Makeig, 2004) was used only for presentation purposes. Results Behavioral Performance Mean RTs to left, right, and central targets were 517 ms (SD 5 81.7 ms), 515.1 ms (SD 5 86.8 ms), and 468 ms (SD 5 74.6 ms), respectively. Separate two-tailed t tests for dependent samples revealed no significant differences in RT to left versus right visual field targets (t 5 0.33, p 5 .74) and significant differences between both left (t 5 7.57, po.05) and right (t 5 6.63, po.05) versus central targets. Mean HR to left, right, and central targets were 80.7% (SD 5 16.6%), 81.8% (SD 5 16.2%), and 94.3% (SD 5 8%), respectively. A similar pattern was obtained for HR, so that no significant difference between left and right visual fields was observed (t 5 ! 0.87, p 5 .39), whereas HR to targets in both left (t 5 ! 5.93, po.05) and right (t 5 ! 5.42, po.05) visual fields were significantly lower than HR to central targets. Mean FA rates to left, right, and central distracters were 7.7% (SD 5 9.9%), 9.7% (SD 5 10.8%), and 1.2% (SD 5 2.8%), respectively. When subject to statistical testing, FA rates followed a slightly different pattern. Although differences between lateral versus central targets behaved as in RTs and HRs (t 5 4.41 and t 5 5.16 for left and right vs. center, respectively, both po.05), we found significantly higher FAs for distracters on the right visual field when compared to the left visual field (t 5 ! 2.57, po.05). Electrophysiology ERPs to distracters. In order for the central no-shift condition to be an adequate control, it is necessary to verify that at-
Attention shifts and parietal alpha band activity
317
A
B
Figure 4. Alpha band activity during covert attention orienting according to alpha subband (higher and lower), cue (shift left, no shift, and shift right), hemisphere (left and right parietal) and latency (L1 through L3). A: Higher alpha band. B: Lower alpha band. All error bars indicate SEM. Electrodes used to obtain amplitude values are indicated in the first interpolated scalp on the upper left panel by two gray circles. Scalp interpolations indicate alpha amplitude differences between the control no-shift condition and the corresponding lateral shift according to the color bar in the upper left corner. Comparing lateral panels against the central no-shift results shows the specificity of ipsilateral-to-attended alpha band activity in both subbands.
tention is not distributed to lateral positions when the subject is instructed to perform the task at fixation. We hypothesized that if attention is maintained at fixation as a consequence of the noshift informative cue, invalid distracters will evoke early (P1/N1) brain responses comparable to when attention is directed to lateral positions and the stimulus appears at the opposite location. This is in contrast to neutral noninformative cues, where a diffuse or divided attention state is induced and therefore an intermediate response between valid and invalid conditions is observed (Hillyard et al., 1994; Jongen et al., 2006; McDonald & Green, 2008; Wright, Richard, & McDonald, 1995). Figure 2 presents the results obtained for the P1/N1 analysis on the distracters. It can be seen that when compared to valid
distracters, invalid distracters evoke a reduced P1/N1 response after lateral shifts of attention. When attention is cued to stay at fixation, peripheral distracters evoke a smaller P1/N1 response that is comparable to when attention has been shifted away. To quantify these apparent differences, the data obtained in the P1 and N1 range were subject to a separate repeated measures ANOVA with the factors Visual Field (two levels: left and right) and Attentional Condition (three levels: lateral valid, central invalid, and lateral invalid). For the P1 peak, statistical analysis revealed no main effect of Visual Field, F(1,17) 5 1.71, p 5 .21, indicating no overall differences regarding where the stimulus was presented. A main effect of Attentional Condition was found, F(2,34) 5 5.42, po.05 (see Figure 2B, left: P1 amplitude).
318 Planned comparisons between lateral valid and lateral invalid conditions revealed a significantly higher P1 amplitude for the valid condition, F(1,17) 5 8.02, po.05, and no differences between central invalid and lateral invalid conditions, F(1,17) 5 1.15, p 5 .3. The two-way Visual Field ! Attentional Condition interaction was not significant, F(2,34) 5 0.72, p 5 .49. For the N1 peak, a similar pattern emerged from the statistical analysis (see Figure 2B, right: N1 amplitude): We found no main effect of Visual Field, F(1,17) 5 1.72, p 5 .21 or two-way Visual Field ! Attentional Condition interaction, F(2,34) 5 2.88, p 5 .07. As for the P1 peak, a main effect of Attentional Condition was also detected, F(2,34) 5 12.67, po.05. Subsequent planned comparisons revealed that N1 amplitude was more negative for lateral valid than lateral invalid, F(1,17) 5 26.96, po.05, and showed no difference between central invalid and lateral invalid conditions, F(1,17) 5 1.49, p 5 .24. These results reproduce previous reports (Hillyard & AnlloVento, 1998; Luck, Heinze, Mangun, & Hillyard, 1990) regarding the effects of lateral shifts of attention, confirming that the subjects correctly oriented their attention in our experiment. More importantly, they show that when attention is cued to stay at fixation, peripheral stimuli evoke responses that are indistinguishable from those evoked when attention has been actively shifted away. The no-shift informative cue is, therefore, an adequate control condition because, when presented, it does not produce facilitation of peripheral positions. Cue-related ERPs. The results for the evoked responses in the cue–stimulus interval are summarized in Figure 3. As shown in panel 3A, we observed a clear frontal negativity contralateral to the direction of attentional orienting in the 350–650-ms postcue window, consistent with a traditional ADAN (Hopf & Mangun, 2000; Nobre et al., 2000; van Velzen & Eimer, 2003). We analyzed these apparent differences first by comparing contralateral versus ipsilateral ERP amplitudes using a repeated measures ANOVA with factors Hemisphere (two levels: right hemisphere and left hemisphere) and Cue Relative Direction (two levels: ipsilateral and contralateral). We observed a main hemisphere effect whereby the left frontal regions showed a more negative deflection than the right frontal regions, F(1,17) 5 9.88, po.05. The classical ADAN effect was observed as the main effect of cue relative directionality, F(1,17) 5 6.11, po.05, indicating that covert orienting of attention was associated with a more negative potential over contralateral frontal electrodes (corresponding to FC5/6; see inset in Figure 3A). No two-way interaction was observed, F(1,17) 5 0.03, p 5 .86. Interestingly, when we analyzed the threeFabsoluteFcueing directions (shift left, shift right, and no shift) in a separate ANOVA, we observed a main effect of cue direction, F(2,34) 5 3.63, po.05), indicating that besides being specifically contralateral to the shift direction, both left and right shifts produce a more negative deflection than the no-shift instruction. We found no evidence of a late directing attention positivity (LDAP) in our data, observing only a slight tendency to a positive deflection over the right posterior electrodes upon left shifts that did not reach significance, F(1,17) 5 1.25, p 5 .28, Univariate Test of Significance. Besides the more traditional cue-related potentials, we observed two stereotyped deflections in the cue–stimulus interval. As shown in Figure 3B, a clear P3-type positivity peaking around 300 ms after the presentation of the cue is evident. Analysis of variance of the amplitude in the 250–400-ms postcue window
D. Cosmelli et al. confirmed a clear cue direction effect, F(2,34) 5 13.82, po.05, due to a more positive amplitude during lateral shifts. A post hoc Bonferroni test (within MS 5 1.1808, df 5 34) showed that there was no difference between lateral shifts in the amplitude of this potential but confirmed the apparent difference between no shift and both left and right shifts (both po.01). A similar pattern emerged for the CNVanalysis (see Figure 3C), which showed that lateral shifts were associated with a stronger negative deflection in the 400–50-ms prestimulus interval, F(2,34) 5 6.90, po.01. A post hoc Bonferroni test again showed no differences between lateral shifts and a more negative amplitude for both left and right shifts when compared to the no-shift condition (both po.01). Oscillatory alpha-band activity during the cue–stimulus interval. We then turned to study the specificity of alpha band responses to attentional displacement by contrasting lateral shifts with the no-shift control condition. To quantify dynamical changes in alpha band power, we divided the 1100-ms cue–stimulus interval into three equal time windows (latencies, L) of 365 ms each (L1: " 1100 to " 735 ms; L2: " 734 to " 370 ms; L3: " 369 to " 4 ms) and compared the average alpha band amplitude over parietal regions in these windows for all three cues. Average values of the corresponding time–frequency regions, defined by latency and frequency limits, were subject to three-way repeated measures ANOVA designs for both upper and lower alpha subbands separately. Within-subjects factors were Cue (three levels: shift left, no shift, and shift right), Hemisphere (two levels: left parietal and right parietal), and Latency (three levels: L1 through L3). No main effects of any factor were observed in either alpha band except for a latency effect in the lower alpha subband, F(2,34) 5 5.56, po.01, due to an overall drop in power during L2 and L3 of the cue–stimulus interval. A two-way Cue ! Hemisphere interaction was evident in both higher, F(2,34) 5 12.43, po.01, and lower, F(2,34) 5 5.68, po.01, alpha subbands. This was due to overall higher amplitude over parietal regions ipsilateral to the direction of attention when covert orienting was cued. No Cue ! Latency (higher alpha: F[4,68] 5 0.80, p 5 .53; lower alpha: F[4,68] 5 0.35, p 5 .84) or Hemisphere ! Latency (higher alpha: F[2,34] 5 0.61, p 5 .55; lower alpha: F[2,34] 5 2.10, p 5 0.14) interactions were detected. However, a clear three-way Cue ! Hemisphere ! Latency interaction was observed for both higher, F(4,68) 5 2.83, po.05, and lower, F(4,68) 5 8.04, po.01, alpha. Figure 4 is the graphical representation of this three-way interaction and presents the results obtained for both alpha subbands, according to hemisphere, cueing condition, and cue–stimulus interval latency. Scalp interpolations in lateral panels present the difference between lateral shifts and the no-shift condition (i.e., the difference between lateral and central panels). A clear pattern can be observed when the three conditions are contrasted (post hoc Bonferroni-corrected comparisons, only po.05 are informed; lower alpha, within MS 5 0.00808; higher alpha, within MS 5 0.00769; both df 5 68). When comparing only left and right shifts (i.e., ignoring central panels in Figure 4A,B) it can be observed that ipsilateral (contralateral) parietal activity was always higher (lower) than contralateral (ipsilateral) activity in mid- and late latencies for both alpha bands. Yet, taken in isolation, these results cannot distinguish whether ipsilateral increase or contralateral drop is specific for the process of covert orienting; comparing them against the no-shift condition is necessary to resolve this ambiguity. Indeed, we observed that when attention was cued to stay at fixation, left and right parietal regions exhibited the same behavior for both higher and lower alpha
Attention shifts and parietal alpha band activity bands across the entire cue–stimulus interval (Figure 4A,B respectively, central panel): Amplitude in the lower alpha band dropped from L1 to L2 and then stayed at the same level until the appearance of the stimulus; in the higher alpha the pattern was somewhat different, showing an initial drop and a subsequent tendency to increase back to baseline levels. However, when attention was shifted away from fixation toward lateral positions, only the ipsilateral parietal regions showed activity that was different from the no-shift condition. This was observed in both subbands, albeit with different dynamics (see lateral panels in Figure 4, A vs. B): Whereas in the lower alpha band ipsilateral parietal regions maintained a level near to baseline activity, showing only a slight drop, amplitude in the higher alpha band showed a consistent increase throughout the cue–stimulus interval. Alpha band amplitude over the contralateral hemisphere, on the other hand, was indistinguishable from the no-shift condition, as can be seen by comparing in Figures 4A and 4B lateral versus central panels. These results show that alpha band activity over ipsilateralto-attended parietal regions is specifically engaged during the displacement of the focus of attention away from fixation. They likewise show that alpha band activity over contralateral parietal regions behaves the same way as if no shift was required. Discussion The main results of this study can be summarized as follows: (i) Behavioral data confirmed that subjects were able to perform the task consistently, but, as expected, performance was always much better at fixation than at the periphery. (ii) The use of a central informative no-shift cue was effective in keeping attention at fixation as revealed by smaller P1/N1 amplitudes evoked by invalid (lateral, peripheral) stimuli. (iii) The cue-related ADAN component was consistently more negative over frontal regions contralateral to the shift of attention and more pronounced than the no-shift condition. Additionally, P3-type and CNV waveforms in the cue– stimulus interval were enhanced in both shift conditions as compared to the no-shift task. (iv) When attention was covertly displaced from fixation, alpha band activity over the parietal regions ipsilateral to the attended hemifield (contralateral to the ignored side) was enhanced relative to the no-shift condition. Alpha band activity over parietal regions contralateral to attended hemifield, on the other hand, were comparable to the no-shift task. In the following we discuss a number of points raised by these results. The informative no-shift cue/central task condition Traditional studies in attentional shifting follow Posner’s (1980) now classical cue–target paradigm whereby the subject is instructed to direct attention to one of two possible peripheral locations upon endogenous or exogenous cues. In some cases, neutral (noninformative regarding the most probable location of stimulus appearance) cues are used. Usually, brain activity is contrasted between conditions, and differences between shifting to one side or another are revealed. However, such comparisons can only show what is going on in the brain that differentiates shifting to the right from shifting to the left (or any other alternative locations used) but do not contrast shifting attention to the periphery against not doing so (Nobre et al., 2000). By incorporating a behaviorally relevant no-shift condition at fixation, we sought to transcend the limitations of using neutral or interpret cue conditions as contrasts that do not ensure the absence of attentional covert modulation (see de Gonzaga Gawryszewski et al., 1987; Jonides, 1980).
319 Although some studies have dealt with the question of differences between central and peripheral attention (de Gonzaga Gawryszewski et al., 1987; Miniussi, Rao, & Nobre, 2002; Neville & Lawson, 1987; Tassinari, Aglioti, Chelazzi, Marzi, & Berlucchi, 1987), these have been mainly behavioral or have focused on targets rather than on cues, therefore being unable to specify brain activity related to the orientation phase. Nevertheless, they do show that attention can be controlled to stay put at fixation through central cues (Yantis & Jonides, 1990), and that this has behaviorally analogous consequences as attending to a peripherally cued location in the sense that nonattended locations suffer perceptual detriment. Our results confirm and extend these findings: Keeping attention at fixation by central informative no-shift cues avoids any lateral facilitation, as revealed by the reduced P1/N1 to subsequent (invalid) peripheral stimuli (Figure 2). We believe that the incorporation of this condition is therefore a valuable contrast for the study of brain activity specifically related to the covert orienting of visuo-spatial attention. Moreover, we propose to consider distinguishing between noninformative (neutral) cues, which do not necessarily ensure a no-shift condition (McDonald & Green, 2008), and the more explicit informative no-shift cue as used here, which is tailored to keep the attentional focus at fixation. Behavioral results did show, however, that the task at fixation was always easier than the lateral (peripheral) tasks. This result has important consequences for the validity of the no-shift control as used in our study. Indeed, because alpha band activity is known to be highly dependent on changes in alertness or mental effort (Dockree, Kelly, Foxe, Reilly, & Robertson, 2007; Keil, Mussweller, & Epstude, 2006; Ray & Cole, 1985), it poses the question as to what extent unspecific effects such as these might be at play when the subject is confronted with the more demanding task of shifting attention away from fixation. One possibility would be to match the central and lateral tasks for difficulty. However, this has the drawback of making the no-shift condition a less ecologically valid control: Displacing the focus of attention from where one is fixating is inherently effortful and energy demanding, whereas fixating where one expects something to happen is comparatively easy. The validity of the central control, therefore, cannot depend on it demanding equivalent effort, because voluntary covert displacements of attention are never effortless (James, 1950). Trying to match these conditions in terms of effort would mean distancing oneself from the way voluntary covert shifts of attention are undertaken naturally. This said, and as pointed out by an anonymous reviewer, the existence of two difficulty levels in our task offers a way of estimating if effort has a substantial impact on our results. If effort or alertness were responsible for changes in alpha band dynamics during covert shifts, the two difficulty levels would differ in this variable. As the analysis presented in the supplementary material shows, however, no significant changes in alpha band activity were detected between difficulty levels during lateral shifts despite clear behavioral differences. This suggests that the modulations we see cannot be explained by unspecific changes in effort or alertness but seem more likely to reflect a topographical redistribution of cortical activity in the alpha band related to brain mechanisms underlying covert attentional shifts. Cue-Related ERPs Although we did not find evidence for a clear LDAP in our data, we observed a significant ADAN waveform in the 350–650-ms postcue window (Figure 3). In addition to showing the tradi-
320 tional contralateral preference, we observed that both shifts were associated with more negative values as compared to the no-shift condition. This suggests that, although some of the brain processes underlying the ADAN might overlap with activity related to the control of nonspatial attention, an important part likely reflects activity that is specific for the covert orienting of attention (Jongen et al., 2007; Kiss et al., 2008; van Velzen & Eimer, 2003). Furthermore, our results appear consistent with the study by Grent-’t-Jong and Woldorff (2007) that showed that a frontal negative deflection beginning around 400 ms postcue remained when an interpret-cue condition was subtracted from lateral shifts. The P3-type component we observed was much greater in amplitude in the shift versus the no-shift condition, especially over superior parietal regions, but with no evident lateralization. This could indicate the presence of an underlying location-independent mechanism for displacing the focus of attention. Indeed, in a recent study (Slagter, Kok, Mol, Talsma, et al., 2005) a similar pattern of parieto-central P3-type positivity around 260– 520 ms was observed when spatial versus nonspatial attentional control was compared: P3-type activation was stronger for trials where the cued feature had changed regardless of whether the feature was spatial or not. In view of these results, the authors suggested that this P3 component could be indicative of processes underlying the generic switching (updating) of the ongoing attentional set (see also Slagter, Kok, Mol, & Kenemans, 2005). This interpretation would be compatible with our results, as both shift-left and shift-right conditions imply an important cognitive change with respect to the previous moment, whereas the central cueing is more of an ongoing maintenance and focalization of the current state. Finally, an influential work by Corbetta et al. (1993) used a central-cueing control in a positron emission tomography study. The authors showed that, when compared to lateral shifts, central cues produced lower overall superior frontal and superior parietal activity, which is in line with the P3-type results presented here (see Figure 3B). The last cue-related component we studied was the CNV (Brunia & van Boxtel, 2001; Walter, Cooper, Aldridge, McCallum, & Winter, 1964). The results presented here are in overall agreement with previous reports regarding the presence of the CNV component, as stimulus appearance was completely predictable in our setting. Interestingly, more negative values were observed for both attentional shifts, although there was no difference between left versus right shifts. The neural origin of the CNV is not fully understood, but it is proposed to have frontal, fronto-central, medial frontal, and superior and medial parietal generators depending both on the task and whether it is an early or late CNV (Gomez et al., 2001; Sieroka, Dosch, Specht, & Rupp, 2003). An enhanced CNV would be compatible with a naturally more effortful gesture during covert shifts of attention if, indeed, these were of premotor origin (Rizzolatti, Riggio, Dascola, & Umilta, 1987). One would expect a stronger involvement of supplementary motor cortex and related frontal and fronto-medial areas in such cases (see also Small et al., 2003). This interpretation is supported by a study on covert orienting (Wright, Geffen, & Geffen, 1995), where directional cues induced stronger CNV when compared to nondirectional, neutral cues. Alpha Band Oscillatory Activity Oscillatory activity that presented clear attention-shift-related modulations was confined to a broad alpha band (6–14 Hz). This is in line with numerous noninvasive studies in human beings that have shown that this electroencephalographic spectral compo-
D. Cosmelli et al. nent is highly reactive to attentional manipulations (Babiloni et al., 2003; Bastiaansen, Bocker, Cluitmans, & Brunia, 1999; Foxe et al., 1998; Fu et al., 2001; Kelly et al., 2006; Klimesch, 1999; Rihs et al., 2007, 2009; Sauseng et al., 2005; Thut et al., 2006; Worden et al., 2000). However, a number of studies have reported modulations in posterior alpha band activity when subjects are expecting forthcoming events, even if orienting attention to a different location in space is not required (Babiloni et al., 2003; Bastiaansen & Brunia, 2001; Filipovic´, Jahanshahi, & Rothwell, 2001). On the other hand, differential reactivity of intra-alpha bands has been observed depending on whether the subject has to memorize, make semantic judgments, or simply attend to upcoming stimuli (Fink, Grabner, Neuper, & Neubauer, 2005; Klimesch, 1999), thus suggesting a rather complex functional role of this widespread cortical oscillation (Palva & Palva, 2007). In the specific case of visual attention, this raises the question of to what extent changes in cortical activity, in particular occipito-parietal alpha and subbands within, are specifically related to the spatial displacement of the focus of attention or rather reflect a complex mixture of preparation, expectation, and so forth and the actual spatial orientation processes. More specifically, an important issue that remains controversial in the literature pertains the relative cortical distribution of changes in alpha power (Kelly et al., 2006; Rihs et al., 2007, 2009; Sauseng et al., 2005; Worden et al., 2000): whether lateral covert shifts of attention are related to contralateral alpha decrease (reflecting putative facilitation of the attended visual field) or ipsilateral alpha increase (reflecting putative inhibition of the ignored visual field). Making this distinction in light of the informative no-shift cue control has been the main objective of this study. Our results are in line with the notion that the overall process of selective attention depends on a complex mechanism involving both facilitation of potentially relevant features and inhibition of irrelevant aspects (Hillyard et al., 1994; Rihs et al., 2007, 2009; Suffczynski, Kalitzin, Pfurtscheller, & Lopes da Silva, 2001). A recent study by Rihs et al. (2009) revealed both contralateral-to-attended decrease and contralateral-to-unattended increase depending on whether the early or late phase of the cue– stimulus interval was considered, respectively. The results presented here are in partial agreement with these findings and extend them by suggesting potential mechanisms of suppression/ facilitation in the differential dynamics present within subbands in the alpha range (see Figure 4). Indeed, we observed a sustained increased activity in the higher alpha range over parietal sites ipsilateral to the attended location (contralateral to the unattended site) that became more evident in the later phase of the cue–stimulus interval. This increase in alpha oscillatory activity was specific for the attentional orientation process as contrasting it to the informative no-shift cue demonstrates: Contralateral-to-attended parietal sites were indistinguishable from the no-shift control. These results are consistent with recent work on the relation between P1 and alpha band oscillations showing that upper alpha band activity could be specifically related to inhibition of task-irrelevant brain regions (Freunberger et al., 2008). In the lower alpha band, on the other hand, parietal activity contralateral to the attended hemifield showed a rapid decrease that was maintained throughout the cue–stimulus interval. This result appears initially in agreement with previous reports showing that attentional orienting is associated with a specific drop in posterior alpha band power over regions that process the attended hemifield (Sauseng et al., 2005;
Attention shifts and parietal alpha band activity
321
Thut et al., 2006). Yet only contralateral-to-ignored (ipsilateralto-attended) regions showed specific differences when compared to the no-shift control, whereas contralateral-to-attended parietal cortices maintained the same level of activity as when no attentional orienting was undertaken. It seems possible to hypothesize, therefore, that facilitatory processes during selective attention, if indexed by reduced posterior alpha activity, might be more related to expectation or general preparatory activity necessary for the subsequent target discrimination than to the actual covert displacement of attention (see also Capotosto, Babiloni, Romani, & Corbetta, 2009; Yamagishi et al., 2008). The latter seems to be associated with selective, dynamically complex, ipsilateral-to-attended augmented parietal alpha oscillations.
Conclusions To better understand the dynamics of EEG oscillatory activity during visuo-spatial attention orienting, we contrasted lateral covert shifts with a behaviorally relevant no-shift control at fixation. In this way, we were able to disentangle which aspects of alpha band dynamics are specifically reactive to covert orienting: In the upper alpha band, a gradual increase over parietal regions ipsilateral to the attended hemifield was observed during lateral shifts. In the lower alpha band, a drop in power was evident during covert orienting over both hemispheres. However, only ipsilateralto-attended parietal regions were different from the control, maintaining a comparatively higher level of activity. In both alpha subbands, the activity of parietal regions contralateral to the attended hemifield did not differ from the no-shift task.
REFERENCES Babiloni, C., Brancucci, A., Babiloni, F., Capotosto, O., Carducci, F., Cincotti, F., et al. (2003). Anticipatory cortical responses during the expectancy of a predictable painful stimulation. A high-resolution electroencephalography study. European Journal of Neuroscience, 18, 1692–1700. Bastiaansen, M. C., Bocker, K. B., Cluitmans, P. J., & Brunia, C. H. (1999). Event-related desynchronization related to the anticipation of a stimulus providing knowledge of results. Clinical Neurophysiology, 110, 250–260. Bastiaansen, M. C., & Brunia, C. H. (2001). Anticipatory attention: An event-related desynchronization approach. International Journal of Psychophysiology, 43, 91–107. Brunia, C. H., & van Boxtel, G. J. (2001). Wait and see. International Journal of Psychophysiology, 43, 59–75. Capotosto, P., Babiloni, C., Romani, G. L., & Corbetta, M. (2009). Frontoparietal cortex controls spatial attention through modulation of anticipatory alpha rhythms. Journal of Neuroscience, 29, 5863–5872. Corbetta, M., Miezin, F. M., Shulman, G. L., & Petersen, S. E. (1993). A PET study of visuospatial attention. Journal of Neuroscience, 13, 1202–1226. Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews Neuroscience, 3, 201–215. de Gonzaga Gawryszewski, L., Riggio, L., Rizzolatti, G., & Umilta, C. (1987). Movements of attention in the three spatial dimensions and the meaning of ‘‘neutral’’ cues. Neuropsychologia, 25, 19–29. Delorme, A., & Makeig, S. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134, 9–21. Dockree, P. M., Kelly, S. P., Foxe, J. J., Reilly, R. B., & Robertson, I. H. (2007). Optimal sustained attention is linked to the spectral content of background EEG activity: Greater ongoing tonic alpha (approximately 10 Hz) power supports successful phasic goal activation. European Journal of Neuroscience, 25, 900–907. Doesburg, S. M., Roggeveen, A. B., Kitajo, K., & Ward, L. M. (2008). Large-scale gamma-band phase synchronization and selective attention. Cerebral Cortex, 18, 486–499. Filipovic´, S. R., Jahanshahi, M., & Rothwell, J. C. (2001). Uncoupling of contingent negative variation and alpha band event-related desynchronization in a go/no-go task. Clinical Neurophysiology, 112, 1307–1315. Fink, A., Grabner, R. H., Neuper, C., & Neubauer, A. C. (2005). EEG alpha band dissociation with increasing task demands. Brain Research. Cognitive Brain Research, 24, 252–259. Foxe, J. J., Simpson, G. V., & Ahlfors, S. P. (1998). Parieto-occipital approximately 10 Hz activity reflects anticipatory state of visual attention mechanisms. NeuroReport, 9, 3929–3933. Freunberger, R., Ho¨ller, Y., Griesmayr, B., Gruber, W., Sauseng, P., & Klimesch, W. (2008). Functional similarities between the P1 component and alpha oscillations. European Journal of Neuroscience, 27, 2330–2340. Fries, O., Reynolds, J. H., Rorie, A. E., & Desimone, R. (2001). Modulation of oscillatory neuronal synchronization by selective visual attention. Science, 291, 1560–1563.
Fu, K. M., Foxe, J. J., Murray, M. M., Higgins, B. A., Javitt, D. C., & Schroeder, C. E. (2001). Attention-dependent suppression of distracter visual input can be cross-modally cued as indexed by anticipatory parieto-occipital alpha-band oscillations. Brain Research, 12, 145–152. Gomez, C. M., Delinte, A., Vaquero, E., Cardoso, M. J., Vazquez, M., Crommelinck, M., et al. (2001). Current source density analysis of CNV during temporal gap paradigm. Brain Topography, 13, 149– 159. Green, J., & McDonald, J. (2008). Electrical neuroimaging reveals timing of attentional control activity in human brain. Plos Biology, 6, e81. Grent-’t-Jong, T., & Woldorff, M. G. (2007). Timing and sequence of brain activity in top-down control of visual-spatial attention. Plos Biology, 5, e12. Hillyard, S., Luck, S. J., & Mangun, G. R. (1994). The cueing of attention to visual field locations: Analysis with ERP recordings. In H. J. Heinze, T. F. Munte, & G. R. Mangun (Eds.), Cognitive electrophysiology (pp. 1–25). Boston: Birkhauser. Hillyard, S. A., & Anllo-Vento, L. (1998). Event-related brain potentials in the study of visual selective attention. Proceedings of the National Academy of Sciences, USA, 95, 781–787. Hillyard, S. A., & Kutas, M. (1983). Electrophysiology of cognitive processing. Annual Review of Psychology, 34, 33–61. Hopf, J. M., & Mangun, G. R. (2000). Shifting visual attention in space: An electrophysiological analysis using high spatial resolution mapping. Clinical Neurophysiology, 111, 1241–1257. Hopfinger, J. B., Camblin, C. C., & Parks, E. L. (2010). Isolating the internal in endogenous attention. Psychophysiology (in press). James, W. (1950). Principles of psychology (vol. 1). New York: Dover Publications. (Originally published in 1890). Jensen, O., Kaiser, J., & Lachaux, J.-P. (2007). Human gammafrequency oscillations associated with attention and memory. Trends in Neuroscience, 30, 317–324. Jervis, B. W., Ifeachor, E. C., & Allen, E. M. (1988). The removal of ocular artifacts from the electroencephalogram: A review. Medical & Biological Engineering & Computing, 26, 2–12. Jongen, E., Smulders, F., & Van Der Heiden, J. (2007). Lateralized ERP components related to spatial orienting: Discriminating the direction of attention from processing sensory aspects of the cue. Psychophysiology, 44, 968–986. Jongen, E. M., Smulders, F. T., & van Breukelen, G. J. (2006). Varieties of attention in neutral trials: Linking RT to ERPs and EEG frequencies. Psychophysiology, 43, 113–125. Jonides, J. (1980). Towards a model of the mind’s eye’s movement. Canadian Journal of Psychology, 34, 103–112. Kastner, S., & Ungerleider, L. (2000). Mechanisms of visual attention in the human cortex. Annual Review of Neuroscience, 23, 315–341. Keil, A., Mussweller, T., & Epstude, K. (2006). Alpha-band activity reflects reduction of mental effort in a comparison task: A source space analysis. Brain Research, 1121, 117–127. Kelly, S. O., Lalor, E. C., Reilly, R. B., & Foxe, J. J. (2006). Increases in alpha oscillatory power reflect an active retinotopic mechanism for distracter suppression during sustained visuospatial attention. Journal of Neurophysiology, 95, 3844–3851.
322 Kiss, M., van Velzen, J., & Eimer, M. (2008). The N2pc component and its links to attention shifts and spatially selective visual processing. Psychophysiology, 45, 240–249. Klimesch, W. (1999). EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Research, 29, 169– 195. Luck, S. J., Heinze, H. J., Mangun, G. R., & Hillyard, S. A. (1990). Visual event-related potentials index focused attention within bilateral stimulus arrays. II. Functional dissociation of P1 and N1 components. Electroencephalography and Clinical Neurophysiology, 75, 528–542. Luck, S. J., Hillyard, S. A., Mouloua, M., Woldorff, M. G., Clark, V. O., & Hawkins, H. L. (1994). Effects of spatial cuing on luminance detectability: Psychophysical and electrophysiological evidence for early selection. Journal of Experimental Psychology, 20, 887–904. Luck, S. J., Woodman, G. F., & Vogel, E. K. (2000). Event-related potential studies of attention. Trends in Cognitive Science, 4, 432–440. Malinowski, O., Fuchs, S., & Muller, M. M. (2007). Sustained division of spatial attention to multiple locations within one hemifield. Neuroscience Letters, 414, 65–70. McDonald, J., & Green, J. (2008). Isolating event-related potential components associated with voluntary control of visuo-spatial attention. Brain Research, 1227, 96–109. Miniussi, C., Rao, A., & Nobre, A. C. (2002). Watching where you look: Modulation of visual processing of foveal stimuli by spatial attention. Neuropsychologia, 40, 2448–2460. Mu¨ller, M. M., Malinowski, O., Gruber, T., & Hillyard, S. A. (2003). Sustained division of the attentional spotlight. Nature, 424, 309–312. Nakamura, K., & Colby, C. L. (2000). Visual, saccade-related, and cognitive activation of single neurons in monkey extrastriate area V3A. Journal of Neurophysiology, 84, 677–692. Neville, H. J., & Lawson, D. (1987). Attention to central and peripheral visual space in a movement detection task: An event-related potential and behavioral study. I. Normal hearing adults. Brain Research, 405, 253–267. Nobre, A. C., Sebestyen, G. N., & Miniussi, C. (2000). The dynamics of shifting visuospatial attention revealed by event-related potentials. Neuropsychologia, 38, 964–974. Nunez, P. L., Wingeier, B. M., & Silberstein, R. B. (2001). Spatialtemporal structures of human alpha rhythms: Theory, microcurrent sources, multiscale measurements, and global binding of local networks. Human Brain Mapping, 13, 125–164. Palva, S., & Palva, J. M. (2007). New vistas for alpha-frequency band oscillations. Trends in Neuroscience, 30, 150–158. Pfurtscheller, G. (2001). Functional brain imaging based on ERD/ERS. Vision Research, 41, 1257–1260. Posner, M. I. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 3–25. Ray, W. J., & Cole, H. W. (1985). EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science, 228, 750–752. Raz, A. (2004). Anatomy of attentional networks. Anatomical Record, 281, 21–36. Rihs, T., Michel, C., & Thut, G. (2007). Mechanisms of selective inhibition in visual spatial attention are indexed by alpha-band EEG synchronization. European Journal of Neuroscience, 25, 603–610. Rihs, T., Michel, C., & Thut, G. (2009). A bias for posterior a-band power suppression versus enhancement during shifting versus maintenance of spatial attention. NeuroImage, 44, 190–199. Rizzolatti, G., Riggio, L., Dascola, I., & Umilta, C. (1987). Reorienting attention across the horizontal and vertical meridians: Evidence in favor of a premotor theory of attention. Neuropsychologia, 25, 31–40. Sauseng, O., Klimesch, W., Stadler, W., Schabus, M., Doppelmayr, M., Hanslmayr, S., et al. (2005). A shift of visual spatial attention is selectively associated with human EEG alpha activity. European Journal of Neuroscience, 22, 2917–2926. Sieroka, N., Dosch, H. G., Specht, H. J., & Rupp, A. (2003). Additional neuromagnetic source activity outside the auditory cortex in duration discrimination correlates with behavioural ability. NeuroImage, 20, 1697–1703. Slagter, H. A., Kok, A., Mol, N., & Kenemans, J. L. (2005). Spatiotemporal dynamics of top-down control: Directing attention to location and/or color as revealed by ERPs and source modeling. Brain Research, 22, 333–348.
D. Cosmelli et al. Slagter, H. A., Kok, A., Mol, N., Talsma, D., & Kenemans, J. L. (2005). Generating spatial and nonspatial attentional control: An ERP study. Psychophysiology, 42, 428–439. Small, D. M., Gitelman, D. R., Gregory, M. D., Nobre, A. C., Parrish, T. B., & Mesulam, M. M. (2003). The posterior cingulate and medial prefrontal cortex mediate the anticipatory allocation of spatial attention. NeuroImage, 18, 633–641. Stanislaw, H., & Todorov, N. (1999). Calculation of signal detection theory measures. Behavior Research Methods, Instruments, & Computers, 31, 137–149. Suffczynski, P., Kalitzin, S., Pfurtscheller, G., & Lopes da Silva, F. (2001). Computational model of thalamo-cortical networks: Dynamical control of alpha rhythms in relation to focal attention. International Journal of Psychophysiology, 43, 25–40. Tallon-Baudry, C., & Bertrand, O. (1999). Oscillatory gamma activity in humans and its role in object representation. Trends in Cognitive Science, 3, 151–162. Talsma, D., Slagter, H. A., Nieuwenhuis, S., Hage, J., & Kok, A. (2005). The orienting of visuospatial attention: An event-related brain potential study. Brain Research, 25, 117–129. Tassinari, G., Aglioti, S., Chelazzi, L., Marzi, C. A., & Berlucchi, G. (1987). Distribution in the visual field of the costs of voluntarily allocated attention and of the inhibitory after-effects of covert orienting. Neuropsychologia, 25, 55–71. Thut, G., Nietzel, A., Brandt, S. A., & Pascual-Leone, A. (2006). Alphaband electroencephalographic activity over occipital cortex indexes visuospatial attention bias and predicts visual target detection. Journal of Neuroscience, 26, 9494–9502. van Velzen, J., & Eimer, M. (2003). Early posterior ERP components do not reflect the control of attentional shifts toward expected peripheral events. Psychophysiology, 40, 827–831. Walter, W. G., Cooper, R., Aldridge, V. J., McCallum, W. C., & Winter, A. L. (1964). Contingent negative variation: An electric sign of sensorimotor association and expectancy in the human brain. Nature, 203, 380–384. Weiskrantz, L., Warrington, E. K., Sanders, M. D., & Marshall, J. (1974). Visual capacity in the hemianopic field following a restricted occipital ablation. Brain, 97, 709–728. Worden, M. S., Foxe, J. J., Wang, N., & Simpson, G. V. (2000). Anticipatory biasing of visuospatial attention indexed by retinotopically specific alpha-band electroencephalography increases over occipital cortex. Journal of Neuroscience, 20, RC63. Wright, M. J., Geffen, G. M., & Geffen, L. B. (1995). Event related potentials during covert orientation of visual attention: Effects of cue validity and directionality. Biological Psychology, 41, 183–202. Wright, R. D., Richard, C. M., & McDonald, J. J. (1995). Neutral location cues and cost/benefit analysis of visual attention shifts. Canadian Journal of Experimental Psychology, 49, 540–548. Yamagishi, N., Callan, D. E., Anderson, S. J., & Kawato, M. (2008). Attentional changes in pre-stimulus oscillatory activity within early visual cortex are predictive of human visual performance. Brain Research, 1197, 115–122. Yantis, S., & Jonides, J. (1990). Abrupt visual onsets and selective attention: Voluntary versus automatic allocation. Journal of Experimental Psychology, 16, 121–134.
SUPPORTING INFORMATION Additional Supporting Information may be found in the online version of this article: Table S1: Behavioral measures (Reaction Times, Hit Rates and False Alarms) according to Attentional Condition and Difficulty level. Table S2: Alpha Band Power, Easy vs Difficult comparisons, Ipsilateral-to-shift hemisphere parietal electrodes, Latencies 2–3. Figure S1: Illustration of individual alpha peak differences. Please note: Wiley-Blackwell are not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article. (Received October 30, 2009; Accepted March 31, 2010)
Psychophysiology, 48 (2011), 323–332. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01069.x
Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks
JEFF KLINGNER,a BARBARA TVERSKY,b,c and PAT HANRAHANa a
Department of Computer Science, Stanford University, Stanford, California, USA Teachers College, Columbia University, New York, New York, USA c Department of Psychology, Stanford University, Stanford, California, USA b
Abstract Degree of pupil dilation has been shown to be a valid and reliable measure of cognitive load, but the effect of aural versus visual task presentation on pupil dilation is unknown. To evaluate effects of presentation mode, pupil dilation was measured in three tasks spanning a range of cognitive activities: mental multiplication, digit sequence recall, and vigilance. Stimuli were presented both aurally and visually, controlling for all known visual influences on pupil diameter. The patterns of dilation were similar for both aural and visual presentation for all three tasks, but the magnitudes of pupil response were greater for aural presentation. Accuracy was higher for visual presentation for mental arithmetic and digit recall. The findings can be accounted for in terms of dual codes in working memory and suggest that cognitive load is lower for visual than for aural presentation. Descriptors: Pupil dilations, Cognition, Normal volunteers, Learning/memory
Kahneman (1973) used pupillary dilations as the primary empirical foundation for his attention theory of effort. He identified three criteria desirable in general for physiological proxies for effort and that he observed in pupillary dilations: differences in the magnitude of averaged pupillary dilations reliably reflect (a) different difficulty levels of a single task, (b) differences in difficulty across qualitatively different tasks, and (c) individual differences in ability. In a review nine years later, Beatty (1982b) reaffirmed that the experimental evidence then available did indeed show that pupillary dilations fulfill all three of Kahneman’s criteria. Nearly all the studies that had been done at that time used auditory stimuli, in order to avoid interference from the pupillary light reflex. Since then, several investigators have successfully controlled for reflex dilations while using visual stimuli and shown that task-induced dilations can serve as reliable proxies for cognitive load in visual tasks such as reading (Just & Carpenter, 1993) and visual search (Backs & Walrath, 1992; Porter, Troscianko, & Gilchrist, 2007). These studies all show that the magnitudes of momentary pupil dilations reliably reflect different levels of difficulty within individual tasks and have thus validated pupillary dilations as continuing to fulfill Kahneman’s first criterion even on extension to visual tasks. However, to our knowledge, nobody has examined the effect of aural versus visual presentation mode itself on the magnitude of pupillary dilations. This lack of data confounds the use of dilations for comparing cognitive loads between visual and aural tasks, because it cannot be known how much of the difference is caused by the difference in presentation modalities and how much is caused by differences in postperception task demands. In other words, it is still not known whether Kahneman’s second
Assessing the cognitive load imposed by various visual tasks is important to the design of cognitively efficient visual interfaces. Most interfaces are visual, and many require shifting attention between a variety of tasks with varying loads on perception, attention, memory, and information processing. The psychophysiological study of cognitive load in this context requires a physiological proxy that responds to load quickly and reliably reflects small differences in load. One such proxy is the tendency of pupils to dilate slightly in response to cognitive loads (Loe-wenfeld, 1999). This responsiveness of the pupil can provide detailed information about the timing and magnitude of cognitive loads and has thus been used to study a broad set of cognitive phenomena, including perception, memory, reasoning, and attention. (For general reviews, see Andreassi, 2006, ch. 12; Beatty, 1982b; Beatty & Lucero-Wagoner, 2000; Goldwater, 1972.) This work was funded by the Stanford Regional Visual Analytics Center, through the U.S. Department of Energy’s Pacific Northwest National Laboratory. Portions of this research were supported by NSF grants HHC 0905417, IIS-0725223, IIS-0855995, and REC 0440103. Our eye tracker was funded by the Stanford MediaX project and the Stanford School of Engineering. Some of the data from the aural presentation condition of all three experiments were presented at the Eye Tracking Research and Applications conference (Klingner, Kumar, & Hanrahan, 2008). The experiments described in this article were conducted in compliance with the policies of Stanford’s institutional review board. All participants gave informed consent, and their rights and safety were protected. Address correspondence to: Jeff Klingner, Gates Building Room 3B396, Stanford, CA 94305-9035, USA. E-mail: klingner@stanfordalumni. org 323
324 criterion, intertask comparability, is still fulfilled by pupil dilations when they are used to study visual as well as aural tasks. Developments in graphics have brought interfaces, newspapers, textbooks, and instructions that increasingly present changing visual information. Viewers need to attend to, search through, and evaluate this information in order to integrate it. Are visual interfaces the best way to present this information, or might cognitive load be lessened with auditory presentation? Are the parameters of cognitive load similar for visual and auditory presentation? To address those questions, the present experiments compare visual and auditory presentation for three classic paradigms in attention and perception. In choosing tasks, we sought to (a) span diverse types of cognitive load, (b) replicate well-studied tasks to enable comparisons to prior results, and (c) use simple stimuli that are easy to match between aural and visual presentation. We chose mental multiplication (Experiment 1), digit-span memory (Experiment 2), and vigilance (Experiment 3). In our experimental designs, we took care to control for all known noncognitive pupillary reflexes. The aural and visual conditions employ visual fields with matching brightness and contrast and with stimulus onset effects controlled; the difference is that in the aural conditions, the task-relevant stimuli were heard, and in the visual conditions they were seen. Because visual perception is generally believed to involve less effort, but the subsequent central processing demands were matched between the two presentation conditions, we expected dilations evoked by visually presented tasks to start out smaller but to eventually reach the same peak dilation as those evoked by the aurally presented versions. We also expected this difference in effort to be reflected in lower error rates and quicker responses in the visual conditions.
EXPERIMENT 1: MENTAL MULTIPLICATION Hess and Polt (1964) triggered broad interest in cognitive pupillometry when they reported that solving mental multiplication problems caused pupil dilations and that harder problems evoked larger dilations. Their results were replicated by Bradshaw (1968) for mental division with remainders; Boersma, Wilton, Barham, and Muir (1970) for mental addition in a study of mental retardation; and Ahern and Beatty (1979) in a study of the effect of individual differences in ability as measured by SAT scores. Recently, Marshall (2002) used a mental arithmetic task to validate a wavelet-based method of analyzing pupil measurements. These studies variously investigated the effects of problem difficulty, response mode, and participant ability on pupil dilations, but none investigated the influence of aural versus visual stimulus presentation. Experiment 1 was a replication of Ahern and Beatty’s (1979) mental multiplication study with the addition of two visual stimulus presentation conditions, one with timing matched to the aural condition, and one with simultaneous visual presentation of both parts of the multiplication problem. Method Unless otherwise specified, method details described here apply to all three experiments.
J. Klingner et al. Participants Twenty-four Stanford undergraduates participated in this experiment. All had normal or corrected-to-normal vision. We excluded participants with contact lenses or eyeglasses providing an astigmatism correction or a refractive correction greater than 10 diopters, which can interfere with accurate pupil diameter tracking. We compensated participants with Amazon.com gift certificates. The value of each participant’s gift certificate depended on his or her task performance and varied from about $15 for the lowest scores to about $35 for the highest. Such monetary incentive was shown by Heitz, Schrock, Payne, and Engle (2008) to increase the magnitude of pupillary responses. Apparatus We measured the size of participants’ pupils using a Tobii 1750 remote eye tracker (Tobii Technologies, 2007). This device is designed primarily to track people’s gaze direction, but its method of gaze tracking also enables high-speed pupillometry (Klingner, Kumar, & Hanrahan, 2008). The eye tracker is based on a standard LCD computer display, with infrared lights and a highresolution infrared camera mounted at the edges of the screen. This remote-camera setup requires neither a chin rest nor a headmounted camera, enabling pupil measurements without encumbrance or distraction. Measurements are corrected for changes in apparent pupil size due to head motion toward or away from the camera. Accurate pupil tracking with this equipment requires a head motion speed of less than 10 cm/s within a head box of about 30 ! 15 ! 20 cm at our initial seating distance of 60 cm from the screen. We placed the eye tracker on a desk with the top of the screen approximately 140 cm from the floor. Participants sat in a chair adjusted so that their eyes were at this same height. Participants initiated trials and gave task responses using a two-button computer mouse on the desk between them and the eye tracker. We used a relatively bright room, with 27 cd/m2 of luminance from the surrounding walls at eye level and 32 lx incident at participants’ eyes. Data Processing Under infrared illumination, participants’ pupils appear as bright ovals in the eye tracker’s camera image. The Tobii 1750 measures the size of a participant’s pupil by fitting an ellipse to the pupil image then converting the width of the major axis of that ellipse from pixels to millimeters based on the measured distance from the camera to the pupil. Due to inaccuracy in this measurement of camera–pupil distance, measurements of absolute pupil size may have errors of up to 5%, but sample-tosample changes in pupil diameter are much more accurate. This better accuracy for relative measures makes eye trackers well suited for cognitive pupillometry, where the measurement of interest is usually changes in pupil diameter relative to their diameter at the end of an accommodation period preceding each trial (Beatty & Lucero-Wagoner, 2000). This measure has been found to be independent of baseline pupil diameter and commensurate across multiple laboratories and experimental procedures (Beatty, 1982b; Bradshaw, 1969, 1970). Our apparatus samples pupil size at 50 Hz, with each sample measuring both eyes simultaneously. Because the left and right eyes exhibit matching pupillary responses, we used the average of the two eyes’ pupil diameters to reduce measurement noise. During moments when an eyelid, eyelash, or eyeglasses frame blocked the camera’s view of one pupil, we used the other pupil
Effect of task presentation mode on pupil dilation
325
alone. We performed standard baseline subtraction in each trial based on the average pupil diameter measured over 20 samples (400 ms) at the end of a prestimulus accommodation period. After filling blinks via linear interpolation, we smoothed the raw pupil signals with a 10-Hz low-pass digital filter. We constructed the pupil traces shown in all figures by stimulus aligning and averaging all trials for each illustrated condition.
active eye movements (e.g., Just & Carpenter, 1993). We controlled for saccade-induced luminance changes by presenting all stimuli at a fixed location within an area small enough to fall within the fovea and by helping participants to keep their gaze fixed by presenting a fixation target at all times and keeping trial durations under 20 s.
Data processing for statistical evaluation of differences in dilation magnitude. We quantified dilation magnitudes with the mean amplitude method (Beatty & Lucero-Wagoner, 2000, p. 148; Handy, 2004, p. 38). This method involves first measuring a baseline pupil size for each trial by averaging pupil size during a prestimulus accommodation period, then computing the average pupil size relative to this baseline during a response window defined for each task. We chose the mean dilation quantification method over the also common peak dilation method, because the latter is more sensitive to noise. We quantified each trial separately, enabling statistical evaluation of effect size and significance.
Other visual causes of pupil changes. In addition to the light reflex and the cognitive load response, the pupil also exhibits small dilations or contractions in response to changes in accommodation distance (Loewy, 1990), contrast (Ukai, 1985), spatial structure (Cocker, 1996) and the onset of coherent motion (Sahraie & Barbur, 1997). Kohn and Clynes (1969) showed that simply changing the color content of a visual stimulus, without changing either local or global luminance, can cause the pupils to either dilate or contract, depending on the nature of the color change. We controlled for all of these influences on pupil size by using achromatic, fixed-distance, nonmoving, constant-contrast stimuli.
Significance tests. We used an alpha level of .05 for all statistical tests. Tests of differences in mean dilation magnitude were all based on partitions of variance (ANOVA). Following the policy of Jennings (1987), we applied the Huynh and Feldt (1980) correction to degrees of freedom for within-subjects factors with more than two levels. In such cases, we report the Huynh–Feldt nonsphericity correction parameter ~e, the uncorrected degrees of freedom, and the corrected p value. We evaluated the significance of differences in error rates through onetailed tests for equality of proportions with Yates’ continuity correction (Miettinen & Nurminen, 1985).
Pupillary blink response correction. We followed the common practice of filling gaps in the data caused by blinks with linear interpolation. However, by performing blink-locked averaging of data from a pilot study, we observed that blinks result in changes in pupil diameter that persist for a few seconds after the blink. This pupillary blink response consists of a very brief dilation of about 0.04 mm, followed by a contraction of about 0.1 mm and then a gradual recovery to preblink diameter over the next 2 s. The timing and magnitude of these changes depend on the duration of the blink. To the extent that blinks occur randomly, pupillary blink responses add noise to averaged pupil diameter measurements, and to the extent that blinks are correlated with stimuli, pupillary blink responses add bias to averaged pupil diameter measurements. We gathered data from 20 thousand binocular blinks that occurred during several of our eye-tracking studies, grouped the blinks by duration, and averaged them to determine 3-s-long blink response correction signals. We then removed the pupillary blink responses in the current study by altering the data following each blink by subtracting the blink response correction signal corresponding to the length of that blink. For stimulus-correlated blinks, the general effect of this correction is to decrease the magnitude of pupillary responses measured in the first second following a blink by about 0.03 mm and increase the magnitude of pupillary responses measured in the second second following a blink by about 0.05 mm. For stimulusuncorrelated blinks, the general effect of this correction is to remove measurement noise and thereby decrease the standard errors of the mean in stimulus-locked averages of dilation magnitude. Because this is a new data processing technique for pupil data, we reran the analyses presented here without blink response correction and found that the correction did not change the significance of any of our results and changed the effect sizes by only 0.005–0.01 mm, suggesting that blinks were not well correlated with stimuli for the tasks we examined and contributed only noise to the stimulus-locked averages.
Data publication. All the raw data we collected, including practice trials and all excluded data, as well as the source code for the computer programs we used to collect and analyze it, are published as supporting information in the online version of this article. (See details at end of article.) Controlling for Noncognitive Pupillary Motions Pupillary light reflex. The largest potentially confounding pupillary motion is the pupillary light reflex, which is much larger in magnitude than cognition-induced pupil changes (Loe-wenfeld, 1999). We followed standard practice (e.g., Moresi et al., 2008; Verney, Granholm, & Dionisio, 2001) in maintaining constant visual field luminance across experimental conditions. Additionally, we used prestimulus masks equal in luminance and contrast to the stimulus, to avoid luminance and contrast changes at stimulus onset. Luminance changes caused by shifting gaze. Experiments in which participants shift their gaze to look at many parts of a visual stimulus, including studies of reactions to photographs (Dabbs & Milun, 1999; Libby, Lacey, & Lacey, 1973), visual search (Backs & Walrath, 1992; Porter et al., 2007), and visual scanning (Pomplun & Sunkara, 2003; Van Orden, Limbert, Makeig, & Jung, 2001) are subject to pupillary light reflexes when participants fixate on local areas of the stimulus with varying luminance even though the overall luminance of the stimulus does not change. Reading studies, in which textual stimuli have relatively uniform local luminance and consistent fixation sequences, are not as vulnerable to this problem and have successfully measured small task-evoked pupillary responses amidst
Stimuli Stimuli for all experiments were numbers between 1 and 20. Under the aural condition, stimuli were 500-ms digitized recordings of spoken numbers played over a computer speaker placed
326
J. Klingner et al.
directly behind the screen. Under the visual condition, we displayed these numbers at the center of the eye tracker’s integrated 17-in. 1280 ! 1024 LCD screen. We used a 28-point font size so that the digits spanned 0.731 (about a third of the foveal span) when viewed from participants’ initial seating distance of 60 cm. These numerals were black, and the rest of the screen was always filled with a uniform background of 64 cd/m2 medium gray. The onset timing and duration of stimuli presentation under the aural and visual conditions was matched. During periods of time with no stimulus (between trials, during the prestimulus pupil accommodation period, and in between presentation of numbers during the task), we presented silence in the aural condition and masked the stimulus by displaying an ‘‘X’’ at the center of the screen in place of a number, in order to remove contrast and brightness changes caused by the appearance or disappearance of the numerals. The absence of clear constrictions following the time of visual stimulus change in the visual waveforms provides evidence that these stimulus changes per se had little effect on the pupil in our experiments.
and the rest through 10 ! 10, so all but a few of the easiest problems required mental computation beyond simple recall. In the visual presentation condition, we used two timing variants. In the sequential treatment, which replicates Ahern and Beatty (1979), the multiplicand and multiplier were presented one after the other with timing matched to the aural condition. In the simultaneous treatment, both numbers were shown on the screen together for the full 8 s between the prestimulus accommodation period and the response prompt. This simultaneous and continuous presentation was intended to remove the requirement that subjects quickly read and remember the shortlived stimuli and thereby isolate the cognitive load imposed by mental multiplication from that caused by remembering the numbers. We instructed participants not to provide a response in cases when they forgot one of the two numbers or gave up on computing their product. This occurred in 10% (65/632) of the trials, mostly for hard problems. Because these trials did not involve mental multiplication, we excluded them from analysis.
Procedure Before each task, we explained the task to participants, then allowed them to practice until they were familiar and comfortable with the task presentation and providing their responses. All trials were initiated by participants, who first fixated a small target at the center of the screen before starting the trial by clicking a mouse button. Participants’ gaze thus remained at the center of the screen for the duration of each trial and during most of the short intervals between trials. A run of trials for a single task generally took about 5 min. We told participants that they could take breaks at any point between trials to rest their eyes; 2 did so. In Experiments 1 and 2, where the tasks required numerical responses, we asked participants to type their responses into a low-contrast on-screen keypad. We did this to automate data collection and to avoid pupillary reflexes to varying brightness caused by looking away from the screen. Because button-press responses themselves induce pupillary responses (Richer & Beatty, 1985), and we could not avoid such interference by using spoken responses (Bradshaw, 1967; Kahneman, Onuska, & Wolman, 1968), we limited our analysis to pre-response periods.
Results
Task Description We began each trial with a 2-s prestimulus accommodation period, during which participants rested their eyes on a fixation target in the center of the screen in order to stabilize their pupils. We then presented the participant with two numbers, the multiplicand and multiplier, separated by 2 s. Five seconds after we presented the multiplier, we asked participants for the two numbers’ product. In a departure from Ahern and Beatty’s (1979) procedure, rather than speaking the product, participants typed it into an on-screen keypad using the mouse. For significance testing, we averaged pupil dilation across the 5-s window between multiplier presentation and participant response. For each trial we randomly selected a difficulty level of easy, medium, or hard, then chose the multiplier and multiplicand randomly according to Ahern and Beatty’s (1979) definition of these difficulty levels: easy problems took the form f6, 7, 8, 9g ! f12, 13, 14g (e.g., 7 ! 13), medium were f6, 7, 8, 9g ! f16, 17, 18, 19g, and hard f11, 12, 13, 14g ! f16, 17, 18, 19g. Nine participants had memorized the multiplication table through 12 ! 12
Our results for aurally presented mental multiplication problems matched those of Ahern and Beatty (1979). We observed two peaks in pupil dilation: a brief, small dilation following presentation of the multiplicand and a longer, larger dilation following presentation of the multiplier, during the time when participants computed the numbers’ product. Dilation Magnitude by Presentation Mode Presentation mode affected the overall magnitude of pupil dilations but not their qualitative shape. The onset timing, duration, and overall shape of pupil dilations caused by mental multiplication was the same for both auditory and visual presentation. The size of participants’ dilations, however, was significantly larger in the auditory condition (M 5 0.35 mm, SD 5 0.11 mm vs. M 5 0.16 mm, SD 5 0.13 mm), F(1,22) 5 12.1, p 5 .002. This difference in magnitude is clear in Figure 1, which shows the pupil dilation evoked by the mental multiplication task, averaged across all trials and participants and broken down by task presentation mode. Dilation Magnitude by Task Difficulty Consistent with prior investigations of mental arithmetic, we found a clear difficulty effect on dilation magnitude. Easy multiplication problems caused the smallest pupil dilations (M 5 0.17 mm, SD 5 0.19 mm), hard problems the largest (M 5 0.27 mm, SD 5 0.16 mm), with dilations to medium problems in between (M 5 0.21 mm, SD 5 0.15 mm). These differences were significant, F(2,30) 5 13.1, p 5 .0008, ~e ¼ :67. Pupillary Response to Continuously Visible Problem The pupil dilation evoked by problems with both components visible simultaneously for 8 s had a different pattern: a single long dilation and contraction, rather than the two peaks we observed in the sequential case. In addition, the mean pupil dilation was smaller in the simultaneous case (M 5 0.13 mm, SD 5 0.11 mm vs. M 5 0.30 mm, SD 5 0.13 mm), F(1,22) 5 10.3, p 5 .004. This result is not surprising, because the simultaneous-presentation trials lack a second stimulus event to cause a second peak, and these trials were easier to solve, because they did not require participants to remember the two presented numbers.
Effect of task presentation mode on pupil dilation
Change in pupil diameter (mm)
0.5
327 multiplicand presented
multiplier presented
response prompted
0.4
0.3
0.2
0.1
aural (37 trials) visual (165 trials)
0.0
0
2
4 6 Time (seconds)
8
10
Figure 1. Average pupil dilation evoked by visually and aurally presented mental multiplication problems. The two presentation modes elicited dilations with similar timing, duration, and shape, but different magnitudes. Vertical lines show the times during which the two numbers were spoken or displayed and the time during which the participants responded.
Task Performance by Presentation Mode Participants made significantly more errors on aurally presented problems (40%) than visually presented problems (25%); w2(1, N 5 632) 5 3.39, p 5 .03. Discussion This experiment compared cognitive load under aural and visual presentation of mental arithmetic problems. The overall pattern of task-evoked pupil dilations was similar in both conditions and replicated previous aural work. Intriguingly, both the better performance under visual presentation and greater cognitive load under aural presentation suggest an advantage for visual presentation of mental arithmetic. This may be because poststimulus visual persistence alleviates some load on working memory.
EXPERIMENT 2: DIGIT SPAN SEQUENCE MEMORY Short-term recall of a paced sequence of digits (also known as the digit span task) is the most popular experimental task in cognitive pupillometry. First reported by Kahneman and Beatty (1966), the task was also used to investigate the related processes of longterm recall (Beatty & Kahneman, 1966), grouping (Kahneman, Onuska, & Wolman, 1968), and rehearsal (Kahneman & Wright, 1971). Peavler (1974) showed that the pupil reaches a plateau dilation of about 0.5 mm around the presentation of the seventh digit. Granholm, Asarnow, Sarkin, and Dykes (1996) replicated this finding, confirming that pupil dilation averaging can be used to estimate both the momentary load and the maximum capacity of working memory. As with the mental multiplication task, all prior investigations of pupil dilations evoked by this task presented the digit sequence aurally. Our experiment is a replication of the original Kahneman and Beatty (1966) study, with the addition of a visual presentation condition.
Method Details regarding study participants, equipment, and procedures not specific to this task are described in the Method section within the description of Experiment 1. As with Experiment 1, we started each trial with a pupil stabilization period in which we measured baseline pupil diameter. We then presented a sequence of digits at the rate of one per second, either spoken aloud or displayed on the screen. After a brief retention pause, participants then reported back the sequence using an on-screen keyboard as in Experiment 1. We used the first 2 s of this retention pause as the response window for pupil diameter averaging and significance testing, because this is the moment when Kahneman and Beatty (1966) observed maximum dilations. We randomly varied the length of the presented sequence for each trial independently between six and eight digits for aural presentation and between three and eight digits for visual.
Results Averaged pupil traces from this experiment are shown in Figure 2. Under both aural and visual presentation, changes in pupil diameter followed the same qualitative pattern observed by Kahneman and Beatty’s (1966) aural study: Participants’ pupils gradually dilated as the digits were memorized, reached a peak 2 s after the final digit during the pause while the sequence was retained in memory, then gradually contracted as the participants reported the digits back. Dilation Magnitude by Presentation Mode Aural presentation caused significantly larger pupil dilations during the retention pause than visual presentation (M 5 0.44 mm, SD 5 0.22 mm vs. M 5 0.24 mm, SD 5 0.17 mm), F(1,20) 5 5.9, p 5 .02.
328
J. Klingner et al. Visual
Change in pupil diameter (mm)
Aural 0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0
2
4
6 8 10 Time (seconds)
12
14
0
2
4 6 8 Time (seconds)
10
12
Figure 2. Pupil dilation evoked by a digit-span memory task presented aurally (left) and visually (right). The two charts are aligned and plotted at the same vertical scale. The numbered circles on each line show the times at which each digit was spoken (aural presentation) or displayed (visual presentation). The curves are each shifted horizontally so they are aligned at the end of the stimulus sequence. Thus the longest sequence (eight digits) starts the furthest to the left. Aural presentation caused larger dilations than visual, and under both presentation modes, longer memorization sequences elicited larger pupil dilations.
Dilation Magnitude by Task Difficulty We found a significant effect of sequence length on the magnitude of pupil dilations during the retention pause, F(3,60) 5 3.73, p 5 .02, ~e ¼ :96 (see Figure 2). The magnitude of the dilation increased monotonically with the length of the memorized sequence.
Task Performance Considering sequences of all lengths, participants made significantly more recall errors under aural (30%) than visual (24%) presentation, w2(1, N 5 1232) 5 3.94, p 5 .02, though this result is reversed if only the longest (length 7 and 8) sequences are considered. Average digit span was 6.0 digits for aural presentation and 5.6 digits for visual. Discussion This experiment compared cognitive load under short-term memorization of aurally and visually presented digit sequences. As with the mental arithmetic task, the qualitative shape of average pupil dilations was similar in both presentation modes, but the magnitude of dilations was smaller under visual presentation. Although visual presentation led to significantly greater overall performance, the difference was not large, and rates of recall for the longer sequences and average digit span scores suggest a small performance advantage for aural presentation. A general advantage to serial recall under aural presentation, especially for items late in the sequence, is well documented (Greene, 1992, p. 22; Penney, 1989; but see Beaman, 2002). Our findings on recall performance are mixed, but the larger dilations we observed in the aural condition suggest that task performance in this mode comes with the cost of higher cognitive load.
EXPERIMENT 3: VIGILANCE The mental multiplication and digit span tasks are both strongly dependent on working memory. We designed our third experiment to investigate the effect of aural versus visual stimulus mode on pupil dilations evoked by less memory-dependent processes, using a task that requires intermittent vigilance, stimulus discrimination, and speeded motor responses.
Method Details regarding study participants, equipment, and procedures not specific to this task are described in the Method section within the description of Experiment 1. In each trial, we presented an ascending sequence of numbers from 1 through 20. We told participants that the sequence might progress normally or might contain errors at the number 6, 12, and/or 18. When they noticed an error (a target), they were to push a button as quickly as possible. For example, part of the sequence might be ‘‘. . . 10, 11, 12, 13, . . .,’’ in which case we instructed the participants to do nothing, or it might be ‘‘. . . 10, 11, 7, 13, . . .,’’ in which case we told them to push the button as soon as possible after noticing the ‘‘7.’’ We inserted sequence errors (targets) at the three possible positions independently and randomly with probability one half. Thus, any trial could contain zero, one, two, or three targets, and participants knew exactly when the targets might appear. In the aural condition, ‘‘6’’ was never replaced by ‘‘16,’’ nor ‘‘18’’ by ‘‘8,’’ so that errors were apparent from the start of each spoken target stimulus. As with Experiments 1 and 2, the aural and visual conditions were matched on visual field luminance, contrast, and stimulus timing.
Effect of task presentation mode on pupil dilation
329
Unlike Experiments 1 and 2, this experiment did not replicate a past study, though it incorporated aspects of prior experiments. Beatty (1982a) found pupil dilations evoked by target tones in an auditory vigilance task, though in that experiment target locations were randomized, so that participants could not anticipate them, and continuous rather than intermittent vigilance was required. The anticipated increase in vigilance required by this task was studied by Richer, Silverman, and Beatty (1983).
vigilance. Whether the target was present or absent, the dilation began and peaked slightly later under aural presentation (see Figure 4). This slightly later dilation evoked by aural stimulus was probably due to the time taken for the stimulus to be presented, because hearing is generally believed to have lower latency than vision (Misulis & Fakhoury, 2001; Welford, 1980). This interpretation is consistent with the difference in mean reaction time we observed: 410 ms (SD 5 111 ms) for visual presentation and 713 ms (SD 5 140 ms) for aural.
Results Dilation Magnitude by Presentation Mode Figure 3 shows the average dilation evoked by the vigilance task, comparing aurally and visually presented trials. Both conditions elicited strong dilation peaks beginning about 1 s before and peaking 500–1000 ms after each moment when participants were alert for mistakes in the counting sequence. The 1-s anticipatory dilation is consistent with measurements of the readiness potential made using scalp electrodes by Becker, Iwase, Ju¨rgens, and Kornhuber (1976), who found evidence of motor preparation beginning a bit more than 1 s before action, and is shorter than the 1.5-s lead observed by Richer et al. (1983) before the presentation of an action-determining stimulus. For significance testing, we used a wide response window, starting 3 s before each moment when a target could occur and ending 3 s after, encompassing both the prestimulus anticipatory dilation and the poststimulus motor-response peak. The mean dilation in the auditory presentation condition (M 5 0.096 mm, SD 5 0.048 mm) was significantly larger than for visual presentation (M 5 0.057 mm, SD 5 0.046 mm), F(1,23) 5 7.93, p 5 .01. Dilation Onset and Peak Latency by Presentation Mode In contrast to Experiments 1 and 2, the three task repetitions in each of Experiment 3’s trials effectively tripled the number of trials available for analysis and so provided enough data to pinpoint the peak dilation precisely in time and revealed a minor timing difference between the dilations for aural and visual
Dilation Magnitude and Timing by Target Presence At every potential mistake point, whether or not a target is present, this task required heightened vigilance, motor response preparation, and comparison of the presented number with the expected correct sequence number. We therefore expected dilations in both cases to be similar, perhaps with slightly larger or longer dilations in cases where targets actually appeared, caused by error recognition, the additional requirement of carrying out the motor response, or both. We checked this hypothesis by grouping all time segments surrounding moments when the targets were present and averaging them separately from those when the targets were absent. The resultant pupil dilation averages are shown in Figure 4. Pupil dilations evoked by targets were larger and longer than those measured during moments when targets were possible but did not appear (M 5 0.10 mm, SD 5 0.046 mm vs. M 5 0.037 mm, SD 5 0.047 mm), F(1,23) 5 22.8, po.0001. The averaged pupil diameter trace for cases with a target (right side of Figure 4) showed a secondary peak about 1.5 s after the target appeared. Because mean response time was 515 ms (SD 5 188 ms), the latency between response and this secondary peak was about 1 s. Because Richer and Beatty (1985) observed similar dilation-response latencies in a nonreactive button pushing task and because this secondary peak was only present when motor response was required, we interpreted the secondary peak as an artifact of that motor response. The interaction of stimulus mode and target presence was not significant, F(1,23) 5 0.351, p 5 .6. The larger dilations evoked by aural task presentation persist whether a target is present or absent (see Figure 4).
Change in pupil diameter (mm)
0.4 aural (78 trials) visual (180 trials)
0.3 0.2 0.1 0.0 –0.1
possible target 0
5
possible target 10 Time (seconds)
possible target 15
20
Figure 3. Average pupil dilations evoked by a vigilance task presented aurally and visually. The vertical gray bars show the moments at which participants were vigilant for mistakes in a counting sequence (‘‘targets’’). The aurally presented task led to larger dilations, but the two presentation modes elicited dilation profiles with similar shape and timing.
330
J. Klingner et al. Target Present
Change in pupil diameter (mm)
Target Absent 0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
–0.1
–0.1 –4
–2 0 Time (seconds)
2
–4
–2 0 Time (seconds)
2
Figure 4. Target effect on pupil dilations evoked by heightened vigilance. The data shown are the same as those in Figure 3. Each trial of Experiment 3 had three moments at which we told participants to expect possible targets, which occurred independently at each moment with probability one half. The chart on the left shows the mean dilation in moments in which a target did not occur, and the chart on the right shows the mean dilation in moments when a target did occur. Targets elicited longer and larger pupil dilations, with a secondary peak about 1.5 s after target presentation. This secondary peak corresponds to the motor activity of responding to the target’s presence. Whether a target was present or absent, dilations were larger in the aural condition, and the peak dilation under aural presentation occurred about half a second later than with visual presentation.
Task Performance Participants made more errors in the counting vigilance task when it was presented aurally (8.5%) than visually (6.1%), but this difference was not significant, w2(1, N 5 774) 5 7.80, p 5 .14. Discussion This experiment compared the cognitive load under aurally and visually presented intermittent vigilance tasks. As with the other two tasks we studied, the two presentation modes elicited pupil dilations with very similar timing and overall shape, and although we did not observe a significant performance difference, visual presentation caused lower cognitive load. In addition to the presentation mode effect, we also observed that the presence of targets was associated with larger pupil dilations. This difference is consistent with the additional cognitive demand of pushing the button in cases when the target is present.
GENERAL DISCUSSION Summary of Experiments In our first experiment, participants completed mental multiplication problems either spoken aloud or displayed on a computer screen. Our second experiment examined the digit span shortterm recall task, again presented both aurally and visually, and our third experiment considered a speeded-reaction vigilance task that did not rely heavily on working memory. In all tasks, we controlled the stimulus timing between the two modes as well as controlling all aspects of the visual fieldFbrightness, contrast, and participant fixationFin order to minimize noncognitive pupillary reactions.
Summary of Findings We found that the pupil dilations evoked by all three tasks were qualitatively similar under auditory and visual presentation, but that auditory presentation led to larger pupillary dilations.
Qualitative Match In all three of our experiments, we observed that pupil dilations in both modes had about the same onset timing, duration, and overall shape (see Figures 1, 2, and 3). Additionally, in the two tasks that replicated classic pupillary response studies, mental multiplication (Hess & Polt, 1964) and digit span (Kahneman & Beatty, 1966), we also found a qualitative match between the dilations we observed and the aural-only classic results. Both of these qualitative correspondencesFvisual to aural in our experiments and visual to classic aural findingsFsuggest that the pupil dilations we observed to visually presented tasks reflect the cognitive demands of the tasks and were generally free of distortion caused by noncognitive pupillary reactions to brightness or contrast changes. Quantitative Difference In all three of our experiments, we observed significantly larger pupillary dilations when we presented tasks aurally than when we presented them visually. The differences were 0.19 mm (0.35 mm vs. 0.16 mm) for mental multiplication, 0.18 mm (0.43 mm vs. 0.25 mm) for digit span memory, and 0.08 mm (0.23 mm vs. 0.15 mm) for vigilance.
Implications Because we were careful to control for noncognitive pupillary responses caused by brightness, contrast, and so forth and because of our finding of a qualitative match in dilation trajectories between conditions, we believe that the difference in magnitude between the two conditions was a result of differences in cognitive load. We therefore interpret this result as evidence that visual task presentation leads to lower cognitive load than auditory presentation across all three of the tasks we studied. This finding contradicted our hypothesis that similar task demands would lead to similar magnitude dilations in the two cases, perhaps with an initially smaller dilation under visual presentation caused by the lesser difficulty of seeing versus hearing numbers. Instead, we found that aural task presentation led to
Effect of task presentation mode on pupil dilation
331
larger pupil dilation not only during initial stimulus comprehension but also throughout task completion. Taken together with the better performance we observed in the visual conditions, this finding indicates that visual presentation facilitates processing for all three tasks. That is, comprehending and remembering numbers is easier when they are seen than when they are heard. Relation to Prior Digit Span Findings In the case of digit span, our finding of an advantage for visual presentation seemed to contradict prior studies that found better performance under aural task presentation. Improved recall of heard numbers relative to seen numbers is very well established (Greene, 1992, p. 22; Penney, 1989; but see Beaman, 2002). Indeed, in our measurements of error rates, we found that although visual presentation led to significantly greater overall performance, the difference was not large, and rates of recall for the longer sequences and average digit span scores suggest a small performance advantage for aural presentation, as was found in the cited investigations. This apparent contradiction between lower cognitive load under visual presentation and superior recall of heard numbers can perhaps be resolved by drawing a distinction between levels of effort and levels of performance (cf. Paas & Van Merrie¨nboer, 1994). Although performance was better for heard numbers, our pupillary data suggest that this greater performance may have come with the cost of greater effort and cognitive load. Relation to Prior Mental Arithmetic Findings Prior investigations of mental arithmetic have not often addressed the effect of stimulus mode. In a study of the relative importance of different components of working memory in serial mental addition, Logie, Gilhooly, and Wynn (1994) observed that visual problem presentation led to better performance and less degradation in the context of a variety of interfering tasks. Our finding of better performance in the visual case matches theirs. They concluded that the central executive, the visuospatial store, and subvocal rehearsal are all involved in mental arithmetic. Taken together with these data, our finding of lower cognitive load in the visual case suggests that visual presentation facilitates mental arithmetic performance by aiding the recruitment of all three of these components of working memory. This possibility is supported by recent functional magnetic resonance imaging data collected by Fehr, Code, and Herrmann (2008), who found that presentation mode can significantly impact which regional
neuronal networks are employed in the calculation process for mental arithmetic. Conclusion It is well known that visual presentation can lead to higher performance on complicated tasks such as schema learning (Clark & Paivio, 1991) and finding patterns in data (Chen, 2004). Such advantages are typically attributed to the benefits of a persistent external representation that reduces load on working memory. Our finding of a visual advantage even for simple tasks and even though we controlled presentation duration, displaying the digits exactly as long as they took to speak, suggests that something besides visual persistence underlies this visual advantage. One account for superior performance under visual rather than auditory presentation rests on the role of dual codes in working memory (e.g., Baddeley, 2007; Paivio, 1990). Visual presentation is likely to encourage dual coding of the stimuli (e.g., Paivio, 1990). Extensive research has shown that having two mental representations for something, notably, both visual and verbal, is better for memory than having one. If one internal representation is lost or corrupted, the other can compensate. People tend to spontaneously name visual stimuli, but they do not spontaneously generate visual images to verbal stimuli, so that visual presentation is more likely to generate two codes than verbal presentation. The existence of two codes could facilitate information processing in addition to augmenting memory. Mental operations like arithmetic are regarded as performed by the articulatory loop. If memory for the stimuli is retained in the visuospatial sketchpad, then the articulatory loop, relieved of memory load, has more capacity for information processing. These findings, if replicated and extended, have broad-ranging implications for education as well as interface design. Alternatively, it is possible that the greater effort required by aural presentation is due only to differences in the difficulty of perception and not because of any subsequent processing differences, such as visual persistence or differential recruitment of working memory components. Future work could resolve this question by adjusting stimulus discriminability to equalize perception difficulty between the two modes and then check to see whether the effort differences remain. Further research to determine the true cause of mode-related differences in pupil dilations will help to determine whether such dilations can fulfill Kahneman’s second criterion for an effort proxy, intertask comparability, and thus be useful for comparisons of cognitive load between the auditory and visual domains.
REFERENCES Ahern, S., & Beatty, J. (1979). Pupillary responses during information processing vary with scholastic aptitude test scores. Science, 205, 1289–1292. Andreassi, J. L. (2006). Pupillary response and behavior. In Psychophysiology: Human behavior and physiological response (5th ed, pp. 289–307). Mahwah, NJ: Erlbaum. Backs, R. W., & Walrath, L. C. (1992). Eye movement and pupillary response indices of mental workload during visual search of symbolic displays. Applied Ergonomics, 23, 243–254. Baddeley, A. D. (2007). Working memory, thought, and action. Oxford: Oxford University Press. Beaman, C. P. (2002). Inverting the modality effect in serial recall. Quarterly Journal of Experimental Psychology, A55, 371–389. Beatty, J. (1982a). Phasic not tonic pupillary responses vary with auditory vigilance performance. Psychophysiology, 19, 167–172.
Beatty, J. (1982b). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91, 276–292. Beatty, J., & Kahneman, D. (1966). Pupillary changes in two memory tasks. Psychonomic Science, 5, 371–372. Beatty, J., & Lucero-Wagoner, B. (2000). The pupillary system. In J. T. Cacioppo, L. G. Tassinary, & G. Berntson (Eds.), Handbook of psychophysiology (2nd ed, pp. 142–162). Cambridge, UK: Cambridge University Press. Becker, W., Iwase, K., Ju¨rgens, R., & Kornhuber, H. H. (1976). Brain potentials preceding slow and rapid hand movements. In W. C. McCallum & J. R. Knott (Eds.), The responsive brain (pp. 99–102). Bristol, UK: Wright. Boersma, F., Wilton, K., Barham, R., & Muir, W. (1970). Effects of arithmetic problem difficulty on pupillary dilation in normals and
332 educable retardates. Journal of Experimental Child Psychology, 9, 142–155. Bradshaw, J. (1967). Pupil size as a measure of arousal during information processing. Nature, 216, 515–516. Bradshaw, J. L. (1968). Pupil size and problem solving. Quarterly Journal of Experimental Psychology, 20, 116–122. Bradshaw, J. L. (1969). Background light intensity and the pupillary response in a reaction time task. Psychonomic Science, 14, 271–272. Bradshaw, J. L. (1970). Pupil size and drug state in a reaction time task. Psychonomic Science, 18, 112–113. Chen, C. (2004). Information visualization: Beyond the horizon. London: Springer Verlag. Clark, J., & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 3, 149–210. Cocker, K. D. (1996). Development of pupillary responses to grating stimuli. Ophthalmic and Physiological Optics, 16, 64–67. Dabbs, J. M., & Milun, R. (1999). Pupil dilation when viewing strangers: Can testosterone moderate prejudice? Social Behavior and Personality, 27, 297–301. Fehr, T., Code, C., & Herrmann, M. (2008). Auditory task presentation reveals predominantly right hemispheric fMRI activation patterns during mental calculation. Neuroscience Letters, 431, 39–44. Goldwater, B. C. (1972). Psychological significance of pupillary movements. Psychological Bulletin, 77, 340–355. Granholm, E., Asarnow, R. F., Sarkin, A. J., & Dykes, K. L. (1996). Pupillary responses index cognitive resource limitations. Psychophysiology, 33, 457–461. Greene, R. L. (1992). Human memory. Mahwah, NJ: Erlbaum. Handy, T. C. (2004). Event-related potentials: A methods handbook (1st ed). Cambridge, MA: MIT Press. Heitz, R. P., Schrock, J. C., Payne, T. W., & Engle, R. W. (2008). Effects of incentive on working memory capacity: Behavioral and pupillometric data. Psychophysiology, 45, 119–129. Hess, E. H., & Polt, J. M. (1964). Pupil size in relation to mental activity during simple problem-solving. Science, 143, 1190–1192. Huynh, H., & Feldt, L. S. (1980). Performance of traditional f tests in repeated measures designs under covariance heterogeneity. Communications in Statistics: Theory and Methods, 9, 61–74. Jennings, J. R. (1987). Editorial policy on analyses of variance with repeated measures. Psychophysiology, 24, 474–475. Just, M. A., & Carpenter, P. A. (1993). The intensity dimension of thought: Pupillometric indices of sentence processing. Canadian Journal of Experimental Psychology, 47, 310–339. Kahneman, D. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice Hall. Kahneman, D., & Beatty, J. (1966). Pupil diameter and load on memory. Science, 154, 1583–1585. Kahneman, D., Onuska, L., & Wolman, R. E. (1968). Effects of grouping on the pupillary response in a short-term memory task. Quarterly Journal of Experimental Psychology, 20, 309–311. Kahneman, D., & Wright, P. (1971). Changes of pupil size and rehearsal strategies in a short-term memory task. Quarterly Journal of Experimental Psychology, 23, 187–196. Klingner, J., Kumar, R., & Hanrahan, P. (2008). Measuring the taskevoked pupillary response with a remote eye tracker. In Proceedings of the 2008 Symposium on Eye Tracking Research & Applications (pp. 69–72). Savannah, GA: ACM. Kohn, M., & Clynes, M. (1969). Color dynamics of the pupil. Annals of the New York Academy of Sciences, 156, 931–950. Libby, W. L., Lacey, B. C., & Lacey, J. I. (1973). Pupillary and cardiac activity during visual attention. Psychophysiology, 10, 270–294. Loewenfeld, I. (1999). The pupil: Anatomy, physiology, and clinical applications (2nd ed., Vol. 1). Oxford, UK: Butterworth-Heinemann. Loewy, A. D. (1990). Autonomic control of the eye. In A. D. Loewy & K. M. Spyer (Eds.), Central regulation of autonomic functions (pp. 268–285). New York: Oxford University Press. Logie, R. H., Gilhooly, K. J., & Wynn, V. (1994). Counting on working memory in arithmetic problem solving. Memory & Cognition, 22, 395–410. Marshall, S. (2002). The index of cognitive activity: Measuring cognitive workload. In Proceedings of the 2002 IEEE 7th Conference on Human Factors and Power Plants, 2002 (pp. 7–5–7–9). Miettinen, O., & Nurminen, M. (1985). Comparative analysis of two rates. Statistics in Medicine, 4, 213–226.
J. Klingner et al. Misulis, K. E., & Fakhoury, T. (2001). Spehlmann’s evoked potential primer, 3/e (3rd ed). Oxford, UK: Butterworth-Heinemann. Moresi, S., Adam, J. J., Rijcken, J., Van Gerven, P. W., Kuipers, H., & Jolles, J. (2008). Pupil dilation in response preparation. International Journal of Psychophysiology, 67, 124–130. Paas, F., & Van Merrie¨nboer, J. (1994). Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review, 6, 351–371. Paivio, A. (1990). Mental representations: A dual coding approach. Oxford, UK: Oxford University Press. Peavler, W. S. (1974). Pupil size, information overload, and performance differences. Psychophysiology, 11, 559–566. Penney, C. G. (1989). Modality effects and the structure of short-term verbal memory. Memory & Cognition, 17, 398–422. Pomplun, M., & Sunkara, S. (2003). Pupil dilation as an indicator of cognitive workload in human-computer interaction. In D. Harris, V. Duffy, M. Smith, & C. Stephanidis (Eds.), Human-Centred Computing: Cognitive, Social, and Ergonomic Aspects. Vol. 3 of the Proceedings of the 10th International Conference on Human-Computer Interaction, HCI 2003, Crete, Greece. Porter, G., Troscianko, T., & Gilchrist, I. D. (2007). Effort during visual search and counting: Insights from pupillometry. Quarterly Journal of Experimental Psychology, 60, 211–229. Richer, F., & Beatty, J. (1985). Pupillary dilations in movement preparation and execution. Psychophysiology, 22, 204–207. Richer, F., Silverman, C., & Beatty, J. (1983). Response selection and initiation in speeded reactions: A pupillometric analysis. Journal of Experimental Psychology: Human Perception and Performance, 9, 360–370. Sahraie, A., & Barbur, J. L. (1997). Pupil response triggered by the onset of coherent motion. Graefe’s Archive for Clinical and Experimental Ophthalmology, 235, 494–500. Tobii Technologies, Inc. (2007). Tobii 1750. Stockholm, Sweden: Author. Ukai, K. (1985). Spatial pattern as a stimulus to the pupillary system. Journal of the Optical Society of America, A2, 1094–1100. Van Orden, K. F., Limbert, W., Makeig, S., & Jung, T. (2001). Eye activity correlates of workload during a visuospatial memory task. Human Factors, 43, 111–121. Verney, S. P., Granholm, E., & Dionisio, D. P. (2001). Pupillary responses and processing resources on the visual backward masking task. Psychophysiology, 38, 76–83. Welford, W. T. (1980). Reaction times. London: Academic Press.
SUPPORTING INFORMATION Additional Supporting Information may be found in the online version of this article: Appendix S1. Code for analyzing data. Source code written in R and Python used to organize, clean, analyze, and graph the data. Appendix S2. Code for collecting data. Source code written in Java used to capture and record the pupil and gaze direction measurements made by the eye tracker and synchronize their timestamps with the timing of stimulus presentation and participants’ responses. Appendix S3. Code for presenting stimuli. Source code written in Java used to present the visual and aural stimuli. Appendix S4. Code for running eye tracker. Source code written in C# used to initialize and calibrate the Tobii 1750 eye tracker. Appendix S5. Data. All raw data collected in all experiments, including failed and discarded trials, anonymized. Please note: Wiley-Blackwell are not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article. (Received August 3, 2009; Accepted March 29, 2010)
Psychophysiology, 48 (2011), 333–336. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01079.x
BRIEF REPORT
An electrophysiological measure of visual short-term memory capacity within and across hemifields
JEAN-FRANCOIS DELVENNE,a LAURA A. KADDOUR,a and JULIE CASTRONOVOa,b a
Institute of Psychological Sciences, University of Leeds, United Kingdom Unite´ de Neurosciences Cognitives, University of Louvain, Belgium
b
Abstract Recent electrophysiological research has identified a specific contralateral correlate of the contents of visual short-term memory (VSTM). This posterior contralateral delay activity (CDA) persists during the retention period, and its amplitude is modulated by the number of items held at any one time. Here we show that, despite the contralateral nature of this activity, its amplitude is modulated by the number of objects from both hemifields. When objects from one side of the visual field are held in memory, CDA activity increased for arrays of one, two, and three objects, but stopped getting larger for arrays of four objects. However, when objects from both sides are memorized at the same time, CDA activity reached its asymptotic limit for arrays of two objects per side. These results suggest that, in spite of being contralaterally organized, VSTM is limited by the number of objects from both hemifields. Descriptors: Visual working memory, Visual short-term memory, Capacity limits, Bilateral presentation, CDA, Eventrelated potentials
the amplitude of the CDA increased progressively (i.e., became more negative) with the number of items to be remembered, reaching an asymptotic limit at each individual’s estimated VSTM capacity, typically around three–four objects. The contralateral organization of visual memories raises the possibility that each hemisphere has its own capacity of storage. Delvenne (2005) began to explore this hypothesis in a change detection study. He asked participants to memorize a number of colored squares that were either presented all within a single hemifield or distributed across the left and right hemifields. After a short retention interval, a test array was presented and participants were asked to decide whether the test array was the same as or different from the memory array by the identity of one object. Change detection performance was found to be similar in both conditions, suggesting that the capacity of VSTM does not benefit from the bilateral field presentation of the objects and that the two cerebral hemispheres share the same resources for storage capacity. However, the problem with change detection performance is that it is not a direct measure of VSTM capacity. It reflects what is available for comparison after the onset of the test array, which can lead to underestimation of the number of items in VSTM (e.g., Awh, Barton, & Vogel, 2007; Delvenne, Cleeremans, & Laloyaux, 2010). In the present study, we investigated the capacity of VSTM within and across hemifields by recording ERPs during VSTM maintenance. The excellent temporal resolution provided by this method allowed us to track the on-line maintenance of visual information in short-term memory, prior to the onset of the test array, thus without incurring the problem outlined above. We
Visual short-term memory (VSTM) is extremely limited in capacity, holding information about only three or four items at any one time (e.g., Luck & Vogel, 1997). One unresolved issue about VSTM capacity is whether it is affected by the spatial distribution of the objects in the two hemifields. The human brain consists of two hemispheres, each receiving the visual input from the contralateral hemifield. Visual memories are also assumed to be contralaterally organized. For instance, recent event-related potentials (ERPs) studies have identified a specific contralateral electrophysiological correlate of the contents of VSTM (e.g., McCollough, Machizawa, & Vogel, 2007; Vogel & Machizawa, 2004). In those studies, participants were presented with visual items on both sides of a fixation and were cued to memorize the items of either the left or right side. A sustained posterior negative wave was observed throughout the memory retention period, which was larger over the contralateral side of the brain (with respect to the position of the memory items in the visual field) relative to the ipsilateral side. This contralateral delay activity (CDA) is assumed to reflect the representations held in VSTM. Importantly for present purposes, Vogel and colleagues (McCollough et al., 2007; Vogel & Machizawa, 2004) found that
This research was supported by the Experimental Psychology Society, UK, and the Belgian National Fund for Scientific Research. We thank Dean Salisbury and two anonymous reviewers for their constructive comments and suggestions. Address correspondence to: Jean-Francois Delvenne, Institute of Psychological Sciences, University of Leeds, Leeds, LS2 9JT, United Kingdom. E-mail:
[email protected] 333
334
J.-F. Delvenne et al.
measured CDA amplitude while participants remembered colored squares from one hemifield (unilateral condition) or from both hemifields (bilateral condition). If each cerebral hemisphere has its own storage capacity, the amplitude of the CDA should be modulated by the number of remembered objects from the corresponding hemifield only, regardless of the number of remembered objects from the other hemifield. For example, if the CDA amplitude reaches its maximum at set size three or four in the unilateral condition (e.g., McCollough et al., 2007; Vogel & Machizawa, 2004), it may also reach its asymptotic limit at set size three or four in the bilateral condition even though additional objects from the ipsilateral hemifield are also memorized. In contrast, if the two hemispheres share the same capacity, the CDA amplitude should be modulated by the total number of remembered objects from both hemifields. So, if the CDA amplitude reaches its maximum at set size three or four in the unilateral condition, it may reach its asymptotic limit at approximately set size two in the bilateral condition where two objects are presented in each side (i.e., four objects in total).
Method Participants Nineteen paid volunteers with normal (self-reported) or corrected-to-normal vision took part in the experiment. Four participants were excluded from the analyses due to too many artifacts in their electroencephalogram (EEG), leaving a final sample of 15 participants (5 females; mean age of 22.7, SD 5 2.7). Stimuli and Procedure Visual stimuli were displayed on a gray background (127 on the 0–255 scale of red, green, and blue phosphors) on a 17-inch cathode ray tube monitor. The experimental script was generated by E-prime software (Psychology Software Tools, Inc.; www. pstnet.com/eprime), and responses were collected from the computer keyboard. The stimuli were colored squares subtending approximately 0.611 ! 0.611 of visual angle at a viewing distance of approximately 60 cm. The squares were randomly positioned within 4.21 ! 7.11 regions that were centered 3.21 to the left and right of a central fixation cross with the constraint
that the distance between the squares was more than 1.61 (center to center). The color of each square was selected at random from a set of eight different colors produced by permutation of the presence (255 on the 0–255 scale) or absence (0 on the 0–255 scale) of red, green, and blue phosphors. Each memory array consisted of 1, 2, 3, or 4 colored squares on both sides of fixation, and a given color could not repeat within the same hemifield. On each trial, participants were shown a central fixation cross of variable duration (600 to 700 ms), followed by a cue for 200 ms that instructed the participants to remember the stimuli of either the left or the right hemifield (in the unilateral condition) or both hemifields (in the bilateral condition). A single arrow was used as the cue in the unilateral condition, whereas a double arrow was used in the bilateral condition (see Figure 1). After a blank interval of variable duration (300 to 400 ms), the memory array was presented for 100 ms, followed by a 900-ms blank delay, after which the test array was presented for 2000 ms. The test array contained the stimuli from one hemifield only (the cued hemifield in the unilateral condition; the left or the right hemifield in the bilateral condition), so the number of comparisons to be made was comparable between the two conditions. The position of the stimuli remained the same between the memory and test arrays. Participants were instructed to keep their eyes fixated at the center of the screen while remembering the squares in the cued hemifield(s). They were asked to indicate as accurately as possible whether the stimuli in the test array were the same as, or different from, the stimuli in the memory array by pressing one of two keyboard keys. On half of the trials, the color of one square in the test array was different from the memory array. The new color value was selected at random from the remaining color values not previously allocated to any square from the same hemifield. Trials were blocked by condition (unilateral versus bilateral) and alternated throughout the experiment. Each condition consisted of 10 blocks of 64 trials. As a result, the entire experiment consisted of a total of 1280 trials (i.e., 2 conditions [unilateral, bilateral] ! 4 set sizes [1, 2, 3, 4] ! 2 sides [left, right] ! 80 trials) and lasted approximately 90 min. EEG Recordings and Analysis The EEG was recorded from a 64-channel ‘Quick-cap’ and amplified by SynAmps (NeuroScan Inc., El Paso, TX) using a sampling rate of 1000 Hz and a bandpass of 0.01–100 Hz. All
Figure 1. Sequence of events in the unilateral and bilateral conditions.
VSTM capacity across hemifields impedances were below 10 kO. The EEG data were re-referenced offline to the average of the left and right mastoids, and eyes movements were corrected using the algorithm incorporated in the Neuroscan v. 4.3 software package (Semlitsch, Anderer, Schuster, & Presslich, 1986). The ERPs were time-locked to 100 ms prior to the onset of the memory array and extended through the retention period (i.e., 900 ms) until the test array was presented. Pre-stimulus baselines were corrected, and trials containing artifacts exceeding ! 80 mV in amplitude were discarded for further analyses, which accounted for the exclusion of an average of 6% of trials. Participants with trial rejection rate of more than 20% were excluded from the sample. In the unilateral condition, the contralateral activity referred to the activity recorded at left hemisphere electrode sites when participants were cued to remember the right side of the memory array and to the activity recorded at right hemisphere electrode sites when participants were cued to remember the left side of the memory array. The ipsilateral activity referred to the activity recorded at left hemisphere electrode sites when participants were cued to remember the left side of the memory array and to the activity recorded at right hemisphere electrode sites when participants were cued to remember the right side of the memory array. CDA was then computed by subtracting the ipsilateral activity from the contralateral activity. In the bilateral condition, where participants were cued to remember both sides of the memory array, all the brain activity was regarded as contralateral. Therefore, in order to compute CDA, the ipsilateral activity of the unilateral condition was used and subtracted from the activity of the bilateral condition. Note that, because only half of the brain activity was ipsilateral in the unilateral condition, we used half of the brain activity in the bilateral condition to obtain the closest counterpart. More precisely, the contralateral activity in the bilateral condition referred to the activity recorded at left hemisphere electrode sites when the test array contained the stimuli in the right hemifield and to the activity recorded at right hemisphere electrode sites when the test array contained the stimuli in the left hemifield. Finally, CDA was averaged across thirteen posterior lateralized electrodes (posterior to the Cz plane: O1/O2, CB1/CB2, PO3/PO4, PO5/PO6, PO7/PO8, P1/ P2, P3/P4, P5/P6, P7/P8, CP1/CP2, CP3/CP4, CP5/CP6, TP7/ TP8) and smoothed at low-pass filter of 20 Hz (6 dB). Results Behavioral Data In all analyses, Greenhouse-Geisser corrections for nonsphericity were applied where appropriate. Change detection accuracy for set sizes one, two, three, and four were 95.4%, 91.2%, 81.4%, and 76%, respectively, in the unilateral condition, and 93.5%, 80.1%, 71.1%, 67.7%, respectively, in the bilateral condition. The condition " set size analysis of variance (ANOVA) (repeated measures) revealed a significant condition effect, F(1,14) 5 75.20, po.001, with the best performance in the unilateral condition, a significant set size effect, F(3,42) 5 83.99, po.001, where accuracy declined as the set size increased, and a significant interaction of these factors, F(3,42) 5 8.01, po.001. The interaction showed a faster decline in accuracy in the bilateral condition as the set size increased. ERP Data We first conducted a 2-way ANOVA on the mean amplitudes of activity in a window of 300–900 ms relative to the onset of the
335
Figure 2. Grand-averaged ERP difference waves at posterior electrode sites for arrays of one, two, three, and four items per hemifield obtained in (a) the unilateral and (b) bilateral conditions.
memory array with the factors of memory condition (rememberleft, remember-right) and hemisphere (left, right) in the unilateral condition. The results revealed no main effects of either memory condition or hemisphere (both ps4.9), suggesting that both hemispheres produce equivalent activity when the memory items were displayed in the contralateral visual field. However, a significant interaction of those factors was found, F(1,14) 5 7.6, po.02, supporting the contralateral organization of visual memories. In addition, we asked whether the activity in the bilateral condition has the same scalp distribution as the activity in the unilateral condition. In an ANOVA, we found no interaction between electrode site and condition, F(61,864) 5 1.01, p 5 .462, indicating that the activity in both conditions has the same scalp distribution. Figure 2 shows the averaged CDA for arrays of one, two, three, and four in the unilateral and bilateral conditions. Consistent with previous research (McCollough et al., 2007; Vogel & Machizawa, 2004), the amplitude of the CDA increased as a function of set size. However, the differences appear to decrease towards the end of the memory delay period. We therefore divided the 300–900 ms period into two time windows (300–600 ms and 600–900 ms) and conducted a repeated measures ANOVA with the factors of time window, condition (unilateral, bilateral), hemisphere (left, right), and set size on the mean CDA amplitude. This analysis revealed a main effect of time window, F(1,14) 5 5.30, po.05, a main effect of set size, F(3,42) 5 3.36, po.05, and a significant interaction of these two factors, F(3,42) 5 5.56, po.005. The other aspects were all non-significant. The set size effect was then analyzed in more detail within each condition and within each time window. In the unilateral
336
J.-F. Delvenne et al.
condition, pairwise comparisons yielded significance differences in amplitude between set sizes of one, two, and three (po.05), but no difference between set sizes of three and four (p4.70) in the 300–600 ms period, whereas the difference between set sizes of one and two did not reach the significance level in the 600–900 ms period (p4.77). These ERP results in the unilateral condition clearly showed that the amplitude of CDA increases as a function of set size, reaching a limit with arrays of approximately three items per side. In the bilateral condition, however, the amplitude of the CDA reaches a limit with arrays of approximately two items per side. Indeed, pairwise comparisons yielded a significant difference in amplitude between set sizes of one and two (po.05), but no differences between set sizes of two, three, and four (p4.77) in the 300–600 ms period. In the 600–900 ms period, no difference between any set sizes was observed (p40.47). Discussion The goal of the present study was to measure the capacity of VSTM for objects presented unilaterally and bilaterally using human electrophysiology. To this end, we used the CDA as an electrophysiological marker of VSTM capacity. Recent research has shown that the amplitude of the CDA is strongly modulated by the number of items that an individual is holding in memory at the same time, reaching an asymptotic limit for arrays of approximately three–four items (McCollough et al., 2007; Vogel & Machizawa, 2004). Here, we show that the amplitude of the
CDA is modulated by the total number of items held in memory, independently of their spatial distribution in the visual field. When individuals had to remember one side of the memory array, the CDA activity increased for arrays of one, two, and three items, but ceased to get larger for arrays of four items. However, when individuals had to memorize the items from both sides of the memory array, this contralateral activity reached its asymptotic limit for arrays of two items per side. These results provide the first electrophysiological evidence that VSTM capacity is limited by the number of objects from both hemifields. VSTM may consist of a pool of resources that can be allocated flexibly to one or both hemifields and allow a maximum of three–four objects to be maintained simultaneously. Further research needs to be carried out with other types of stimuli to test the generalization of this conclusion. The CDA is a powerful tool to measure VSTM capacity as it can track its on-line content. The contralateral nature of this activity strongly suggests that the representations in VSTM are organized retinotopically. The information from the left visual field is stored in the right hemisphere, and the information from the right hemifield is represented in the left hemisphere. However, the current findings suggest that this organization does not benefit VSTM capacity. Indeed, the CDA waveform was found to be sensitive to the total number of objects presented in both hemifields. Hence, despite the contralateral nature of this activity, the present study demonstrates that, when used appropriately, it is possible to take CDA as an index to diagnose what is stored in VSTM bilaterally.
REFERENCES Awh, E., Barton, B., & Vogel, E. K. (2007). Visual working memory represents a fixed number of items, regardless of complexity. Psychological Science, 18, 622–628. Delvenne, J.-F. (2005). The capacity of visual short-term memory within and between hemifields. Cognition, 96, B79–B88. Delvenne, J.-F., Cleeremans, A., & Laloyaux, C. (2010). Feature bindings are maintained in visual short-term memory without sustained focused attention. Experimental Psychology, 57, 108–116. Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 390, 279–281.
McCollough, A. W., Machizawa, M. G., & Vogel, E. K. (2007). Electrophysiological measures of maintaining representations in visual working memory. Cortex, 43, 77–94. Semlitsch, H., Anderer, P., Schuster, P., & Presslich, O. (1986). A solution for reliable and valid reduction of ocular artifacts applied to the P300 ERP. Psychophysiology, 23, 695–703. Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature, 428, 748–751. (Received December 1, 2009; Accepted May 8, 2010)
Psychophysiology, 48 (2011), 337–349. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01075.x
The effect of musical experience on emotional self-reports and psychophysiological responses to dissonance
DELPHINE DELLACHERIE,a,b MATHIEU ROY,c LAURENT HUGUEVILLE,d,e,f ISABELLE PERETZ,c and SE´VERINE SAMSONa,b a
Laboratoire de Neurosciences Fonctionnelles et Pathologies, CNRS-UMR 8160, University of Lille-Nord de France, Lille, France Epilepsy Unit, La Salpeˆtrie`re Hospital, Paris, France BRAMS Laboratory, University of Montre´al, Montre´al, Que´bec, Canada d Universite´ Pierre et Marie Curie-Paris 6, Centre de Recherche de l’Institut du Cerveau et de la Moelle Epinie`re, UMR-S975, Paris, France e Inserm, U975, Paris, France f CNRS, UMR 7225, Paris, France b c
Abstract To study the influence of musical education on emotional reactions to dissonance, we examined self-reports and physiological responses to dissonant and consonant musical excerpts in listeners with low (LE: n 5 15) and high (HE: n 5 13) musical experience. The results show that dissonance induces more unpleasant feelings and stronger physiological responses in HE than in LE participants, suggesting that musical education reinforces aversion to dissonance. Skin conductance (SCR) and electromyographic (EMG) signals were analyzed according to a defense cascade model, which takes into account two successive time windows corresponding to orienting and defense responses. These analyses suggest that musical experience can influence the defense response to dissonance and demonstrate a powerful role of musical experience not only in autonomic but also in expressive responses to music. Descriptors: Emotion, Music, Dissonance, Musical experience, Psychophysiology, SCR, EMG, HR
isolation (Tramo et al., 2001). This phenomenon should be distinguished from ‘‘musical’’ dissonance, which refers to dissonance manipulated by composers in a harmonic or melodic context in order to induce tension and expectation effects (Meyer, 1956). In the present study, we did not consider musical dissonance but focused rather on sensory dissonance. According to neuropsychological, electrophysiological, and brain imaging studies, it has been suggested that perceptual and emotional processing of dissonance depends on distinct cerebral structures. Whereas perception of dissonance requires the function of auditory cortical areas within the superior temporal gyrus (Fishman, Volkov, Noh, Garell, Bakken, et al., 2001; Peretz, Blood, Penhune, & Zatorre, 2001), its emotional processing involves various brain structures including mesial temporal lobe and precuneus regions (Blood, Zatorre, Bermudez, & Evans, 1999; Gosselin, Samson, Adolphs, Noulhiane, Roy, et al., 2006; Koelsch, Fritz, Von Cramon, Muller, & Friederici, 2006). At the behavioral level, it is well known that dissonant chords induce more negative valence judgments than consonant chords (Blood et al., 1999; Brattico, Pallesen, Varyagina, Bailey, Anourova, et al., 2009; Gosselin et al., 2006; Khalfa, Roy, Rainville, Dalla Bella, & Peretz, 2008; Koelsch et al., 2006; Koelsch, Remppis, Sammler, Jentschke, Mietchen, et al., 2007; Pallesen, Brattico, Bailey, Korvenoja, Koivisto, et al., 2005; Passynkova, Neubauer, & Scheich, 2007; Peretz et al., 2001; Sammler, Grigutsch, Fritz, & Koelsch, 2007; Schoen, Regnault, Ystad, & Besson, 2005). Even infants prefer consonance to dissonance, suggesting a natural
Since Helmholtz (1877), musical dissonance has been considered by psychologists and neuroscientists as a puzzling topic at the interface between auditory perception and emotional processing. The simplest definition of dissonance refers to an unpleasant sensation induced by the simultaneous presentation of two sounds. As proposed by Pythagoras two and half millennia ago, this affective sensation produced by a pair of tones is related to the ratio of the fundamental frequencies of the sounds that are played together. When the ratio is simple, consonance is heard (e.g., octave 2/1; perfect fifth 3/2) and when the ratio is more complex, dissonance occurs (e.g., tritone chord, 45/32) (Plomp & Levelt, 1965; Tramo, Cariani, Delgutte, & Braida, 2001). This physical component of dissonance results in a perception of beating that is perceived as unpleasant and is referred to as ‘‘sensory’’ dissonance, which appears when chords are played in The authors are grateful to Laurence Conty, Daniela Sammler, Se´verine Farley, and Sean Hutchins for their helpful assistance and comments on previous versions of the manuscript. This study was supported by a PhD scholarship from the Regional Council of Nord-Pas de Calais to Delphine Dellacherie and by a grant from ‘‘Agence Nationale pour la Recherche’’ of the French Ministry of Research (project no. NT05-345987) to Se´verine Samson and from Eisai Inc. Isabelle Peretz is supported by grants from the Canada Research Chair in Neurocognition of Music and Mathieu Roy by a Canadian fellowship from NSERC. Address correspondence to: Se´verine Samson, Department of Psychology, Universite´ de Lille 3, BP 60 149, 59653 Villeneuve d’Ascq Cedex, France. E-mail:
[email protected] 337
338 aversion to dissonance very early in life (Masataka, 2006; Trainor & Heinmiller, 1998; Trainor, Tsang, & Cheung, 2002; Zentner & Kagan, 1996). However, it has also been suggested that emotional responses to dissonance can be influenced by musical experience. As reported by Pallesen et al. (2005) and Schoen et al. (2005), adults with musical experience considered dissonant chords as more unpleasant than did musically untrained individuals. These self-reported results suggest that aversion to dissonance could in some instances be reinforced by formal musical training and possibly by learning mechanisms. Further lines of evidence suggest that, at the neurophysiological level, musical experience could facilitate the processing of dissonance in musicians in comparison to nonmusicians (Brattico et al., 2009; Minati, D’Incerti, Pietrocini, Valentini, Scaioli, et al., 2009; Regnault, Bigand, & Besson, 2001; Schoen et al., 2005). However, the effect of musical education on emotional responses to dissonance is not a well-established phenomenon (Brattico et al., 2009). To test the effect of musical experience in emotional responses to dissonance, we recorded psychophysiological correlates of musical listening, which is an appropriate method to examine objective measures of emotional experiences (Bradley, Codispoti, Cuthbert, & Lang, 2001; Bradley & Lang, 2000; Lang, Bradley, & Cuthbert, 1998; Sanchez-Navarro, Martinez-Selva, & Roman, 2006; Witvliet & Vrana, 1995). Several lines of evidence show that physiological changes in skin conductance responses (SCRs), heart rate (HR) and electromyographic (EMG) responses vary systematically with judgments of affective valence or arousal to emotional stimuli, although evidence in the musical domain is more mixed. In general, SCR and HR responses that are indexes of autonomic activity are shown to be higher for negative than for positive emotion (Cacioppo, Bernston, Klein, & Poehlmann, 1997; Cacioppo, Bernston, Larsen, Poehlmann, & Ito, 2000; Taylor, 1991). For example, by comparing the results obtained in 13 physiological studies, Cacioppo et al. (2000) found that heart rate was significantly greater during negative than positive emotions. The authors interpreted this effect as part of a ‘‘negativity bias,’’ which was defined in behavioral studies as ‘‘the greater impact of negative information on people’s evaluations than comparably extreme positive information’’ (Peeters & Czapinski, 1990). Classical animal and behavioral studies (for review, see Cacioppo et al., 2000; Taylor, 1991) as well as more recent physiological studies (Ito, Larsen, Smith, & Cacioppo, 1998) investigated the concept of ‘‘negativity bias,’’ which can be considered as a normal manifestation of adaptive functioning. Cacioppo and colleagues (Cacioppo & Bernston, 1994; Cacioppo, Gardner, & Berntson, 1997) have then incorporated the negativity bias in a more general model of evaluative space in which positive and negative evaluative processes are assumed to involve separate motivational substrates (Lang et al., 1998). In this context, negativity bias refers to a tendency for the negative motivational system to respond more intensely than the positive motivational system to comparable amounts of activation (Ito, Cacioppo, & Lang, 1998; Ito et al., 1998). This is the definition that we adopted in the present study, and we hypothesized that this negativity bias exists also with music. In the musical domain, several studies have focused on the effect of various musical variables on autonomic responses to emotional stimuli (Baumgartner, Esslen, & Jancke, 2006; Chapados & Levitin, 2008; Grewe, Nage, Kopiez, & Altenmuller, 2007; Khalfa, Peretz, Blondin, & Robert, 2002; Khalfa, Roy, et al., 2008; Koelsch, Kilches, Steinbeis, & Schelinski, 2008;
D. Dellacherie et al. Krumhansl, 1997; Steinbeis, Koelsch, & Sloboda, 2006; Witvliet & Vrana, 2007), but only three studies specifically explored the effect of musical valence on autonomic responses (Nater, Abbruzzese, Krebs, & Ehlert, 2006; Roy, Mailhot, Gosselin, Paquette, & Peretz, 2008; Sammler et al., 2007). A greater HR deceleration in response to aversive (heavy metal) music was reported by Nater et al. (2006). Sammler et al. (2007) also found a significant decrease of HR induced by dissonant music in comparison to consonant music. This cardiac deceleration does not fit with the classic cardiac defense response obtained with aversive loud noises, which is mainly characterized by a late acceleration (Cook & Turpin, 1997). However, when unpleasant pictures or sounds are presented, a marked cardiac deceleration has been repeatedly reported (Bradley et al., 2001; Cook & Turpin, 1997; Sanchez-Navarro, Martinez-Selva, & Roman, 2006). Such a cardiac deceleration is characteristic of an orienting response (Bradley et al., 2001; Cook & Turpin, 1997). Following Cacioppo et al. (1997, 2000), this greater response to unpleasant than to pleasant affective stimuli could be interpreted as resulting from the negativity bias. According to these results, we could predict a higher HR deceleration in response to dissonant stimuli than to consonant stimuli. Although SCR has been traditionally linked to variation of arousal (Bradley et al., 2001; Bradley & Lang, 2000), some evidence suggests that SCR duration and amplitude can also be influenced by emotional valence (Cacioppo et al., 1997, 2000; Norris, Larsen, & Cacioppo, 2007; Ohman & Wiens, 2003). In the musical domain, Baumgartner et al. (2006) and Nater et al. (2006) found that SCR was more elevated for unpleasant or negatively valenced music than for pleasant or positively valenced music. Therefore, SCR could be larger for dissonant than for consonant music. According to the defense cascade model (Bradley et al., 2001; Lang, Bradley, & Cuthbert, 1997; Ohman & Wiens, 2003), emotional autonomic responses to aversive stimuli display a two-step dynamic characterized by an initial orienting response followed by a subsequent defense response. This model describes an aversive motivational circuit that triggers reactions ranging from orienting to fight/flight. The orienting response elicits a large SCR response as well as a HR deceleration in an early time window and is part of an automatic lower level appraisal that could be processed in a short period of time, without any apparent implication of cortical association areas or of awareness (Ohman & Soares, 1994; Ohman & Wiens, 2003). The defense response defined as a controlled appraisal of the emotion is reflected by further increase of SCR and HR acceleration in a subsequent time window. In addition to these two steps of response, HR responses to emotional stimuli typically show a third decelerative component that corresponds to the return to baseline. Because this third component is not part of the affective response per se, we focused here on the two first components of the HR response. Here, we hypothesized that negative emotions created by musical dissonance should elicit a similar pattern with two steps of responses (an orienting response, followed by a defense response). EMG responses of facial expression provide another index of somatic activity related to emotional valence (Cacioppo, Klein, Bernston, & Hatfield, 1993). More specifically, corrugator activity (used in frowning) is increased for unpleasant or negatively valenced stimuli (pictures: Bradley et al., 2001; mental imagery: Witvliet & Vrana, 1995; sounds: Bradley & Lang, 2000; films: Ellis & Simons, 2005). Conversely, zygomatic activity increases
Psychophysiology of musical emotion with pleasantness (Ellis & Simons, 2005; Lang et al., 1998; Witvliet & Vrana, 1995). Although the corrugator activity in response to emotional valence in music has been already reported (Roy et al., 2008; Witvliet & Vrana, 2007), no study to our knowledge has recorded corrugator and zygomatic EMG activity in response to dissonance. Based on previous results in nonmusical domains, we can hypothesize that corrugator muscles known to be influenced by negative valence should be more activated when listening to dissonance than to consonance. However, zygomatic muscles generally modulated by positive valence should be more activated in response to consonance than to dissonance. Since no study interpreted facial EMG responses in relation to the defense cascade model, no specific prediction about the dynamic changes of such responses was proposed. Finally, it has also been demonstrated that automatic physiological responses to aversive stimuli can be modulated by our own experience with the material. Some evidence in a non-musical domain suggests that the relevance of the task for a given subject could modulate SCRs. For example, parachutists exhibit larger skin conductance orienting response to parachutist-relevant words and pictures than do inexperienced controls (Epstein & Fenz, 1962; Fenz & Epstein, 1962). Since musical experience acquired through lessons or practice can influence the emotional response to dissonance, it seems relevant to explore the effect of musical experience on emotional reaction to dissonance by combining subjective rating judgments with objective autonomic (SCR and HR) and somatomotor (EMG) measurements in response to musical listening. For this purpose, consonant musical excerpts and their dissonant counterparts of the same musical pieces matched for arousal were presented to participants with low and high musical experience that were carefully controlled in terms of age, sex, education, and mood. Subjective ratings of emotional valence (according to pleasantness) as well as autonomic (i.e., HR, phasic SCR) and somatomotor (i.e., facial EMG) indexes related to valence judgments were simultaneously recorded. We predicted that dissonance will induce more negative valence judgments than consonance, and this effect would be larger in high than in low musically experienced participants. For all the participants, we also predicted higher SCR and larger HR deceleration in response to dissonance as compared to consonance, these effects being modified by musical experience as well. Based on the defense cascade model with two-step responses, two successive patterns of autonomic responses were expected as a function of the time window, early and late time windows corresponding to orienting and defense responses, respectively. Therefore, we can suppose that SCR and HR can be modified by musical experience differently in early and late time window. We also predicted that facial EMG would be modulated by emotional valence: corrugator muscles known to be influenced by negative valence should be more activated when listening to dissonance than to consonance, whereas zygomatic muscles generally modulated by positive valence should be more activated in response to consonance than to dissonance.
Methods Participants Twenty-seven participants (13 men and 14 women), aged between 19 and 31 years (mean age 5 29.41. SD ! 8.53) took part
339 in this study. Based on their responses on a musical experience questionnaire, participants were sorted into Low Experience (LE, n 5 15) and High Experience (HE, n 5 12) groups. The musical experience questionnaire included items related to their music listening habits (listening subscale) as well as their level of musical education and actual practice (practice subscale), in accordance with a multi-dimensional definition of musicianship (Ehrle, 1998). All HE participants, except one who was an autodidact, received training in classical music, and none of them reported listening to music genres which generally have a lot of dissonance, such as contemporary music or free jazz. As displayed in Table 1, the mean scores on both subscales were higher for HE (mean listening score 5 6, SD 5 1.65 and mean practice score 5 6.92, SD 5 2.57) than LE (mean listening score 5 3.87, SD 5 1.25; mean practice score 5 3.87, SD 5 1.04) participants (t(25) 5 3.829, po.005 for listening and t(25) 5 9.16, po.001 for practice). Finally, the two groups did not differ on variables unrelated to musical experience, such as age (mean age for LE 5 31.2, (SD 5 9.08); for HE 5 27.2 (SD 5 2.18); t(25) 5 0.94, n.s.), years of education (mean years of education for LE 5 14.33 (SD 5 3.24); for HE 5 15.83 (SD 5 2.25); t(25) 5 0.92, n.s.), and gender (for LE: 9M, 6F; for HE: 4M, 8F; w2(1) 5 1.90, n.s). The results remain the same when only participants kept in the SCR and HR analyses were included in the analyses. Mood Questionnaires Two mood questionnaires were administered: the State and Trait Anxiety Inventory (STAI; Spielberger, 1983) and the Profile of Mood Scales (POMS; McNair et al., 1992). The STAI comprises two subscales, one assessing the individual general level of anxiety (i.e., trait anxiety), which is presumed to be stable over time, and the other one assessing the present level of anxiety (i.e., state Table 1. Description of the Musical Experience of the Two Groups of Participants with Low (LE) and High Experience (HE) Low experience (LE)
High experience (HE)
(A) Scores on the musical experience questionnaire Global score 4.13 (1.25)n 12.91 (3.21)n Listening subscale score 3.87 (1.24)n 6 (1.65)n Practice subscale score 3.87 (1.03)n 6.92 (2.57)n (B) % of the participants reporting Listening Habits Listen music every day 80 91.60 Listen music with attention 20 66.60 Listen classical music 33 58.30 Go to concert 46.60 75 Practice Practice actually 0 33 Have received an institutional training 6.60 75 Have received lessons during at least 3 6.60 92 years Self-educated 0 8.30 (C) Details on practice for HE Age at start of study – 8.54 (3.56) Duration of the practice – 8.41 (4.30) Notes: (A) Mean scores on the musical experience questionnaire (standard deviation in parentheses). (B) Percentage of participants reporting different listening habits and musical practice details. (C) For HE participants, mean age at start of study and mean number of years of musical practice (standard deviation in parentheses). n po.05.
340
D. Dellacherie et al.
anxiety). The POMS is a questionnaire comprising 30 items, each consisting of an emotional adjective for which subjects have to rate to what extent it describes their current mood (0F‘not at all,’ 4F‘extremely’). Six emotional sub-scales can be derived from the 30 items: Tension, Depression, Anger, Vigor, Fatigue, and Confusion. Stimuli Ten musical excerpts of classical music were taken from a set used in previous studies (Gosselin et al., 2006; Khalfa, Guye, Peretz, Chapon, Girard, et al., 2008; Peretz et al., 2001; Peretz, Gagnon, & Bouchard, 1998). They were instrumental in that they were not originally sung with lyrics. The excerpts were selected to evoke happiness and were played in major mode (e.g., ‘‘Brindisi’’ from Verdi’s ‘‘Traviata’’), with a fast tempo (the quarter note value varies from 80 to 255, conventionally written by M.M. for Metronome Marking). All stimuli were transcribed for piano, computer-generated, and delivered with a piano timbre. Each excerpt had duration of 7 s. The dissonant versions of each original consonant excerpt were created by shifting the pitch of the tones of the leading voice by one semitone either upward or downward, leading to 20 dissonant versions of the 10 original consonant excerpts. During the experiment, each consonant version was presented twice to match the number of dissonant excerpts. Experimental Procedure The general time course of the experiment is depicted in Figure 1. All participants started the experiment by filling out the musical experience and the mood questionnaires. Then, the physiological sensors were attached while the participants sat comfortably in a quiet room. Before the experiment started, participants performed a handgrip squeeze task in which they were asked 4 times to squeeze repeatedly on a handgrip for 7 s. This control task was performed in order to obtain an index of individual electrodermal reactivity in each participant before the beginning of the experiment. The experimental task comprised four testing blocks.
A
musical expertence expert handgrip + squeeze mood questionnaires task
Experimental block 1
Each block consisted of 10 excerpts, including 5 consonant and 5 dissonant excerpts presented in an alternating order. The orders of presentation of the consonant and dissonant trials within each block were counterbalanced within subjects following an ABBA design, as well as between subjects with half of the participants starting with a consonant excerpt and the other half with a dissonant one. All blocks started with a loud and sudden burst of white noise (50 ms, 85–90 db) intended to elicit a startle response. These startle responses served as another index of individual electrodermal reactivity and of habituation effect on the signal throughout the experiment. Each trial started with a warning signal asking the subject not to move in order to avoid contamination of physiological recordings by movement artifacts. This signal was followed by a rest period lasting between 7.1 s and 7.7 s, in which subjects were asked to do nothing. The length of the rest period varied randomly in order to reduce subject’s expectancies and response habituation. The musical excerpt was then played for 7 s and was followed by another rest period of 1 s, in which subjects were again asked not to move. This second rest period allowed the recording of music-related physiological activity overflows and also prevented the activity recorded at the end of the musical excerpt from being contaminated by response preparation artifacts. The end of this 1-s rest period was indicated by another signal allowing participants to move. Then, participants were asked to judge the excerpt they just heard using a rating scale for emotional valence (from ! 5 5 ‘unpleasant’ to 5 5 ‘pleasant’) and arousal (from ! 5 5 ‘relaxing’ to 5 5 ‘stimulating’). No time constraints were given for the ratings. All participants signed informed consent to participate in the study. Data Acquisition and Transformation SCR, HR, and EMGs were monitored continuously using an MP150 Biopac system (Biopac Systems, Inc., Goleta, CA) at a sampling rate of 2000 Hz and processed using AcqKnowledge
Experimental block 2
Experimental block 3
Experimental block 4
B startle l probe + recovery period
Musical ti l trials
10 sec
CDCCDCDDCD or DC D C C D C D D C ≈ 350 sec
C
“ l “please don’tt don move”
rest period
2 sec
7.7 7 1 to 7 7.1 7 sec
rest musical excerpt period 7 sec
1 sec
“ “you can move”
valence and arousal ratings
1 5 sec 1.5
≈ 10 sec
Figure 1. Experimental procedure: (A) Participants first filled out the musical experience and mood questionnaires and performed the handgrip squeeze task. The musical excerpts were then presented in four experimental blocks. (B) Each block started with a startle probe. Five consonant and 5 dissonant excerpts were presented in an alternate order. The blocks either started with a consonant ‘‘C’’ or dissonant ‘‘D’’ excerpt. (C) Each musical trial started with a rest period in which participants were asked not to move. At the end of each musical excerpt, participants were asked to remain still for 1 s before they had to rate the valence and arousal of the excerpt.
Psychophysiology of musical emotion
341
software. Skin conductance was recorded on the index and middle finger of the non-dominant hand using the EDA finger transducer BSL – SS3LA filled with an isotonic conducting gel. The signal was filtered between 0.05 and 10 Hz. Then, the SCRs to each musical trial were visually inspected and checked for failures of the measuring device as well as movement-related artifacts. Trials in which there was electrodermal activity prior to the onset of the musical excerpts were excluded. Responses were selected if they occurred in a 1–4-s latency window following stimulus onset. Three participants from the low expertise group that showed no or very feeble responses to grasping and startle trials were considered as non-responders and excluded from the analysis. Facial EMGs were recorded over the left corrugator and zygomatic sites as recommended by Fridlund and Cacioppo (1986), using 8 mm Ag/AgCl shielded electrodes, and were filtered between 100 and 500 Hz. After the recording, EMG was rectified using the root mean square function of the software and then smoothed by a factor of 200 samples. HR was recorded using a bipolar montage with an electrode on the right carotid artery and another below the left ribs. The signal was filtered (0.5 to 35 Hz). Instantaneous inter-beat (RR) intervals (in ms), corresponding to the inverse of HR, were calculated from the electrocardiogram using a peak detection algorithm to detect successive R-waves and obtain a continuous RR tachogram. Careful examination of the electrocardiogram and the tachogram ensured that the automatic R-wave detection procedure had been performed correctly. Careful examination of each participant’s electrocardiogram signal led to the exclusion of four participants (3 LE and 1 HE) due to technical failures. Electrocardiograms of the remaining participants were then cleaned by removing the trials in which the presence of artifacts prevented the exact extraction of RR Intervals (RRI). The participants who were excluded from the SCR analyses were not the same ones who were excluded from the HR analyses.
Results Before analyzing the data, we checked for group differences between LE and HE participants on mood questionnaires. Then, we tested the effects of dissonance and musical experience on valence and arousal ratings, as well as on the physiological re-
sponses to the musical excerpts. Finally, we used regression analyses to explore the relationship between valence ratings and personal (mood and musical experience) and physiological (SCR, facial EMGs, HR) variables. For all analyses, the Fmax statistic was used to test if the homogeneity of variance assumption was met. Following the guidelines of Tabachnick and Fidell (2001) for similar samples sizes (i.e., within a ratio of 4 to 1), the Fmax was in an acceptable range (i.e., below 10) for all analyses, indicating that the variances between samples were sufficiently homogeneous to proceed with the analyses. Partial eta-squared (Z2) were used as the effect sizes for the analyses of variance (ANOVAs). According to Cohen’s (1988) guidelines, Z2 5 .01 corresponds to a small effect, Z2 5 .09 to a medium effect, and Z2 5 .25 to a large effect. Mood Questionnaires The mean ratings of the mood questionnaires STAI and POMS for the LE and HE groups are displayed in Table 2, along with the results of the t-tests to verify whether baseline mood levels differed between the LE and HE groups. As can be seen in this table, there were no differences on any of the mood parameters that were assessed, confirming that both groups did not differ in this regard. Valence and Arousal Ratings Separate ANOVAs with one repeated measure (consonance vs. dissonance) were carried out on the valence and the arousal rating scores. Figure 2 shows the mean ratings of valence and arousal for consonant and dissonant excerpts as a function of musical experience. As expected, dissonant excerpts were rated as more unpleasant than consonant ones (F(1,25) 5 73.19, po.05, Z2 5 0.75). This effect was modulated by the degree of musical experience of the listener, as revealed by the significant Musical experience by Dissonance interaction (F(1,25) 5 8.32, po.05, Z2 5 0.25). HE participants rated dissonant excerpts as more unpleasant than LE participants did (F(1,25) 5 8.81, po.05, Z2 5 0.26), whereas their ratings of consonant excerpts did not differ across groups (F(1,25) 5 0.02, p 5 n.s., Z2 5 0.001). The analysis of arousal ratings showed no effect of Dissonance (F(1,25) 5 0.28, p 5 n.s., Z2 5 0.015) or of Musical experience (F(1,25) 5 0.97, p 5 n.s., Z2 5 0.51) nor any interaction (F(1,25) 5 0.08, p 5 n.s., Z2 5 0.003).
Table 2. Inter-Group Comparisons of the Mean (! SD) Ratings on the Mood Questionnaires and Electrodermal Reactivity Tests Musical experience Dependent variable STAI subscales Trait anxiety State anxiety POMS subscales Anger Anxiety Depression Confusion Vigor Fatigue Electrodermal reactivity Handgrip squeeze Startle Note: n.s. 5 not significant.
Low experience
High experience
Result of t-test
37.33 (! 8.86) 34.40 (! 7.15)
34.83 (! 7.63) 30.58 (! 5.65)
t(25) 5 1.51, p 5 n.s. t(25) 5 0.77, p 5 n.s.
0.73 (! 1.55 (! 0.91 (! 4.90 (! 11.55 (! 4.18 (!
1.10) 2.25) 1.64) 0.83) 3.59) 2.27)
t(25) 5 1.28, p 5 n.s. t(25) 5 0.25, p 5 n.s. t(25) 5 0.62, p 5 n.s. t(25) 5 1.06, p 5 n.s. t(25) 5 0.42, p 5 n.s. t(25) 5 0.99, p 5 n.s.
0.48 (! 0.23) 0.58 (! 0.40)
t(22) 5 0.28, p 5 n.s. t(22) 5 0.86, p 5 n.s.
1.40 (! 1.73 (! 1.33 (! 4.27 (! 9.80 (! 3.13 (!
1.45) 1.62) 1.80) 1.87) 2.68) 2.92)
0.45 (! 0.46) 0.38 (! 0.34)
342
D. Dellacherie et al.
Figure 2. Mean valence and arousal ratings of the dissonant and consonant musical excerpts for the low and high experience groups.
Physiological Recordings Electrodermal Responses After exclusion of three non-responder participants, there were 12 LE and 12 HE left in the SCR analyses. To make sure that the LE and HE groups exhibited similar levels of electrodermal reactivity, the mean SCRs for the grasping and startle trials were compared by means of t-tests (Table 2). The results showed that there were no differences in baseline electrodermal reactivity between the two groups (Handgrip: t(22) 5 0.28, p 5 n.s.; Startle: t(22) 5 0.86, p 5 n.s.). The SCRs to the consonant and dissonant musical excerpts were then analyzed for the two groups of participants. First, the mean SCR values were extracted by slices of 500 ms and averaged by musical condition and experimental group in order to provide average response curves for the dissonant and consonant excerpts and LE and HE participants (see Figure 3A). Visual inspection of the resulting graph led to the identification of two response periods according to the guidelines proposed by Dawson (2007). The first response ranged from 1.5 to 6 s and corresponded to the orientation response triggered by the onset of the excerpt. The second response ranged from 6 to 8 seconds and appeared to reflect later responses to the musical excerpts. The amplitude of each SCR was then computed by extracting the maximum value of the response in each time window and subtracting it from the baseline level. For the first time
window, the baseline period was the mean skin conductance level of the 1-second baseline preceding the onset of the excerpt. For the second time window, the mean skin conductance level between the 5th and 6th second served as the baseline. The resulting values were transformed in log (SCR11) and averaged for each participant. An ANOVA with Dissonance (consonance vs. dissonance) and Time window (early or later) as within-subjects factors were carried out on the SCR values for the two groups of participants (LE vs. HE). The averaged peaks for the LE and HE groups are displayed in Table 3. SCRs were higher when participants listened to dissonant than to consonant excerpts (F(1,22) 5 9.85, po.05, Z2 5 0.31), and this effect was influenced by musical experience, as revealed by the significant interaction (F(1,22) 5 4.32, po.05, Z2 5 0.16) which demonstrates that the difference in SCRs between consonance and dissonance was higher for HE (F(1,11) 5 7.89, po.05, Z2 5 0.42) than LE (F(1,11) 5 2.05, p 5 n.s., Z2 5 0.16) group. The effect of Dissonance also marginally interacted with Time window (F(1,22) 5 4.04, p 5 .057, Z2 5 0.001), the effect of Dissonance being more pronounced in the early (F(1,22) 5 9.01, po.05, Z2 5 0.29) than in the late time window (F(1,22) 5 1.63, p 5 n.s., Z2 5 0.066). Finally, the interaction between Dissonance, Time window, and Musical experience was not significant
Table 3. Mean Values (! SD) of the Physological Recordings for Consonant and Dissonant Excerpts by Time Window and Level of Musical Experience First window Consonant SCR amplitude Log (maximum 11), mS Corrugator EMG Area under the curve, mVnsec Zygomatic EMG Area under the curve, mVnsec RR interval In the first window : Maximum acceleration, msec In the second window : Minimum deceleration, msec Note: All displayed values are differences from baseline.
Low experience 0.08 (! High experience 0.12 (! Low experience 0.03 (! High experience 0.36 (! Low experience " 0.12 (! High experience " 0.26 (! Low experience High experience
0.08) 0.08) 0.28) 0.90) 0.22) 0.42)
33.73 (! 13.78) 24.41 (! 12.64)
Second window
Dissonant 0.10 (! 0.20 (! 0.06 (! 0.43 (! " 0.03 (! " 0.11 (!
0.07) 0.17) 0.24) 0.65) 0.56) 0.26)
Consonant 0.01 (! 0.04 (! 0.03 (! 0.10 (! " 0.05 (! 0.26 (!
0.03) 0.05) 1.19) 01.17) 1.43) 2.44)
Dissonant 0.01 (! 0.07 (! 0.94 (! 1.92 (! 0.10 (! 3.03 (!
0.03) 0.03) 1.53) 4.53) 2.51) 6.70)
29.50 (! 11.02) " 36.82 (! 28.55) " 35.92 (! 26.96) 29.27 (! 11.44) " 32.06 (! 20.57) " 32.38 (! 14.66)
Psychophysiology of musical emotion
A
343
B
Skin Conductance Responses
0.150
0.50
0.125
2nd window
0.075
0.40
µvo olt s
0.100
µsie men ns
Corrugator EMG
0 0.60
1st window
0.050
03 30 .0
1st window
2nd window
0.20 0.10
0.025
0 0 −0.025
1
2
3
4
5
6
7
8
9
−0.10
1
2
4
5
6
7
Time in seconds
Time in seconds
C
D
Zygomatic EMG
R-R interval
1.20
10
1.00
8
2nd window
0.40
mil llis e eco n nds
0.60
1st window
0.20
4 2 0 −2 −4
1
2
3
4
5
6
7
1 window
−6
0 −0.20
2nd window
6
0.80
µvo olts
3
1
2
3
4
5
6
7
Time in seconds
−8 −10 −12
Time in seconds
Figure 3. Average time-courses of the physiological responses during the presentation of consonant and dissonant excerpts for low and high musical experience groups. Each physiological measure was divided in two phases of responding. (A) For skin conductance responses, a first orientation response (0–6 s) was followed by a second slower response (6–8 s). Note that the averaging of slight inter-trials and inter-subjects differences in the onsets of the second non-orientation response smoothes the shape of the mean response curves, compared to individual trials, from which the analyzed responses were extracted. (B) Corrugator EMG started to diverge between consonant and dissonant excerpts after the 2nd second. (C) Zygomatic EMG started to diverge between consonant and dissonant excerpts after the 2nd second. (D) RR Intervals first increase between 0–3.5 s (slowing of heart rate) and decrease between 3.5–7 s (speeding of heart rate).
(F(1,22) 5 0.17, p 5 n.s., Z2 5 0.075). For exploratory reasons, we tested if the interaction between Dissonance and Musical experience was different in the two time windows by performing ANOVAs in the first and second time window with Dissonance and Musical experience as factors. The results showed that the effect of Dissonance did not interact with Musical experience in the early time window (F(1,22) 5 1.78, p 5 n.s., Z2 5 0.075), but it did interact with Musical experience in the second time window (F(1,22) 5 4.11, po.05, Z2 5 0.16), indicating that SCRs were higher for dissonant than for consonant excerpts in the late time window. This effect holds for HE (t(11) 5 2.39, po.05, Z2 5 0.25) but not for LE (t(11) 5 1.91, po.05, Z2 5 0.05) participants.
acceleration of heart rate), and then by a final increase of RRI. The amplitude of the first increase and of the following decrease were then extracted and subtracted from a 1-s baseline preceding the onset of the musical excerpts. An ANOVA with Dissonance and Phase of response was carried out on the RRI for the two groups of participants. The amplitude of the first increase and secondary decrease in RRI for the HE and LE groups are displayed in Table 3. No significant effect of Dissonance (F(1,21) 5 0.001, p 5 n.s., Z2 5 0.00) or of Musical experience (F(1,21) 5 0.01, p 5 n.s., Z2 5 0.00) was observed. Only the expected effect of Phase was observed (F(1,21) 5 162.53, po.05, Z2 5 0.87), confirming that RRI were higher during the first increase than during the following decrease.
Heart Rate There were 12 LE and 11 HE participants left in the HR analyses. RRI was extracted by slices of 500 ms and averaged by musical condition and experimental group in order to provide average response curves for the dissonant and consonant excerpts and LE and HE participants (see Figure 3D). Inspection of the mean RRI curves revealed a typical triphasic pattern (Bradley et al., 2001) comprising an initial increase of RRI (i.e., deceleration of heart rate), followed by a decrease of RRI (i.e.,
Facial Electromyography The corrugator and zygomatic EMG responses to musical excerpts were calculated as the difference between the raw signal during each musical excerpts and the mean activity observed in an 800-ms baseline preceding the onset of the excerpts. The resulting response curves were then averaged by slices of 100 ms for the dissonant and consonant excerpts and for LE and HE groups (see Figures 3B and 3C). Visual inspection of the resulting graphs indicated that facial muscles appeared to show two phases of
344 responding: a first phase ranging from 0 to 2 s, where there were no differences between the consonant and dissonant trials, and a second phase from 2 to 7 s, where responses to consonant and dissonant musical excerpts started to diverge. The area under the curve was extracted within these two time windows and was averaged for consonant and dissonant versions for each participant. The averaged area under the curve for the LE and HE groups are displayed in Table 3. An ANOVA with two repeated measures, Dissonance and Time window, was carried out on the averaged area under the curve for the LE and HE groups. Corrugator activity differed between consonant and dissonant excerpts as a function of Time window, as revealed by the significant interaction between Dissonance and Time window (F(1,25) 5 4.09, p 5 .05, Z2 5 0.14). Decomposition of the interaction revealed that corrugator activity was higher for dissonant than for consonant excerpts in the second window (F(1,25) 5 4.06, p 5 .05, Z2 5 0.14), but not in the first window (F(1,25) 5 0.40, p 5 n.s., Z2 5 0.02). Zygomatic activity differed as a function of Dissonance, Time window, and Musical experience, as revealed by the significant interaction between the three factors (F(1,25) 5 4.09, po.05, Z2 5 0.14). Decomposition of the interaction showed that there was no difference between consonant and dissonant excerpts in the first time window (F(1,25) 5 0.01, p 5 n.s., Z2 5 0.00) whereas zygomatic activity was higher for dissonant than for consonant excerpts in the second window for the HE (F(1,11) 5 4.84, p 5 .05, Z2 5 0.31), but not for the LE group (F(1,14) 5 0.01, p 5 n.s., Z2 5 0.00). Regression Analyses Stepwise Regressions In order to assess the relationship between personal variables, valence ratings, and physiological responses, we performed two stepwise regressions with the effects of dissonance on valence ratings as the dependent variable and physiological responses (SCR, facial EMGs, and RRI within all time windows) and personal variables (Musical experience and STAI and POMS subscales) as the independent variables. The goal of these regressions was to identify the physiological responses and personal variables that best predicted the effects of dissonance on valence ratings. For all the variables included in the analysis, the difference between the mean value for the dissonant and consonant excerpts (dissonant–consonant) was used as an index of the effects of dissonance on these variables. Because the regression models included all the physiological measurements, we only kept the participants for which SCR and HR data were available (9 LE and 11 HE). The results of the first stepwise regression revealed that zygomatic activity in the second time window was the best predictor of the decrease in valence in reaction to dissonant excerpts (r 5 0.61, po.05). None of the other variables (SCRs, corrugator activity, RRI, and zygomatic activity in the first time window) significantly improved the proportion of variance explained by the model once the variance explained by the zygomatic activity in the second time window was taken into consideration. The results of the second stepwise regression showed that musical experience was the best predictor of the decrease in valence in reaction to dissonant excerpts (r 5 0.53, po.05). Again, all other personal variables (STAI and POMS subscales) did not significantly improve the proportion of variance explained by the model. Thus, the participants who scored higher on the musical experience questionnaire or who had the largest increases in zygomatic activity in reaction to dissonant excerpts were the ones
D. Dellacherie et al. showing the largest decreases in valence ratings in response to dissonance. A final correlational analysis indicated that musical experience was also correlated to the amount of zygomatic activity in response to dissonance (r(26) 5 0.59, po.05), suggesting that musical experience predicted zygomatic responses to dissonance, which in turn strongly predicted the decreases in valence ratings in response to dissonance.
Discussion The purpose of our study was to determine the effect of musical experience on emotional reaction to dissonance by recording emotional self-reports as well as their psychophysiological correlates. For this purpose, we compared the emotional responses of two categories of listeners. One consisted of nonmusicians with low musical experience (LE). The other one was composed of musicians with high musical experience (HE) who had received at least 3 years of formal training or who were still practicing a musical instrument although they did not reach professional levels. Therefore, we were able to investigate the interaction between musical experience and emotional pleasantness judgments on autonomic and somatic responses by measuring SCR, HR, and facial EMG for the zygomatic and corrugator muscles in response to consonant and dissonant musical excerpts. Before we discuss the results in further detail, we should first point out that the two groups did not differ in terms of age, sex, education, or mood. Moreover, given that the only difference between the stimuli concerned the pitch shift of the melodic line, tempo differences could not have contaminated physiological measurements, in contrast to most previous studies on musical emotions where different musical excerpts were used to test different emotions. Behavioral Measures As predicted, the behavioral results confirm previous findings, indicating that listening to dissonance induces a more negative affect than listening to consonance (Blood et al., 1999; Gosselin et al., 2006; Koelsch et al., 2006; Pallesen et al., 2005; Passynkova et al., 2007; Peretz et al., 2001; Sammler et al., 2007). Arousal ratings, which were used as a proxy for motivational activation (Ito, Larsen, et al., 1998) did not differ between consonant and dissonant excerpts. In agreement with Pallesen et al. (2005) and Schoen et al. (2005), we also found that the effect of dissonance on emotional valence judgments is more salient in HE than in LE participants. Given that arousal judgments obtained for consonant and dissonant stimuli did not differ between the two groups of listeners, we can therefore suggest that musical experience modulates valence judgments. Moreover, regression analyses showed that musical experience was a very good predictor of valence ratings in the present study, suggesting that musical experience may have enhanced the participants’ sensitivity to dissonance. However, since almost all HE participants were trained in classical music, it remains to be determined if the observed results can be generalized to training in other musical genres more marked by dissonance, such as free jazz or contemporary music. Nevertheless, this result appears to be consistent with data of another study (Bigand, Parncutt, & Lerdahl, 1996) showing that dissonant chords induce stronger ratings of musical tension than consonant chords, this effect being more pronounced in musicians than in nonmusicians. We can therefore conclude that musical experience can modulate emotional judgment of music even if the role of the specific background remains to be clarified.
Psychophysiology of musical emotion Physiological Responses The analysis of physiological measures revealed that dissonance elicited stronger responses than consonance in the two groups of participants. Given that dissonant excerpts were judged by all participants as more unpleasant than consonant excerpts but equally arousing, we interpret this reaction to dissonance as resulting from the emotional negativity bias, defined as the tendency for the negative motivational system to respond more intensely than the positive motivational system to comparable amounts of activation (Cacioppo et al., 2000; Taylor, 1991). The main finding of our study confirmed that autonomic and somatomotor responses to musical dissonance are influenced by musical experience, with HE participants presenting stronger physiological responses to dissonance in comparison to consonance than LE participants. Since HE participants judge dissonance as even more unpleasant than consonance as compared to LE participants, the differential reactivity to dissonance between the two groups seems to be related to the subjective ratings of emotional valence. HR Measures HR is normally responsive to the affective valence of the stimuli. Emotional stimuli generally prompt a triphasic HR response characterized by an initial brief deceleration, followed by a short acceleration and a late moderate deceleration (Lang et al., 1997; Sanchez-Navarro et al., 2006). Although we found the predicted triphasic response when listening to the musical excerpts, it was not modulated by dissonance. This result does not seem to be consistent with previous findings obtained with unpleasant pictures (Bradley et al., 2001). However, these authors found an effect of valence on HR response only with high (e.g., erotic pictures or mutilation scenes) but not with low arousal pictures from International Affective Picture System (Lang, Bradley, & Cuthbert, 1999). Similarly, Sammler et al. (2007) found an effect of dissonance on HR deceleration with real but not synthetic excerpts that might have been more arousing than our computer-synthesized versions of the stimuli. It might be therefore possible that the musical stimuli used in the present study do not induce a sufficiently high emotional response to affect HR. Notwithstanding the fact that we did not observe any significant difference in HR responses between the two emotional conditions, the successive phases of decelerative and accelerative cardiac responses clearly argue in favor of the two-step defensive cascade model (Bradley et al., 2001; Lang et al., 1997). SCR The analysis of phasic SCR revealed two different peaks of response to emotional musical excerpts in the 2–6 s and in the 6–8 s windows. As the late response could not be ascribed to the onset of the musical excerpt, this second bump indicates a second, superimposed skin conductance response. Indeed, SCRs are monophasic, and a secondary increase indicates that another SCR has been triggered after the first one (Dawson, 2007). These two successive responses match our hypothesis based on a twostep cascade model and are consistent with the HR response pattern (Bradley et al., 2001). The initial response, which occurs immediately after the onset of the stimulus (early time window) and the following response beginning a few seconds later (late time window) are both larger for dissonant than for consonant excerpts. Based on the cascade defense model (Bradley et al., 2001; Lang et al., 1997; Ohman & Wiens, 2003), these two successive responses could correspond to the (automatic) orienting
345 response in the first time window and the (more controlled) defense response in the second one. Curiously, such a biphasic response is rarely seen in most psychophysiological studies of emotion, where visual stimuli are used to induce emotions. For instance, in Bradley and Lang’s study (2000), although the affective pictures had a 6-s duration, no second phasic response after the initial orienting response was observed. Classically, a ‘‘more sustained’’ electrodermal response is observed as defense response (Norris et al., 2007; Ohman & Wiens, 2003). One explanation for this original finding could be that musical emotions unfold in time (Grewe et al., 2007) whereas for pictures, all the relevant emotional information is available immediately. Interestingly, musical experience seems to have contributed to the enhancement of this second, more controlled response. Indeed, after verifying that baseline electrodermal reactivity did not differ between the two groups of participants, we showed that the SCR difference between dissonance and consonance is more pronounced in HE than in LE group, as predicted. Subsequent SCR analysis showed that musical experience interacted with dissonance in the second but not in the first time window, suggesting that these orienting and defense responses were differently modulated by musical experience. The orienting response is known to be processed at a preattentive level and without implication of cortical association areas (Ohman & Wiens, 2003). The orienting SCR to dissonance that we observed in the present study could be interpreted as an automatic response reflecting an emotional unconscious processing. Given that this orienting SCR was higher for dissonance than for consonance in HE and LE participants, we suggest that nonmusicians can discriminate dissonance from consonance on the basis of emotional feeling as musicians do. We were not able to demonstrate an interaction between musical experience and dissonance in this first time window, although larger SCRs were recorded for HE than for LE participants. This finding indicates that LE as well as HE participants showed stronger orienting responses to dissonance than to consonance. We note that this result was observed despite the lack of formal musical training in LE participants. At a more evaluative level (defense response in the second time window), we also observed larger responses to dissonance (unpleasant) than to consonance (pleasant) but only in HE participants. This response seems to be enhanced in musicians. The influence of musical valence on the SCR in the second time window is compatible with previous results demonstrated by recording the tonic (and not phasic) SCRs when listening to musical stimuli for more than 1 min (Baumgartner et al., 2006; Nater et al., 2006). The fact that a sustained and higher response was obtained for dissonance than for consonance suggests that dissonance affects a more elaborate processing in addition to the automatic response. This later processing of musical stimuli could be specific to musicians. An interpretation of this result is that formal musical learning led musicians to consciously reject dissonant stimuli in a controlled defense response that follows the pre-attentive orienting one. Taken together, analysis of SCR suggests that musical experience could modulate musical dissonance processing, particularly in the late SCR. This result indicates that musical education and training enhance emotional response to dissonance. An alternative interpretation of these results could be that stronger SCRs to dissonance than to consonance may be related to an effect of unfamiliarity of dissonance. It is clear that consonant excerpts are more frequent or familiar than dissonant
346 ones because we are immersed in a consonant musical environment. Moreover, Terhard (1984) hypothesized that prenatal exposure to the overtone structure of maternal speech could serve as the basis for developing preferences for consonance. It is also clear that musical experience contributes to the fact that some people are more exposed to consonance than others are, even more so if the formation is classical as it is in the present study. It is therefore impossible to differentiate the pleasantness induced by consonance from the frequency or exposure effects (Zajonc, 1980) and to dissociate the unpleasantness to dissonance from a novelty or incongruity effect. However, the aim of the study was not to explain the origin of aversion to dissonance, which is still a hotly debated topic (Hauser & McDermott, 2003), but rather to explore the relationships between physiological and self-reported measures of emotional responses to dissonance (Bradley et al., 2001; Ohman & Wiens, 2003). Because subjective emotional ratings and physiological responses converge, the interpretation of physiological responses as reflecting emotional reactions to dissonance appears highly probable, even if this emotion might be at least partly explained by the unfamiliarity of dissonance. Several lines of evidence in non-musical domains showed that the relevance of the task for a given subject plays a role in modulating the SCR (Dindo & Fowles, 2008; Epstein & Fenz, 1962; Fenz & Epstein, 1962; Lang et al., 1998; Ohman & Soares, 1994; Perpina, Leonard, Bond, Bond, & Banos, 1998; Stormark, Laberg, Nordby, & Hugdahl, 2000). These studies, carried out with psychiatric patients and normal volunteers, revealed that higher SCRs specific to a given object were dependent on individual experience. For example, studies examining phobic individuals (Lang et al., 1998; Ohman & Soares, 1994) showed hyperactivity to phobic objects, by expressing an intense and sustained fear in response to stimuli that were not necessarily dangerous. According to Mineka and Ohman (2002), the learning of fear could arise from a classic Pavlovian conditioning: phobic patients would learn to be frightened by a neutral stimulus because of an association between such a stimulus and an aversive event. Because formal musical education trains listeners to detect and reject dissonance when it is not integrated in an appropriate musical context, we can hypothesize by analogy that musical experience leads to a hypersensitivity to dissonance. Consequently, HE participants could produce larger SCRs to dissonance than do LE participants in the same way as phobic subjects respond to phobia-relevant stimuli (Lang et al., 1998). In other terms, even if aversion for dissonance is natural (Schellenberg & Trainor, 1996; Zentner & Kagan, 1996) or influenced by an exposure effect (Witvliet & Vrana, 2007), HE participants could be seen as conditioned to react to dissonance, resulting in an enhanced autonomic reactivity. EMG Measures Another index of the emotional appraisal of the stimuli was the EMG responses. Corrugator and zygomatic muscles are known to accompany unpleasant and pleasant emotions, respectively. In the present study, we found a response of these muscles that arose late (during the second time window), which probably reflected a controlled mechanism. We can clearly see that there is no zygomatic activity before 2 s after the onset of the stimulus. Thus, the zygomatic response to musical excerpts is particularly late. This cannot be explained by the characteristics of EMG latencies. Indeed, unlike skin conductance responses, which generally take around 2 s to develop because of sweat gland physiology, EMG responses can be triggered within 100
D. Dellacherie et al. ms. Therefore, this late response cannot be interpreted as an orienting automatic response. Following the two-step cascade model and given the coherent HR and SCR responses in regard to this model, we thus propose to interpret a posteriori the late EMG reactivity (more than 2 s) as reflecting more controlled psychological processes related to conscious emotional evaluation, which is characteristic of the defense response (Ohman & Wiens, 2003). This interpretation is confirmed by the regression analysis, which shows a strong relation between EMG responses and valence ratings. Indeed, the zygomatic activity in the second window was the best predictor of valence ratings. Moreover, 2 s after the onset of the aversive stimulus, corrugator activity was higher for dissonant than for consonant excerpts, confirming that such a measure is relevant to the study of emotional reaction to music (Roy et al., 2008; Witvliet & Vrana, 2007). One probable reason why this facial response seems to take more time to be initiated in the case of music compared to pictures (Bradley et al., 2001) is the fact that the musical emotions unfold in time whereas for pictures, all the relevant emotional information is immediately available. More studies on the dynamics of the response to musical emotion are needed to clarify the specificity of emotional responses to music in comparison to other types of stimuli, such as faces or brief sounds. In addition, we discovered that the late increase of zygomatic contraction 2 s after the stimulus onset was specific to experienced participants when they were listening to dissonant excerpts. This unexpected result seems to be in contradiction with the responses observed in other studies showing such a response with pleasant stimuli (Ellis & Simons, 2005; Lang et al., 1998; Witvliet & Vrana, 1995). This zygomatic activity might be interpreted as an ironic smile or as a grimace related to the displeasure induced by dissonance. Since we witnessed that some participants paradoxically laughed (Ansfield, 2007; Craig & Patrick, 1985; Keltner & Bonanno, 1997; Papa & Bonanno, 2008; Prkachin & Solomon, 2008) at the dissonant excerpts, we tend to favor the former interpretation, although we lack objective empirical data to confirm it. Regression analysis revealed that more unpleasant dissonances were associated with stronger zygomatic activities and that this smile/grimace is the best predictor of the subjective responses to dissonance. The zygomatic response could therefore constitute not only a controlled but also a communicative part of the emotional response to dissonance, and this reaction could be more developed in HE participants than in LE participants. Mimics to dissonance would have a communicative function that musical learning might have contributed to create in HE participants. We could hypothesize that experience enhances the ability to communicate musically induced negative emotions. Taken together, these results show that musical experience plays a role in physiological and communicative responses induced by dissonance that are linked to emotional valence. Based on our results, it seems that dissonance produces larger SCR and EMG responses in HE than LE listeners. The results could be further explained by brain reorganization induced by musical experience. Indeed, even in amateur musicians or in children, musical training can produce functional and morphological brain changes in auditory areas (Gaser & Schlaug, 2003; Schneider, Scherg, Dosch, Specht, Gutschalk, & Rupp, 2002) as well as in emotional structures such as the amygdala and the insula (James, Britz, Vuilleumier, Hauert, & Michel, 2008), the anterior cingulate cortex (Foss, Altschuler, & James, 2007) and the frontal-lobe areas (Koelsch, Fritz, Schulze, Alsop, & Schlaug, 2005; Minati et al., 2009).
Psychophysiology of musical emotion
347
Taken together, based on the obtained physiological results, we argue for the existence of a first automatic response to dissonance, which is mainly characterized by an orienting SCR. This initial physiological reaction seems to be followed by a second phase, which is less automatic and more controlled. This second step is characterized by facial EMG responses and a second non-orienting SCR. The first response suggests the existence of an alarm system that can automatically interpret dissonant music as an aversive stimulus and then evaluate this stimulus in terms of displeasure. In the nonmusical domain, an alarm system in response to fear was shown to involve the amygdala (Liddell, Brown, Kemp, Barton, Das, et al., 2005). Given that responses of the autonomic nervous system are linked to the amygdala and other structures such as the cingulate cortex, the effect of dissonance on SCR is compatible with neuroimaging studies suggesting that affective responses to dissonance could mainly involve mesio-temporal lobe structures (Blood et al., 1999; Koelsch et al., 2006, 2007). The stronger late responses obtained in HE than in LE participants (second phase response in SCR and zygomatic response) seem consistent with the more important involvement of cortical structures such as the cingulate cortex or frontal lobe areas in musicians than in nonmusicians in response to dissonance and consonance (Foss et al., 2007; Minati et al., 2009). Considering that the cingulate cortex is known to contribute in regulating autonomic functions as well as in attentional control (Critchley, Mathias, Josephs, O’Doherty, Zanini, et al., 2003; Devinsky, Morrell, & Vogt, 1995), we may hypothesize that the SCR and the activity observed with fMRI in the cingulate cortex in musicians may reflect a similar mechanism, resulting in a hypersensitivity to dissonance. This hyperactivation may result in a stronger negativity bias observed in autonomic reactivity in HE than in LE participants in the second phase of processing dissonance.
Conclusion The data of the present study emphasize the effect of musical experience on emotional responses to music and the role of emotional valence in determining the autonomic and somatomotor components of these responses. The results indicate that SCR and EMG responses are sensitive to dissonance. These measures therefore appear to be appropriate tools to assess emotional reaction to unpleasant music. Stronger responses to dissonant than to consonant music were observed in the orienting SCR, and there were no differences between HE and LE participants in the amplitude of this response, suggesting that the first step of emotional response to dissonance could be independent of musical experience. Late skin conductance and EMG responses (corrugator and zygomatic) confirmed the presence of a subsequent, more controlled, response to dissonance. Moreover, whereas corrugator activity when listening to dissonance was found in all participants, late SCRs and zygomatic activity were found only in experienced musicians, suggesting a specific emotional response in this group. This specific reaction expressed by stronger physiological late responses to dissonance in HE than in LE participants was confirmed by self-report responses. This suggests that experience could influence the negativity bias. An interpretation is that musical education could have reinforced the representation of dissonance for musicians by a long and sustained associative learning between dissonance and unpleasant emotions. These results add arguments in favor of both physiological and psychological origins of the feeling of dissonance. Moreover, the smile observed in musicians represents an original finding indicating that musical experience influences not only conscious appraisal of emotional significance but also its means of communication. In other terms, learned aesthetic preference might play a powerful role in affective and expressive response to music.
REFERENCES Ansfield, M. E. (2007). Smiling when distressed: When a smile is a frown turned upside down. Personality and Social Psychology Bulletin, 33, 763–775. Baumgartner, T., Esslen, M., & Jancke, L. (2006). From emotion perception to emotion experience: Emotions evoked by pictures and classical music. International Journal of Psychophysiology, 60, 34–43. Bigand, E., Parncutt, R., & Lerdahl, F. (1996). Perception of musical tension in short chord sequences: The influence of harmonic function, sensory dissonance, horizontal motion, and musical training. Perception & Psychophysics, 58, 125–141. Blood, A. J., Zatorre, R. J., Bermudez, P., & Evans, A. C. (1999). Emotional responses to pleasant and unpleasant music correlate with activity in paralimbic brain regions. Nature Neuroscience, 2, 382–387. Bradley, M. M., Codispoti, M., Cuthbert, B. N., & Lang, P. J. (2001). Emotion and motivation I: Defensive and appetitive reactions in picture processing. Emotion, 1, 276–298. Bradley, M. M., & Lang, P. J. (2000). Affective reactions to acoustic stimuli. Psychophysiology, 37, 204–215. Brattico, E., Pallesen, K. J., Varyagina, O., Bailey, C., Anourova, I., Jarvenpaa, M., et al. (2009). Neural discrimination of nonprototypical chords in music experts and laymen: An MEG study. Journal of Cognitive Neuroscience, 21, 2230–2244. Cacioppo, J. T., & Bernston, G. G. (1994). Relationship between attitudes and evaluative space: A critical review, with emphasis on the separability of positive and negative substrates. Psychological Bulletin, 3, 401–423. Cacioppo, J. T., Bernston, G. G., Klein, D. J., & Poehlmann, K. M. (1997). The psychophysiology of emotion across the lifespan. Annual Review of Gerontology and Geriatrics, 17, 27–74.
Cacioppo, J. T., Bernston, G. G., Larsen, J. T., Poehlmann, K. M., & Ito, T. A. (2000). The psychophysiology of emotion. In R. Lewis & J. M. Haviland-Jones (Eds.), The handbook of emotion (2nd Edition, pp. 173–191). New York: Guilford Press. Cacioppo, J. T., Gardner, W. L., & Berntson, G. G. (1997). Beyond bipolar conceptualizations and measures: The case of attitudes and evaluative space. Personality and Social Psychology Review, 1, 3–25. Cacioppo, J. T., Klein, D. J., Bernston, G. G., & Hatfield, E. (1993). The psychophysiology of emotion. In Handbook of emotions (pp. 119– 142). New York: Guilford Press. Chapados, C., & Levitin, D. J. (2008). Cross-modal interactions in the experience of musical performances: Physiological correlates. Cognition, 108, 639–651. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd Edition). Hillsdale, NJ: Lawrence Erlbaum Associates Inc. Cook, E. I., & Turpin, G. (1997). Differentiating orienting, startle, and defense responses: The role of affect and its implications for psychopathology. In Attention and orienting: Sensory and motivational processes (pp. 137–164). Mahwah, NJ: Lawrence Erlbaum Associates. Craig, K. D., & Patrick, C. J. (1985). Facial expression during induced pain. Journal of Personality and Social Psychology, 48, 1080–1091. Critchley, H. D., Mathias, C. J., Josephs, O., O’Doherty, J., Zanini, S., Dewar, B. K., et al. (2003). Human cingulate cortex and autonomic control: Converging neuroimaging and clinical evidence. Brain, 126, 2139–2152. Dawson, G. J. (2007). The electrodermal system. In J. T. Cacioppo, L. G. Tassinary, & G. G. Bernston (Eds.), Handbook of psychophysiology (Third Edition, pp. 159–181). Cambridge: Cambridge University Press.
348 Devinsky, O., Morrell, M. J., & Vogt, B. A. (1995). Contributions of anterior cingulate cortex to behaviour. Brain, 118, 279–306. Dindo, L., & Fowles, D. C. (2008). The skin conductance orienting response to semantic stimuli: Significance can be independent of arousal. Psychophysiology, 45, 111–118. Ehrle, N. (1998). Traitement temporel de l’information auditive et lobe temporal. Unpublished thesis, Universite´ de Reims ChampagneArdennes. Ellis, E., & Simons, R. F. (2005). The impact of music on subjective and physiological indices of emotion while viewing films. Psychomusicology, 19, 15–40. Epstein, S., & Fenz, D. (1962). Theory and experiment on the measurement of approach-avoidance conflict. Journal of Abnormal and Social Psychology, 64, 97–112. Fenz, D., & Epstein, S. (1962). Measurement of approach-avoidance conflict along a stimulus dimension by a thematic apperception test. Journal of Personality, 30, 613–632. Fishman, Y. I., Volkov, I. O., Noh, M. D., Garell, P. C., Bakken, H., Arezzo, J. C., et al. (2001). Consonance and dissonance of musical chords: Neural correlates in auditory cortex of monkeys and humans. The Journal of Neurophysiology, 86, 2761–2788. Foss, A. H., Altschuler, E. L., & James, K. H. (2007). Neural correlates of the Pythagorean ratio rules. NeuroReport, 18, 1521–1525. Fridlund, A. J., & Cacioppo, J. T. (1986). Guidelines for human electromyographic research. Psychophysiology, 23, 567–589. Gaser, C., & Schlaug, G. (2003). Brain structures differ between musicians and non-musicians. The Journal of Neuroscience, 23, 9240–9245. Gosselin, N., Samson, S., Adolphs, R., Noulhiane, M., Roy, M., Hasboun, D., et al. (2006). Emotional responses to unpleasant music correlates with damage to the parahippocampal cortex. Brain, 129, 2585–2592. Grewe, O., Nage, F., Kopiez, R., & Altenmuller, E. (2007). Emotions over time: Synchronicity and development of subjective, physiological, and facial affective reactions to music. Emotion, 7, 774–788. Hauser, M. D., & McDermott, J. (2003). The evolution of the music faculty: A comparative perspective. Nature Neuroscience, 6, 663–668. Helmholtz, H. (1954/1877). On the sensation of tones. New York: Dover. Ito, T. A., Cacioppo, J. T., & Lang, P. J. (1998). Eliciting affect using the International Affective Picture System: Trajectories through evaluative space. Personality and Social Psychology Bulletin, 24, 855–879. Ito, T. A., Larsen, J. T., Smith, N. K., & Cacioppo, J. T. (1998). Negative information weighs more heavily on the brain: The negativity bias in evaluative categorizations. Journal of Personality and Social Psychology, 75, 887–900. James, C. E., Britz, J., Vuilleumier, P., Hauert, C. A., & Michel, C. M. (2008). Early neuronal responses in right limbic structures mediate harmony incongruity processing in musical experts. Neuroimage, 42, 1597–1608. Keltner, D., & Bonanno, G. A. (1997). A study of laughter and dissociation: Distinct correlates of laughter and smiling during bereavement. Journal of Personality and Social Psychology, 73, 687–702. Khalfa, S., Guye, M., Peretz, I., Chapon, F., Girard, N., Chauvel, P., et al. (2008). Evidence of lateralized anteromedial temporal structures involvement in musical emotion processing. Neuropsychologia, 46, 2485–2493. Khalfa, S., Peretz, I., Blondin, J. P., & Robert, M. (2002). Event-related skin conductance responses to musical emotions in humans. Neuroscience Letters, 328, 145–149. Khalfa, S., Roy, M., Rainville, P., Dalla Bella, S., & Peretz, I. (2008). Role of tempo entrainment in psychophysiological differentiation of happy and sad music? International Journal of Psychophysiology, 68, 17–26. Koelsch, S., Fritz, T., Schulze, K., Alsop, D., & Schlaug, G. (2005). Adults and children processing music: An fMRI study. Neuroimage, 25, 1068–1076. Koelsch, S., Fritz, T., Von Cramon, D. Y., Muller, K., & Friederici, A. D. (2006). Investigating emotion with music: An fMRI study. Human Brain Mapping, 27, 239–250. Koelsch, S., Kilches, S., Steinbeis, N., & Schelinski, S. (2008). Effects of unexpected chords and of performer’s expression on brain responses and electrodermal activity. PLoS ONE, 3, 2631. Koelsch, S., Remppis, A., Sammler, D., Jentschke, S., Mietchen, D., Fritz, T., et al. (2007). A cardiac signature of emotionality. European Journal of Neuroscience, 26, 3328–3338.
D. Dellacherie et al. Krumhansl, C. L. (1997). An exploratory study of musical emotions and psychophysiology. Canadian Journal of Experimental Psychology, 51, 336–353. Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1997). Motivated attention: Affect, activation, and action. In P. J. Lang, R. F. Simons, & M. T. Balaban (Eds.), Attention and orienting: Sensory and motivational processes (pp. 97–135). Mahwah, NJ: Lawrence Erlbaum Associates. Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1998). Emotion, motivation, and anxiety: Brain mechanisms and psychophysiology. Biological Psychiatry, 44, 1248–1263. Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1999). International Affective Picture System (IAPS): Technical manual and affective ratings. Gainsville, FL: The Center for Research in Psychophysiology, University of Florida. Liddell, B. J., Brown, K. J., Kemp, A. H., Barton, M. J., Das, P., Peduto, A., et al. (2005). A direct brainstem-amygdala-cortical ‘alarm’ system for subliminal signals of fear. NeuroImage, 24, 235–243. Masataka, N. (2006). Preference for consonance over dissonance by hearing newborns of deaf parents and of hearing parents. Developmental Science, 9, 46–50. McNair, D. M., Lorr, M., & Droppleman, L. F. (1992). Profile of mood states (Revised). San Diego, CA: EdITS: Educational and Industrial Testing Service. Meyer, L. B. (1956). Emotion and meaning in music. Chicago: University of Chicago Press. Minati, L., Rosazza, C., D’Incerti, L., Pietrocini, E., Valentini, L., Scaioli, V., et al. (2009). Functional MRI/event-related potential study of sensory consonance and dissonance in musicians and nonmusicians. NeuroReport, 20, 87–92. Mineka, S., & Ohman, A. (2002). Phobias and preparedness: The selective, automatic, and encapsulated nature of fear. Biological Psychiatry, 52, 927–937. Nater, U. M., Abbruzzese, E., Krebs, M., & Ehlert, U. (2006). Sex differences in emotional and psychophysiological responses to musical stimuli. International Journal of Psychophysiology, 62, 300–308. Norris, C. J., Larsen, J. T., & Cacioppo, J. T. (2007). Neuroticism is associated with larger and more prolonged electrodermal responses to emotionally evocative pictures. Psychophysiology, 44, 823–826. Ohman, A., & Soares, J. J. (1994). ‘‘Unconscious anxiety’’: Phobic responses to masked stimuli. Journal of Abnormal Psychology, 103, 231–240. Ohman, A., & Wiens, S. (2003). On the automaticity of autonomic responses in emotion: An evolutionary perspective. In R. J. Davidson, K. R. Scherer, & H. H. Goldsmith (Eds.), Handbook of affective sciences (pp. 256–275). Oxford: Oxford University Press. Pallesen, K. J., Brattico, E., Bailey, C., Korvenoja, A., Koivisto, J., Gjedde, A., et al. (2005). Emotion processing of major, minor, and dissonant chords: A functional magnetic resonance imaging study. Annals of the New York Academy of Science, 1060, 450–453. Papa, A., & Bonanno, G. A. (2008). Smiling in the face of adversity: The interpersonal and intrapersonal functions of smiling. Emotion, 8, 1–12. Passynkova, N., Neubauer, H., & Scheich, H. (2007). Spatial organization of EEG coherence during listening to consonant and dissonant chords. Neuroscience Letters, 412, 6–11. Peeters, G., & Czapinski, J. (1990). Positive-negative asymmetry in evaluations: The distinction between affective and informational negativity effects. In W. Stroebe & M. Hewstone (Eds.), European review of social psychology (Vol. 1, pp. 33–60). New York: Wiley. Peretz, I., Blood, A. J., Penhune, V., & Zatorre, R. (2001). Cortical deafness to dissonance. Brain, 124, 928–940. Peretz, I., Gagnon, L., & Bouchard, B. (1998). Music and emotion: Perceptual determinants, immediacy, and isolation after brain damage. Cognition, 68, 111–141. Perpina, C., Leonard, T., Bond, J. T., Bond, A., & Banos, R. (1998). Selective processing of food- and body-related information and autonomic arousal in patients with eating disorders. The Spanish Journal of Psychology, 1, 3–10. Plomp, R., & Levelt, W. J. (1965). Tonal consonance and critical bandwidth. The Journal of the Acoustical Society of America, 38, 548–560. Prkachin, K. M., & Solomon, P. E. (2008). The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain. Pain, 139, 267–274.
Psychophysiology of musical emotion Regnault, P., Bigand, E., & Besson, M. (2001). Different brain mechanisms mediate sensitivity to sensory consonance and harmonic context: Evidence from auditory event-related brain potentials. Journal of Cognitive Neuroscience, 13, 241–255. Roy, M., Mailhot, J. P., Gosselin, N., Paquette, S., & Peretz, I. (2008). Modulation of the startle reflex by pleasant and unpleasant music. International Journal of Psychophysiology, 71, 37–42. Sammler, D., Grigutsch, M., Fritz, T., & Koelsch, S. (2007). Music and emotion: Electrophysiological correlates of the processing of pleasant and unpleasant music. Psychophysiology, 44, 293–304. Sanchez-Navarro, J. P., Martinez-Selva, J. M., & Roman, F. (2006). Uncovering the relationship between defence and orienting in emotion: Cardiac reactivity to unpleasant pictures. International Journal of Psychophysiology, 61, 34–46. Schellenberg, E. G., & Trainor, L. J. (1996). Sensory consonance and the perceptual similarity of complex-tone harmonic intervals: Tests of adult and infant listeners. Journal of the Acoustical Society of America, 100, 3321–3328. Schneider, P., Scherg, M., Dosch, H. G., Specht, H. J., Gutschalk, A., & Rupp, A. (2002). Morphology of Heschl’s gyrus reflects enhanced activation in the auditory cortex of musicians. Nature Neuroscience, 5, 688–694. Schoen, D., Regnault, P., Ystad, S., & Besson, M. (2005). Sensory consonance: An ERP Study. Music Perception, 23, 105–117. Spielberger, C. D. (1983). Manual for the State-Trait Anxiety. Palo Alto: Consulting Psychologists Press Inc. Steinbeis, N., Koelsch, S., & Sloboda, J. A. (2006). The role of harmonic expectancy violations in musical emotions: Evidence from subjective, physiological, and neural responses. Journal of Cognitive Neuroscience, 18, 1380–1393. Stormark, K. M., Laberg, J. C., Nordby, H., & Hugdahl, K. (2000). Alcoholics’ selective attention to alcohol stimuli: Automated processing? Journal of Studies on Alcohol and Drugs, 61, 18–23.
349 Tabachnick, B., & Fidell, L. S. (2001). Using multivariate statistics (4th Edition). Boston: Allyn & Bacon. Taylor, S. E. (1991). Asymmetrical effects of positive and negative events: The mobilization-minimization hypothesis. Psychological Bulletin, 110, 67–85. Terhard, E. (1984). The concept of musical consonance: A link between music and psychoacoustics. Music Perception, 1, 176–195. Trainor, L. J., & Heinmiller, B. M. (1998). The development of evaluative responses to music: Infants prefer to listen to consonance over dissonance. Infant Behavior & Development, 21, 77–88. Trainor, L. J., Tsang, C. D., & Cheung, V. H. W. (2002). Preference for sensory consonance in 2- and 4-month-old infants. Music Perception, 20, 187–194. Tramo, M. J., Cariani, P. A., Delgutte, B., & Braida, L. D. (2001). Neurobiological foundations for the theory of harmony in Western tonal music. Annals of the New York Academy of Science, 930, 92–116. Witvliet, C. V., & Vrana, S. R. (1995). Psychophysiological responses as indices of affective dimensions. Psychophysiology, 32, 436–443. Witvliet, C. V., & Vrana, S. R. (2007). Play it again Sam: Repeated exposure to emotionally evocative music polarises liking and smiling responses, and influences other affective resports, facial EMG, and heart rate. Cognition and Emotion, 21, 3–25. Zajonc, R. B. (1980). Feeling and thinking: Preferences need no inferences. American Psychologist, 35, 151–175. Zentner, M. R., & Kagan, J. (1996). Perception of music by infants. Nature, 383, 29.
(Received April 20, 2009; Accepted April 13, 2010)
Psychophysiology, 48 (2011), 350–361. Wiley Periodicals, Inc. Printed in the USA. Journal compilation r 2010 Society for Psychophysiological Research No claim to original US government works DOI: 10.1111/j.1469-8986.2010.01074.x
Abnormal mechanisms of antisaccade generation in schizophrenia patients and unaffected biological relatives of schizophrenia patients
SEUNG SUK KANG,a,b DAPHNE P. DIONISIO,a and SCOTT R. SPONHEIMc,b,a a
Department of Psychology, University of Minnesota, Twin Cities, Minnesota Department of Psychiatry, University of Minnesota, Twin Cities, Minnesota c Veterans Affairs Medical Center, Minneapolis, Minnesota b
Abstract Although errant saccadic eye movements may mark genetic factors in schizophrenia, little is known about abnormal brain activity that precedes saccades in individuals with genetic liability for schizophrenia. We investigated electrophysiological activity preceding prosaccades and antisaccades in schizophrenia patients, first-degree biological relatives of schizophrenia patients, and control subjects. Prior to antisaccades, patients had reduced potentials over lateral prefrontal cortex. Smaller potentials were associated with worse antisaccade performance. Relatives also exhibited reduced pre-saccadic potentials over lateral frontal cortex but additionally had reduced potentials over parietal cortex. Both patients and relatives tended toward increased activity over orbital frontal cortex prior to saccades. Results are consistent with lateral prefrontal dysfunction marking genetic liability for schizophrenia and underlying deficient saccadic control. Descriptors: Schizophrenia, Eye-movement control, EEG, Genetics
Recently, large multi-site studies have used antisaccade performance as an endophenotype for studying the genetics of the disorder (Calkins, Dobie, Cadenhead, Olincy, Freedman, et al., 2007). Nonetheless, since the first identification of elevated error rates in biological relatives of schizophrenia patients on antisaccade tasks (Clementz, McDowell, & Zisook, 1994), there has been debate about which biological relatives under what experimental conditions exhibit impaired antisaccade performance (Levy, O’Driscoll, Matthysse, Cook, Holzman, & Mendell, 2004). Because errors in ocular motion may serve as a behavioral marker of genetically related pathophysiology in schizophrenia, it is informative to investigate neural processes that underlie the control of saccades. Two functional magnetic resonance imaging (fMRI) studies that investigated brain function of biological relatives of schizophrenia patients during a volitional saccade task reported diminished brain activations in relatives. Relatives exhibited limited activity in the caudate nucleus during antisaccades as compared to prosaccades (Raemaekers, Ramsey, Vink, van den Heuvel, & Kahn, 2006) as well as decreased activations in dorsolateral prefrontal cortex, anterior cingulate cortex, cuneus, insula, and middle occipital gyrus during volitional saccade tasks (e.g., antisaccade and ocular motor delayed response) (Camchong, Dyckman, Austin, Clementz, & McDowell, 2008). Although these findings are suggestive of frontal-striatal and prefrontal abnormalities marking risk for schizophrenia, it is unclear whether the functional brain abnormalities precede eye movements, leaving open the possibility that the activations were a consequence rather than a cause of errant saccades. To date, no
Individuals with schizophrenia have difficulty inhibiting the reflexive movement of the eye toward a stimulus in the periphery when instructed to direct their gaze in the opposite direction (Hutton & Ettinger, 2006). Because excessive saccadic errors during an ‘‘antisaccade’’ task have been observed in first-degree biological relatives of schizophrenia patients (see Calkins, Curtis, Iacono, & Grove, 2004 for a recent review), it has been suggested that deficits in eye-movement control are indicators of genetic liability (i.e., endophenotypes) for schizophrenia and therefore mark brain abnormalities that reflect central nervous system effects of genes predisposing schizophrenia (Gottesman & Gould, 2003; Schulze, MacCabe, Rabe-Hesketh, Crawford, Marshall, et al., 2006). We thank Monica Calkins, Ph.D., Clayton Curtis, Ph.D., and Kevin Haroian for assistance in task design and implementation. Kathryn A. McGuire, Ph.D., John. J. Stanwyck, Sarah M. Sass, and Robb Hunter assisted with task implementation and data acquisition, and Carly Smitkowski assisted in scoring the eye movement data. We are also grateful to Jennifer McDowell, Ph.D., for consultation and guidance during the processing and quantification of eye movement data. This work was supported by grants from the Department of Veterans Affairs Medical Research Service, the National Institutes of Mental Health (5R24MH069675) to Scott Sponheim, as well as by the Mental Illness and Neuroscience Discovery (MIND) Institute and the Mental Health Patient Service Line at the Veterans Affairs Medical Center, Minneapolis, Minnesota. Address correspondence to: Scott R. Sponheim, Ph.D., Minneapolis VA Medical Center (116B), One Veterans Drive, Minneapolis, MN 55417. E-mail:
[email protected] 350
Mechanisms of antisaccade generation in schizophrenia published studies of individuals who carry genetic liability for schizophrenia have specifically isolated neural anomalies that precede the onset of saccadic eye movements; therefore, it remains to be determined which neural events are the cause of impairments on the antisaccade task. To characterize neural events contributing to poor antisaccade performance in schizophrenia, we used electrophysiological recordings to capture brain activity prior to the execution of saccades. Specifically, we examined the neural responses of schizophrenia patients, unaffected first-degree biological relatives of schizophrenia patients, and control subjects during antisaccade and prosaccade tasks. The study employed two fixation conditions to facilitate delineation of relevant pre-saccadic neural events. Inclusion of unaffected biological relatives provided a test of whether anomalies were independent of schizophrenia spectrum psychopathology in a sample likely to carry genetic liability for the disorder. In schizophrenia, deficient ocular-motor control, as well as deficits in working memory and sustained attention, are associated with dysfunction of both prefrontal (PFC) and parietal cortices (Carter, Perlstein, Ganguli, Brar, Mintun, & Cohen, 1998; McDowell, Brown, Paulus, Martinez, Stewart, et al., 2002; Ojeda, Ortuno, Arbizu, Lopez, Marti-Climent, et al., 2002). Scientists have proposed that these functional abnormalities result from a failure in a dopaminergic PFC mechanism (ServanSchreiber, Cohen, & Steingard, 1996), a dysfunction in a frontoparietal attentional network (Maruff, Danckert, Pantelis, & Currie, 1998), or a functional disconnection between PFC and parietal cortex (Kim, Kwon, Park, Youn, Kang, et al., 2003). Functional MRI studies indicate that schizophrenia patients fail to normally activate PFC during antisaccade execution (Fukumoto-Motoshita, Matsuura, Ohkubo, Ohkubo, Kanaka, et al., 2009; McDowell, et al., 2002; Tu, Yang, Kuo, Hsieh, & Su, 2006). These studies suggest that antisaccade deficits in schizophrenia are primarily due to dysfunction in the PFC and are consistent with the cortical region being central to saccadic control (Guitton, Buchtel, & Douglas, 1985; Pierrot-Deseilligny, Rivaud, Gaymard, Muri, & Vermersch, 1995; Rivaud, Muri, Gaymard, Vermersch, & Pierrot-Deseilligny, 1994). Yet, slowed saccadic responses of schizophrenia patients have also been related to reduced white matter organization in anterior cingulate cortex, parietal cortex, and the frontal eye field (Manoach, Ketwaroo, Polli, Thakkar, Barton, et al., 2007). Studies that have specifically examined neural activity preceding eye movements showed impaired modulation of brain activity (i.e., the contingent negative variation) in schizophrenia patients prior to antisaccades, suggesting poor eye movement preparation, selection, and execution (Franke, Reuter, Schulz, & Kathmann, 2007; Reuter, Herzog, Endrass, & Kathmann, 2006; Reuter, Jager, Bottlender, & Kathmann, 2007), which has been interpreted as reflective of prefrontal dysfunction (Klein, Heinks, Andresen, Berg, & Moritz, 2000). In addition, previous investigations of eye movement control in schizophrenia have solely examined pre-saccadic neural activity locked to stimulus events and not the saccade itself and, therefore, may not fully reveal brain abnormalities related to saccade preparation and generation in schizophrenia. Finally, given evidence of parietal cortex involvement in goal-directed eye movement through spatial computation and sensorimotor transformations (Moon, Barton, Mikulski, Polli, Cain, et al., 2007; Rafal, 2006; Zhang & Barash, 2000) and parietal cortex activity during inhibitory periods prior to saccade generation (Ettinger, Ffytche, Kumari, Kathmann, Reuter, et al., 2007), it is possible that dysfunction of parietal brain regions
351 contributes to abnormal antisaccadic control in schizophrenia patients and their biological relatives. Researchers have noted that characteristics of the fixation stimulus before saccades have effects on saccade latencies. Onset latencies of correct antisaccades are reduced for a ‘‘step’’ condition (where the fixation stimulus extinguishes simultaneously with the appearance of the peripheral stimulus) compared to an ‘‘overlap’’ condition (where fixation and peripheral stimuli briefly overlap in time) (Braun & Breitmeyer, 1988; Fischer & Weber, 1993, 1997). McDowell and Clementz (1997) found that overlap fixation was superior for increasing group separation between schizophrenia patients and nonpsychiatric control subjects. Yet, it is unclear what mechanism for antisaccade generation is affected by overlap fixation and how such a mechanism is altered in schizophrenia. Overlap effects can be understood in terms of Posner’s theory of visual attention, which posits three different elements of attentionFdisengage, move, engage (Fischer & Weber, 1993; Posner, 1980; Posner, Walker, Friedrich, & Rafal, 1984). Accordingly, a benefit in latency is obtained when attention is engaged at the position where the stimulus is going to be presented and a cost is exacted when the stimulus appears somewhere else. Studies of patients with brain lesions suggest disengagement of attention is unique to the parietal lobe (Posner, et al., 1984). The overlap fixation condition requires subjects to disengage visual attention from the fixation location and thus likely elicits additional parietal activation before the eyes are directed to the target location. The increased demand to disengage attention by parietal cortex does not exist in the step fixation. Therefore, an antisaccade task with overlap fixation may effectively elicit abnormal activity not only in prefrontal regions but also in parietal cortex in schizophrenia patients. Cue disengagement for successful saccade performance has been proposed as mediated by parietal cortex (Richards, 2003). Analyzing electrophysiological activity of normal subjects from pre-stimulus, post-stimulus, and pre-saccadic periods, Richards (2003) identified pre-saccadic components that peaked over fronto-central, parietal, fronto-polar, and lateral frontal brain regions (the maximum of principal components were found at scalp sites FCz/Cz, PZ, Fp1/Fp2, and Af2/Af4/F6 in the 10–20 electrode system, respectively). The component with the parietal maximum was reduced when cues were provided prior to target onset that informed subjects of the upcoming target location. In the absence of such a cue, the parietal component was increased, possibly reflecting demands on spatial attention to efficiently locate the upcoming target stimulus. The fronto-central negative component observed by Richards (2003) was maximal in the prestimulus period and interpreted as the contingent negative variation (CNV) that had been previously identified prior to saccades (Everling, Krappmann, & Flohr, 1997; Klein, et al., 2000). Richards (2003) identified fronto-polar and lateral frontal electrophysiological components perhaps due to the use of a high density of electrodes across the scalp and various target cueing conditions where pro- and antisaccade trials were mixed in a block. The fronto-polar and lateral frontal components steadily increased from about 300 to 200 ms before saccade onset and the magnitude was greater in antisaccades than prosaccades. Richards (2003) localized the components to orbital PFC and anterior/lateral PFC, respectively. Although the role of the orbital PFC neural activity is not fully understood, the lateral PFC neural activity was found to be increased in antisaccades compared to prosaccades as well as when a cued condition was compared to the no-cue condition. These saccade type and cue effects
352
S. S. Kang et al.
suggest that the underlying neural activity may reflect planning of the targeted eye movement. In other words, the lateral frontal component activity might reflect contextual processing by dorsolateral PFC to guide targeted eye movements. Given previously identified PFC and posterior parietal cortex dysfunction in schizophrenia, pre-saccadic electrophysiological components identified by Richards (2003) may well be altered in the disorder. To investigate neural activity preceding saccades and examine possible abnormal PFC and parietal function associated with antisaccade performance in schizophrenia, we studied electrophysiological responses of schizophrenia patients and first-degree biological relatives of schizophrenia patients using antisaccade and prosaccade paradigms that included step and overlap fixation conditions. We hypothesized that schizophrenia patients and their relatives would show abnormal electrophysiological activity over PFC and parietal cortex, especially in the overlap fixation condition. We focused on pre-saccadic positive potentials that have been observed prior to visually triggered saccades and modulated by whether the eye movement is a prosaccade or antisaccade (Everling, et al., 1997), as well as signals possibly related to the three electrophysiological components of greatest magnitude at lateral frontal, parietal, and fronto-polar electrodes prior to saccades (Richards, 2003). Through contrasts of the electrophysiological activity preceding antisaccades and prosaccades, we were able to demonstrate abnormal neural responses that appear to reflect genetic liability for schizophrenia.
Methods Participants Twenty participants diagnosed with DSM-IV schizophrenia, 21 of their first-degree biological relatives, and 20 control subjects were studied. Five schizophrenia patients and two relatives were excluded in the later analysis due to having too few trials without blink and saccadic errors for computing event-related potentials (ERPs). Table 1 presents the characteristics of participants. Schizophrenia patients were recruited from the Minneapolis Veterans Affairs Medical Center and regional mental health centers, and research staff identified first-degree biological relatives of patients by completing a pedigree from the patient’s report. Interested relatives completed a telephone interview to determine their demographic and medical characteristics and
were excluded if they had a physical problem that would render study measures impossible to obtain, or were younger than age 18 or older than age 68. Control subjects were recruited through flyer advertisement throughout the community. To obtain diagnostic information, a trained doctoral-level clinical psychologist completed the Diagnostic Interview for Genetic Studies (Nurnberger, Blehar, Kaufmann, York-Cooler, Simpson, et al., 1994) (DIGS) with each patient. Using all available clinical information for a patient, the interviewer completed the Operational Criteria for Psychotic Illness (McGuffin, Farmer, & Harvey, 1991) (OPCRIT) to derive a DSM-IV diagnosis. A second doctoral-level clinical psychologist functioned as a consensus reviewer and also completed the OPCRIT for the participant. Any diagnostic disagreement between the interviewer and consensus reviewer was resolved by reviewing OPCRIT items on which ratings differed. From the clinical interview, the psychologist rated current symptomatology using the 24-item version of the Brief Psychiatric Rating Scale (Ventura, Nuechterlein, Subotnik, Gutkind, & Gilbert, 2000) (BPRS). Of the patients in the study, 14 were being prescribed antipsychotic medication (exclusively novel antipsychotics). The average chlorpromazine equivalent for antipsychotic dosages was 565 mg (448 SD). Nine patients were being prescribed antidepressant medication. To assess psychopathology in first-degree relatives and control subjects, a doctoral-level psychologist or a trained and supervised research assistant completed the Structured Clinical Interview for DSM-IV Axis I Disorders (First, Spitzer, Gibbon, & Williams, 1996) (SCID-I). Relatives and controls also completed the Structured Clinical Interview for DSM-IV Axis II Personality Disorders Personality Questionnaire (Ekselius, Lindstrom, von Knorring, Bodlund, & Kullgren, 1994) (SCIDII-PQ), the Structured Interview for Schizotypy (Kendler, Lieberman, & Walsh, 1989) (SIS) and completed ratings of schizotypal, schizoid, and paranoid personality disorders. To assess self-reported schizotypal characteristics, we administered the Schizotypal Personality Questionnaire (Raine, 1991) (SPQ) to first-degree biological relatives of patients and control subjects. IQ was estimated using the Block Design and Vocabulary subtests of the Wechsler Adult Intelligence Scale, Third Edition (Brooker & Cyr, 1986). No relatives had a schizophrenia spectrum disorder. Although four relatives had past major depressive episodes, they were in full remission when they participated in the study.
Table 1. Participant Characteristics Patients N 5 15
Relatives N 5 19
Controls N 5 20
Variable Age (year) Percent female Education (years) Estimated IQ BPRS total score SPQ total scoreb
Mean (SD)
Mean (SD)
Mean (SD)
Test value (df)
p value
38.20 (9.77) 26.67 12.80 (3.10) 96.80 (15.19) 40.87 (12.00)a NA
49.16 (9.52) 68.42 14.26 (2.47) 100.11 (14.91) NA 13.35 (10.27)
40.60 (10.25) 45.00 15.60 (2.11) 111.90 (10.35) NA 7.94 (4.70)
F(2,51) 5 6.07 w2(2) 5 5.98 F(2,51) 5 5.23 F(2,51) 5 6.34 NA F(1,32) 5 3.90
.004 .050 .009 .003 NA .057
Note: Estimated IQ was derived from Vocabulary and Block Design subtests (Brooker & Cyr, 1986) of the Wechsler Adult Intelligence ScaleFRevised. SD 5 standard deviation; IQ 5 intelligence quotient; BPRS 5 Brief Psychiatric Rating Scale 24-item version (Ventura et al., 2000); NA 5 not applicable; SPQ 5 Schizotypal Personality Questionnaire (Raine, 1991). a Mean BPRS score was equivalent to an average item rating of 1.8, which is between absent (1) and very mild (2) for prominence. b SPQ score failed to be associated with saccade accuracy and latency indices.
Mechanisms of antisaccade generation in schizophrenia Gender had no effect on behavioral performance, and age was only associated with prosaccade latency (see Table 1). All participants were free of serious physical health problems, and absent of known neurological hard signs. Exclusion criteria included English as a second language, charted IQ less than 70 or a diagnosis of mental retardation, current alcohol or drug abuse, past drug dependence, a current or past central nervous system disease or condition, a medical condition or disease with likely significant central nervous system effects, history of head injury with skull fracture or loss of consciousness of greater than 20 min, significant tardive dyskinesia as indicated by a Dyskinesia Identification System: Condensed User Scale (Sprague, Kalachnik, & Slaw, 1989) (DISCUS). Procedure Each participant was seated in a dimly lit room with their head stabilized by a chin and forehead rest. A computer display faced subjects at 51 cm from the eyes. Before the task, subjects were presented with calibration targets at 51, 101, and 151 on either side of the central fixation. Experimental trials started with the 0.51 illuminated target dot at fixation. Following a 2- to 3-s pseudorandom interval, the dot was extinguished and then appeared at 151 to the right or left of central fixation for 1 s. The trial ended with the dot returning to central fixation. In the prosaccade task, subjects were instructed to move their eyes as quickly and accurately as possible to the position of the dot. In the antisaccade task, subjects were instructed to look as quickly as possible to the position opposite the dot with respect to the central fixation point. The subjects were instructed not to blink during the saccades. Two different, pseudorandomly intermixed central fixation conditions were used. In the step condition, the central fixation disappeared contemporaneous with the illumination of the cue. In the overlap condition, the central fixation dot remained illuminated for 200 ms after illumination of the target dot on the periphery (see Figure 1 for task procedures). Eight blocks of 24 trials were presented (pro, anti, anti, pro, pro, anti, anti, pro). There were 48 total trials per fixation condition (step, overlap) per saccade type (pro, anti) with equal numbers of trials requiring saccades to the left and right of central fixation. Electrophysiological Data Collection and Analyses Electroencephalograms (EEG) were recorded using a 16-bit analog-to-digital amplifier and 29 tin electrodes embedded in an
353 elastic cap (Electro-Cap International, Inc. [ECI], Eaton, OH). Electrodes were placed on the head to conform to 10–10 nomenclature and referenced to the left earlobe. Electrodes were filled with ECI gel, and sites were abraded to yield impedances below 5 kO. EEG signals were digitized at a rate of 500 Hz with 0.5 low frequency and 100 Hz high frequency filters and a 60 Hz notch filter. Vertical electro-oculograms (VEOG) recorded from above and below the right eye were used to reject epochs containing ocular artifact, while horizontal electro-oculograms (HEOG) recorded from outer ocular canthi were used to measure horizontal eye movements. Following data collection, recordings from scalp sites were rereferenced to linked earlobes. Using HEOG channel potentials, we quantified saccade performance in terms of saccade latency and direction (i.e., not amplitude gain). The onset of a saccade was identified in the HEOG as the beginning of an abrupt deviation from baseline. For saccadic eye-movement identification, four criteria were used: (1) velocity of eye movement (4301/sec), (2) degree of eye movement (425% of target movement), (3) timing of saccade (from 100 to 800 ms after cue onset), and (4) absence of blink (from 250 ms before cue onset to onset of saccade). The target saccade movement for each subject was determined by the mean of eye movements of calibration trials. Inspection of saccade latency distributions showed that 800 ms was an appropriate limit for saccade response. Saccades that occurred ! 250 to 100 ms relative to stimulus onset were regarded as anticipatory saccades (Fischer & Weber, 1993). During the saccade scoring, trials whose saccade onsets could not be clearly determined were rejected. To verify the accuracy of saccade scoring using HEOG signals, we also scored saccade performances with infra-red eyemovement tracking data, which were simultaneously collected with HEOG and EEG recordings for 10 subjects (1 schizophrenia patient, 5 control subjects, 4 relatives). Saccade latencies measured through HEOG and infra-red signals correlated .985, and saccade direction for the two signals perfectly agreed (i.e., tetrachoric correlation was 1.00). Thus, HEOG provided a very accurate indication of both saccade latency and direction. To examine potentials that were response-related, the 0 ms point for each EEG epoch was set at the saccade onset. EEG recordings were epoched to extend from 500 ms pre-saccade to 250 ms post-saccade in order to include saccade-related brain activity from the onset of a cue to the onset of a saccade and
Figure 1. Antisaccade and prosaccade task procedure showing the order of stimulus presentation for step and overlap conditions (left panel) and intended eye-movement directions at cue onset in each task condition (right panel).
354 beyond. EEG epochs were visually inspected, and some epochs whose amplitudes exceeded ! 100 mV before saccade onsets were rejected to ensure only epochs free of eye- and body-movement artifacts were included for averaging. Only trials with correct saccadic responses were averaged. The minimum number of trials averaged for a subject in any condition was 25. Average response-related potentials were computed for each task, and fixation condition and the baseline for averages was defined as the median voltage from 500 to 400 before saccade onset (i.e., first 100 ms of the epoch). The 500 to 400 baseline period was selected to use a time period prior to any stimulus movement but that was near the onset of the saccade stimulus (most saccade latencies ranged between " 250 and " 200 ms for pro-saccades and " 350 and " 300 ms for anti-saccades). Saccade accuracy reflected the percentage of trials where the subject emitted a saccade in the correct direction and was defined as the number of correct trials divided by the number of trials without artifacts. Thus, the accuracy measure was computed excluding both incorrect and anticipatory saccades. We subtracted scalp-recorded potentials of prosaccades from antisaccade potentials to generate anti–pro difference potentials that revealed activity specific to antisaccade generation. Statistical Analyses For analyses of saccade performance (i.e., accuracy and latency), we conducted repeated measures analyses of variance (ANOVAs) with one between-subjects factor (group: schizophrenia, relatives, controls) and two within-subjects factors (task: pro-, antisaccade; fixation: overlap, step). Demographic variables (see Table 1) were included as covariates in ANOVAs when groups differed on the variable and it was significantly associated with the behavioral or ERP indices.1 Consistent with the findings in normal controls (Everling et al., 1997; Richards, 2003), temporally and spatially broad saccade-related activity was observed at midline frontal (Fp1, Fp2, Af3, Af4, F3, FZ, F4) and centroparietal (C3, CZ, C4, P3, PZ, P4) electrodes beginning about 210 ms before saccade onset (see Supplementary Materials for depiction of waveforms from all these electrode sites). Because of an absence of clear peaks and valleys in the saccade-locked ERP activity, we computed mean amplitudes for every 30-ms period from " 210 to 0 ms at the thirteen frontal and the centro-parietal electrodes to quantify changes in the magnitude of preparatory neural responses across time. The analysis strategy involved three steps intended to characterize group effects on the broad presaccadic potentials and determine at which points the pre-saccadic potentials were most sensitive to the fixation, saccade task, and group effects. First, to test the hypothesis that schizophrenia patients and their relatives had neural abnormalities that were more evident prior to antisaccades than prosaccades, we conducted a repeated measure ANOVA with one between-subjects factor (group: schizophrenia, relatives, controls) and two withinsubjects factors (task: pro- and antisaccades; electrodes: Fp1, 1 Gender failed to be associated with any behavioral measure or indices of pre-saccadic potentials. Age and IQ were correlated with saccade latencies (age: r 5 .38, p 5 .005; IQ: r 5 " .34, p 5 .011), but not saccade accuracy or any index of pre-saccadic potentials. Inclusion of IQ and age as covariates failed to alter the pattern of significant effects and resulted in main effects of task (without covariates: F(1,51) 5 276.68, po.001; with covariates: F(1,49) 5 26.13, po.001) and fixation (without covariates: F(1,51) 5 217.72, po.001; with covariates: F(1,49) 5 4.89, p 5 .032) effects but no group effect (without covariates: F(2,51) 5 .80, p 5 .450; with covariates: F(2,49) 5 1.10, p 5 .341) on saccade latencies.
S. S. Kang et al. Fp2, Af3, Af4, F3, FZ, F4, C3, CZ, C4, P3, PZ, P4), with the dependent variable being the mean amplitude for all 30-ms time bins from " 210 to 0 ms. The analysis allowed examination of whether groups differed on the broad temporal pre-saccadic potentials noted in other studies to be modulated by saccade type (e.g., Everling et al., 1997). Second, follow-up analyses were used to determine whether the two saccade tasks (pro and anti) and two fixation conditions (step and overlap) were associated with modulation of the pre-saccadic potentials during specific time periods and over select brain regions. We carried out two repeated measures ANOVAsFone with task, electrode, and time as within-subjects factors and the other with fixation, electrode, and time as within-subjects factors. Third, to fully investigate group differences in pre-saccadic potentials previously identified in studies of normative samples (Everling et al., 1997, Richards, 2003), we conducted one-way ANOVAs with an independent variable of group and a dependent variable of mean amplitudes of difference (anti–prosaccades) potentials recorded at select frontal and parietal electrodes. To investigate fixation-specific group differences in the pre-saccadic potentials, these ANOVAs were separately run for each fixation condition. Electrode selection was based on previously observed scalp locations of presaccadic potentials (Richards, 2003) and the hypothesized location of relevant neural activity. Electrodes F4 and F3 represented the lateral frontal component, Cz and Pz represented the parietal component, and Fp1 and Fp2 were chosen for the fronto-polar component. Given the amplitude change of the predominant presaccadic ERP components across time (Richards, 2003), we defined the largest time windows for the group comparisons based on exploratory analyses that compared mean amplitudes of the difference potentials in seven 30-ms time windows at the select electrode across groups. Magnitude of group difference in mean amplitude was estimated as effect size (Cohen’s d). All reported within-subjects effects were Huynh-Feldt corrected. To examine the relationship of electrophysiological activity with saccade performance and medication status, Pearson correlation coefficients were computed between mean amplitude indices and measures of accuracy, latency, and chlorpromazine equivalents.
Results Saccade Accuracy and Latency Prosaccade vs. antisaccade task. Schizophrenia patients, relatives, and control subjects had lower accuracy on the antisaccade task than the prosaccade task (F(1,51) 5 167.10, po.001). There was an interaction between group and task (F(2,51) 5 4.11, p 5 .02). Follow-up analyses indicated that schizophrenia patients had lower saccade accuracy than controls and relatives, and that they exhibited much larger accuracy deficit in the antisaccade task than the prosaccade task (see Table 2). Saccade latency was strongly influenced by task type (F(1,49) 5 26.13, po.001) with longer latencies evident in the antisaccade task compared to the prosaccade task. Analyses of the percentage of anticipatory saccades revealed a group difference only on the antisaccade task (F(2,51) 5 13.15, po.001). Follow-up analyses indicated that in the antisaccade task schizophrenia patients showed anticipatory saccades on more trials than controls (8.3% of trials vs. 3.3%, respectively), while relatives (3.8% of trials) failed to differ from controls. Step vs. overlap fixation. There was an interaction of group and fixation type for saccade accuracy (F(2,51) 5 3.21, p 5 .049).
Mechanisms of antisaccade generation in schizophrenia
355
Table 2. Behavioral Performances in Saccade Tasks
Accuracy Pro-overlap Pro-step Anti-overlap Anti-step Latencyd Pro-overlap Pro-step Anti-overlap Anti-step
Patients (N 5 15)
Relatives (N 5 19)
Controls (N 5 20)
Mean (SD)
Mean (SD)
Mean (SD)
.83 (.11)a .87 (.10) .59 (.17)b .58 (.15)b
.89 (.11) .88 (.09) .75 (.11)c .71 (.12)c
252.10 (50.94) 209.77 (33.96) 344.75 (52.97) 306.11 (60.67)
255.55 (41.92) 198.43 (35.59) 336.77 (64.60) 302.39 (58.44)
.92 (.07) .90 (.07) .73 (.11) .72 (.11) 235.34 (46.70) 188.73 (38.87) 326.91 (62.16) 286.49 (64.85)
Note: Pro 5 prosaccade; Anti 5 antisaccade; Step 5 step fixation condition; Overlap 5 overlap fixation conditions. a Patients had significantly lower accuracy in prosaccades for overlap fixations than controls (t(33) 5 2.64, p 5 .013). b Patients had significantly lower accuracy in antisaccades than controls (t(33) 5 3.79, p 5 .001) and relatives (t(32) 5 3.45, p 5 .001), regardless of fixation condition. c Relatives had significantly higher antisaccade accuracy for overlap fixations compared to step fixations (t(18) 5 2.56, p 5 .020). d All groups had significantly longer saccade latency for overlap fixations compared to step fixations, regardless of saccade tasks. There was no significant group difference in saccade latency.
Analyses revealed that schizophrenia patients had lower prosaccade accuracy for overlap fixation trials compared to step fixation trials, while relatives had higher antisaccade accuracy for overlap fixation compared to step fixation, and the control group’s saccade accuracy did not vary by fixation condition. In addition, schizophrenia patients had a lower proportion of correct prosaccades for overlap fixation trials than controls (.83 vs. .92). The overlap condition produced much longer saccade latencies than the step condition in all groups. There was a main effect of fixation (F(1,49) 5 4.89, p 5 .03), but no group main effect (F(2,49) 5 1.10, p 5 .34) on saccade latency. Pre-Saccadic Neural Activity All groups showed increasingly positive scalp-recorded potentials over time that peaked immediately before saccade onset.2 A repeated measures ANOVA with the three factors (group, task, and electrode) and dependent variable of the mean amplitudes computed for all time-ranges (! 210 " 0 ms) revealed interactions of group, task, and electrode (F(12,612) 5 2.82, p 5 .01) and task and electrode (F(12,612) 5 8.45, po.001), indicating that group differences in pre-saccade potentials were evident during specific task conditions (i.e., antisaccade) at particular electrodes. There were no main effects of group (F(2,51) 5 .37, p 5 .69) and task (F(2,51) 5 .56, p 5 .46), and trends of a main effect of electrode (F(12,612) 5 2.53, p 5 .09) and a group by
2 To test for an effect of saccade direction on neural data, pro- and anti-presaccadic average potentials were also computed for leftward and rightward saccades, collapsing across fixation conditions. A repeated measures ANOVA was used to test for the effect of saccade direction on pre-saccadic potentials. The analysis included within-subjects factors of laterality, time bin, and electrode, with a between-subjects factor of group, and dependent variables being difference potentials between prosaccades and antisaccades at the various time bins. The analysis revealed no effects of laterality (F(1,51) 5 2.17, p 5 .15) or laterality by group (F(1,51) 5 .20, p 5 .82).
electrode interaction effect (F(24,612) 5 2.09, p 5 .10). Potentials prior to prosaccades were characterized by the largest amplitude increases over the centro-parietal area, while those preceding antisaccades were characterized by amplitude increases over frontal brain regions. Follow-up ANOVAs revealed that there was a significant 3-way interaction of task, electrode, and time (F(72,3816) 5 9.09, po.001), suggesting that spatial and temporal variation of pre-saccadic potentials were dependent on saccade type. Pre-saccadic potentials also showed a main effect of fixation (F(1,51) 5 10.46, p 5 .002), an interaction of fixation and time (F(6,306) 5 2.55, p 5 .034), and a trend for an interaction of fixation, time, and electrode (F(72,3672) 5 1.72, p 5 .067), thereby indicating that the effect of different fixations prior to stimulus onset varied across time and scalp location. Hence, omnibus analyses of pre-saccadic potentials indicated that the scalp distribution and time course of the responses were sensitive to diagnostic group, task, and fixation conditions, suggesting that group differences in the neural activity prior to saccades was likely isolated to select electrodes for specific time periods. Therefore, to fully describe group differences leading up to saccade generation, we carried out one-way ANOVAs for difference potentials (anti–prosaccades) at electrodes selected to be at scalp regions where pre-saccadic potentials were previously observed (Everling, et al., 1997; Richards 2003).
Potentials over lateral frontal cortex. Because lateral frontal cortex has been shown to be involved with the control of saccadic eye movements, we examined electrodes over this region for group differences. To isolate activity that varied depending on the saccade task, we submitted the difference in mean amplitudes between antisaccade and prosaccade potentials to analyses. Figure 2 depicts difference waveforms for selected electrodes and the scalp topography of difference potentials. Groups varied in anti–pro difference potentials over right lateral frontal cortex (site F4) from 180 to 120 ms before saccade onset regardless of fixation type (F(2,51) 5 4.68, p 5 .01). Post hoc analyses indicated that both the schizophrenia subjects (t(33) 5 ! 2.73, p 5 .01, Cohen’s d 5 ! .94) and the relatives (t(37) 5 ! 2.33, p 5 .02, d 5 ! .75) had smaller difference potentials than controls from ! 180 to ! 120 ms. A trend for a group difference was also found in difference potentials for overlap fixations over left lateral frontal cortex (site F3) from 180 to 120 ms (F(2,51) 5 2.99, p 5 .059). In post hoc analyses, it was found that schizophrenia patients had smaller difference potentials than controls from ! 180 to ! 120 ms (t(33) 5 ! 2.15, p 5 .04, d 5 ! .75), while in the same time window relatives had somewhat decreased difference potentials, which was not statistically significant (t(37) 5 ! 1.674, p 5 .10, d 5 ! .55). Thus, prior to antisaccades, schizophrenia patients and their relatives exhibited diminished amplitudes over lateral frontal cortex. Figure 3 depicts separate prosaccade and antisaccade potentials for schizophrenia, relative, and control groups at site F4. Difference potentials over lateral frontal cortex for which group effects were evident showed associations with saccade performance. Schizophrenia patients who had larger lateral frontal difference potentials between 180 and 120 ms before saccade onset had better antisaccade accuracy for step fixation trials (F4: r 5 .57, p 5 .03; F3: r 5 .54, p 5 .04), but not for overlap fixation trials. Also, relatives who had larger difference potentials at F3 between 150 and 120 ms before saccade onset for overlap fixations had shorter antisaccade latencies for overlap fixations (r 5 ! .53, p 5 .02),
356
S. S. Kang et al.
Figure 2. Grand averages of difference potentials (antisaccade–prosaccade) for schizophrenia patients, biological relatives of schizophrenia patients, and control subjects preceding saccadic eye movements. Potentials are displayed for three electrodes (left panel) and mean difference amplitudes are depicted across 7 time ranges in 30-ms intervals (middle panel) to show voltage change across time (see text for further descriptions). Note: the red asterisk indicates a significant group difference between patients and controls (po.05); the green asterisk indicates a significant group difference between relatives and controls (po.05).
but an association was absent for overlap difference potentials at other lateral frontal sites as well as for step antisaccade trials. Potentials over parietal cortex. Because of the suspected role of parietal cortex in spatial attention and previously observed augmentation of potentials over the region prior to saccades, we examined activity recorded at central and parietal electrodes. Group main effects were evident for anti–pro difference potentials for overlap fixation trials from ! 180 to ! 120 ms at Cz (F(2,57) 5 4.61, p 5 .01), and from ! 180 to ! 120 ms at Pz (F(2,57) 5 3.55, p 5 .04). Post hoc tests indicated that relatives had more negative amplitudes than controls (Cz: t(37) 5 ! 3.02, p 5 .005, d 5 ! .99; Pz: t(37) 5 ! 2.64, p 5 .01, d 5 ! .87), while schizophrenia patients failed to differ from controls (Cz: t(33) 5 ! 1.23, p 5 .227, d 5 ! .43; Pz: t(33) 5 ! .32, p 5 .75, d 5 ! .11) and relatives (Cz: t(32) 5 ! 1.69, p 5 .101, d 5 ! .60; Pz: t(32) 5 ! 1.98, p 5 .06, d 5 ! .70). Figure 2 panel (d) depicts anti–pro difference potentials at site Pz and the increasingly negative difference potential from ! 200 ms to saccade onset for overlap fixation trials. Inspection of separate prosaccade and antisaccade potentials at site Pz displayed in Figure 3 revealed relatives to have higher amplitudes for prosaccade trials than
antisaccade trials during ! 210 to 0 ms period. Statistical tests of antisaccade overlap fixation potentials confirmed effects observed in difference potentials by revealing group differences over central and parietal sites from 180 to 120 ms before saccades (Cz: F(2,51) 5 3.91, p 5 .03; Pz: F(2,51) 5 4.21, p 5 .02; P3: F(2,51) 5 3.79, p 5 .03). Post hoc tests indicated that amplitudes during this time window at Cz and Pz were smaller for relatives than controls (Cz: t(37) 5 ! 2.70, p 5 .01, d 5 ! .89; Pz: t(37) 5 ! 3.05, p 5 .004, d 5 ! 1.00), while those of the schizophrenia subjects were intermediate and failed to differ from controls (Cz: t(33) 5 ! 1.51, p 5 .14, d 5 ! .53; Pz: t(33) 5 ! .99, p 5 .33, d 5 ! .34) and relatives (Cz: t(33) 5 1.13, p 5 .27, d 5 .40; Pz: t(33) 5 1.63, p 5 .11, d 5 .58). This group difference was not found in the overlap prosaccade potentials. Thus, in contrast with control subjects, relatives had diminished amplitudes over parietal cortex for antisaccades but normal amplitudes for prosaccades on overlap trials. Control subjects and schizophrenia patients did not show substantial parietal voltage differences between prosaccades and antisaccades for overlap fixations over the same pre-saccadic period. Correlation analyses also revealed that relatives who had lower overlap antisaccade potentials from ! 180 to ! 120 ms at the central and parietal sites had
Mechanisms of antisaccade generation in schizophrenia
Figure 3. Pro- and antisaccade potentials over lateral frontal (site F4), parietal (site Pz), and orbital frontal (site Fp2) cortices from which difference potentials were derived. Scalp topographies for controls, relatives, and schizophrenia patients for time bins (shaded regions on waveforms) showing group differences in pre-saccadic difference potentials.
longer latencies for antisaccades on overlap fixation trials (Cz: r 5 ! .58, p 5 .009; Pz: r 5 ! .49, p 5 .03). There was some indication of parietal dysfunction in schizophrenia patients suggested by a trend toward group differences in amplitudes at Pz (F(2,58) 5 2.60, p 5 .08) from 60 to 0 ms prior to prosaccade onset for overlap fixation trials (see Figure 3). Post hoc tests revealed schizophrenia subjects had lower amplitudes than controls (t(33) 5 ! 2.38, p 5 .02, d 5 ! .83), while relatives failed to differ from controls (t(37) 5 ! 1.01, p 5 .32, d 5 ! .33) and patients (t(33) 5 ! 1.29, p 5 .21, d 5 .46). Potentials over frontal polar area. Given findings of frontal polar brain responses being associated with antisaccades (Richards, 2003; Sweeney, Mintun, Kwee, Wiseman, Brown, et al., 1996) and the role of orbital or inferior frontal cortex in response inhibition (Aron, Robbins, & Poldrack, 2004), we examined activity at Fp1 and Fp2 electrode sites where positive pre-saccadic potentials were observed. There was a trend toward an interaction of group and time for anti–pro difference potentials over
357 right frontal polar area (site Fp2) for overlap fixation trials ( ! 210 to 0 ms, F(12,306) 5 1.88, p 5 .06), suggesting that schizophrenia patients, relatives, and controls had different trajectories in pre-saccadic potentials over the front of the brain. Figure 2 depicts changes in pre-saccadic amplitudes at site Fp2, where controls exhibited increased activity for antisaccade trials from about 120 to 90 ms before saccade onset followed by an amplitude reduction (see panel (h) in Figure 2). In contrast, the schizophrenia subjects had continuously increasing potentials for antisaccades that achieved maximum amplitude just before saccade onset (i.e., ! 30 to 0 ms). The relatives showed a pattern of difference potentials similar to that of schizophrenia subjects reaching a maximum just prior to saccade onset. Relatives who had larger Fp2 potentials from 210 to 60 ms before step antisaccades had lower saccade accuracies (r 5 ! .49, p 5 .03), suggesting that frontal polar functions might be relied upon by relatives to inhibit reflexive prosaccades in a failed attempt to compensate for lateral frontal dysfunction. Follow-up analyses revealed that frontal polar potentials preceding eye movements (! 210 to 0 ms) were larger before antisaccades than prosaccades (Fp1: F(1,51) 5 10.28, p 5 .002; Fp2: F(1,51) 5 13.19, p 5 .001), which is consistent with frontal polar activity reflecting inhibition of reflexive prosaccades. Nevertheless, paired-comparisons of difference potentials failed to reveal group differences at frontal polar electrodes. When potentials for the antisaccades and prosaccades were separately considered (see Figure 3), antisaccade trials showed a trend toward a group effect at FP2 from ! 30 to 0 ms for step fixations (F(2,51) 5 2.61, p 5 .08). Post hoc tests for the ! 30 to 0 ms period revealed that relatives had larger amplitudes than controls prior to antisaccades, especially for step fixation trials (t(37) 5 2.72, p 5 .01, d 5 .89), while schizophrenia patients failed to differ from controls (t(33) 5 .93, p 5 .36, d 5 .33) and relatives (t(32) 5 ! .96, p 5 .34, d 5 ! .34). There was also a significant group difference in frontopolar potentials at FP2 from ! 120 to 0 ms for step prosaccades (F(2,51) 5 3.35, p 5 .04). Follow-up analyses revealed that schizophrenia patients had larger FP2 potentials than controls (t(33) 5 2.70, p 5 .01, d 5 .94), while relatives showed a trend toward larger FP2 potentials than controls (t(37) 5 2.02, p 5 .051, d 5 .64), and there was no group difference between patients and relatives (t(37) 5 .32, p 5 .75, d 5 .11). Medications. Associations of chlorpromazine equivalents with behavioral and electrophysiological indices indicated that a higher antipsychotic dosage was related to lower accuracy for overlap antisaccades (r 5 ! .73, p 5 .002), and lower anti–pro difference potentials during ! 210 to ! 180 ms period at frontal electrodes (Af3: r 5 ! .55, p 5 .03; Af4: r 5 ! .58, p 5 .02; F3: r 5 ! .62, p 5 .02; F4: r 5 ! .58, p 5 .02). Compared to schizophrenia patients who were not on antidepressants, patients who were on this class of medication had higher accuracies for step prosaccades (F(1,13) 5 9.52, p 5 .009) and antisaccades (F(1,13) 5 4.78, p 5 .05), greater frontal anti–pro difference potentials during the ! 180 to ! 120 ms period (Af3: F(1,13) 5 7.97, p 5 .01) and smaller frontal anti–pro difference potentials during the ! 90 to ! 60 ms period (Af4: F(1,13) 5 5.56, p 5 .03; F3: F(1,13) 5 4.71, p 5 .05; F4: F(1,13) 5 8.80, p 5 .01). Although medication status of patients was statistically related to some behavioral and electrophysiological indices, relatives who were generally not taking psychiatric medications exhibited electrophysiological abnormalities similar to patients. Therefore, the electrophysiological abnor-
358 malities during saccade generation are unlikely to be the product of medications. Discussion Results provide evidence that in the 200 milliseconds before executing a volitional saccade both individuals with schizophrenia and first-degree biological relatives of schizophrenia patients exhibit neural abnormalities over prefrontal cortex. Because unaffected relatives who are likely carriers of genes for schizophrenia manifested similar electrophysiological activity to patients, the neural abnormalities appear to represent deviant cortical functions associated with genetic liability for the disorder. Specifically, schizophrenia patients and unaffected relatives of schizophrenia patients showed diminished anti–pro saccade difference potentials over lateral frontal cortex from 180 to 120 ms before the onset of saccades. Importantly, smaller pre-saccadic difference potentials over lateral frontal cortex were associated with lower antisaccade accuracies in schizophrenia patients and longer antisaccade latencies in relatives. Diminished potentials were also evident in relatives over parietal cortex prior to antisaccades for overlap fixations, while schizophrenia patients had diminished pre-saccadic potentials over parietal cortex for prosaccades for overlap fixations. Reduction of presaccadic potentials over parietal cortex in relatives was associated with longer latencies on the overlap antisaccade task. Electrophysiological anomalies over parietal cortex may reflect difficulties in attentional disengagement from the fixation stimulus during saccade preparation (Evdokimidis, Smyrnis, Constantinidis, Gourtzelidis, & Papageorgiou, 2001; Richards, 2003). We also observed relatives to have a tendency toward augmented potentials at the right frontal polar site prior to antisaccades, particularly during step fixation trials. Relatives who had larger frontal polar potentials had lower antisaccade accuracy, suggesting that orbital frontal activity may represent ineffective compensatory inhibition of reflexive saccades in response to compromised control of volitional eye movements by lateral frontal cortex (Everling & Fischer, 1998). Positive potentials over lateral prefrontal areas have been observed prior to antisaccades in a study that analyzed principal components of scalp-recorded voltages during short pre-saccadic time periods (Figure 8 in Richards, 2003). In the study, the potential was localized to lateral middle and anterior middle frontal gyrus and interpreted to reflect planning of targeted eye movements. In the present study, source estimations localized scalprecorded voltages prior to antisaccades to lateral PFC (see Supplementary Materials). Given the demand to control and plan voluntary eye movements in the antisaccade task, the primary source of the lateral frontal potentials may be lateral PFC, especially the dorsolateral PFC (Everling & Fischer, 1998). Given additional evidence for dorsolateral PFC activation during antisaccade generation (DeSouza, Menon, & Everling, 2003; Pierrot-Deseilligny, Milea, & Muri, 2004; Pierrot-Deseilligny, Muri, Ploner, Gaymard, Demeret, & Rivaud-Pechoux, 2003), the role of the PFC in providing contextual information through biasing signals to achieve cognitive control (Miller & Cohen, 2001; PierrotDeseilligny, et al., 2003, 2004), and the well-known dorsolateral PFC deficit in schizophrenia (Dracheva, Marras, Elhakem, Kramer, Davis, & Haroutunian, 2001; MacDonald & Carter, 2003; Weinberger, Berman, & Zec, 1986), diminished activity at lateral frontal recording sites in schizophrenia may indeed reflect impairment of dorsolateral PFC during saccade generation. Ne-
S. S. Kang et al. uroimaging in humans has revealed bilateral activation of dorsolateral PFC during antisaccade tasks (Sweeney et al., 1996), and lesions of the dorsolateral PFC result in increases of erroneous reflexive saccades (Fukushima, Fukushima, Miyasaka, & Yamashita, 1994; Pierrot-Deseilligny, Rivaud, Gaymard, & Agid, 1991). Abnormalities of the parietal cortex may also contribute to saccade control deficits in schizophrenia. In the overlap fixation condition where disengagement of spatial attention from the overlapping fixation is required, schizophrenia patients tended to have diminished activity over parietal cortex and lower prosaccade accuracies than control subjects. The results of the present study are consistent with evidence of centro-parietal pre-saccadic positive potentials observed during prosaccades and voluntary saccades (Everling, et al., 1997; Kurtzberg & Vaughan, 1982; Moster & Goldberg, 1990; Richards, 2003). Increases in centroparietal potentials have been related to cellular activity in parietal area 7 that is associated with the deployment of visual attention during saccadic eye movements (Lynch, Mountcastle, Talbot, & Yin, 1977). Reduced pre-saccadic centro-parietal potentials prior to antisaccades as compared to prosaccades that were observed in the present study are consistent with pre-saccadic activations identified in other studies (Everling et al., 1997; Kurtzberg & Vaughan, 1982). The abnormal potentials over parietal cortex may reflect schizophrenia patients’ poor disengagement of spatial attention from the fixation stimulus before execution of the saccade. Failure to disengage from fixation might underlie prosaccade accuracy deficits in patients, which were found in overlap fixations and not step fixations. This is consistent with a previous finding of deficits in attentional disengagement and inhibitory control in schizophrenia patients as indicated by prolonged latency and decreased accuracy on a saccade task with overlap fixations and distracters (Schwartz & Evans, 1999). Through transcranial magnetic stimulation (TMS), investigators have demonstrated that prolonged saccadic latency can result from applying electromagnetic fields to parietal cortex (Kapoula, Isotalo, Muri, Bucci, & RivaudPechoux, 2001; Yang & Kapoula, 2004). Pre-saccadic negative potentials were not observed in the present study although they have been noted in many studies of volitional saccades. Pre-saccadic negativity has been interpreted to reflect preparatory activity of the frontal eye field or supplementary eye field (Everling et al., 1997; Kurtzberg & Vaughan, 1982; Moster & Goldberg, 1990). The discrepancy appears to stem from differences in the task design and analysis of presaccadic potentials. Compared to previous studies that included block-designs with a single fixation condition where subjects responded to stimuli in a fixed manner (Everling et al., 1997; Kurtzberg & Vaughan, 1982; Thickbroom & Mastaglia, 1990), we used a task in which subjects had to respond to pseudo-randomly mixed fixation conditions. The predictable nature of the fixation in the previous studies likely allows the preparatory neural activity to be reliably elicited. Pre-saccadic negative potentials have not been observed by other investigators who have employed unpredictable cues for saccade generation (Evdokimidis, Liakopoulos, Constantinidis, & Papageorgiou, 1996). Additionally, studies that have observed pre-saccadic negative potentials include longer pre-saccadic periods than in the present study (Everling et al., 1997; Kurtzberg & Vaughan, 1982; Moster & Goldberg, 1990; Thickbroom & Mastaglia, 1990), and the pre-saccadic negativity was related to stimulus onsets rather than saccade onsets (Everling, Spantekow, Krappmann, & Flohr, 1998; Richards, 2003). Richards (2003) also noted that the pre-
Mechanisms of antisaccade generation in schizophrenia
359
saccadic negativity is likely the CNV related to preparatory neural activity, which can occur without overt motor movements. Hence, there is limited possibility that the pre-saccadic activity observed in the present study is affected by a CNV given that we employed unpredictable fixation conditions prior to movements and relatively short time period between fixation onset on saccade generation. Given that relatives had essentially normal accuracy on the antisaccade task despite what may be PFC deficits, and their performance improved on overlap fixation for the antisaccade task (see Table 2), their diminished potentials at parietal sites preceding overlap antisaccades may reflect something other than dysfunction. One possibility is that relatives with lateral frontal cortex dysfunction rely on compensatory suppression of parietal activity to inhibit disengagement of spatial attention and prevent reflexive prosaccades during the antisaccade task. Reduction of activity in parietal cortex may inhibit reflexive saccades by persistently engaging spatial attention on the fixation stimulus during overlap trials. Estimated sources (see Supplemental Materials) of potentials observed in the overlap fixation condition were consistent with unaffected first-degree biological relatives having little parietal activity prior to antisaccades but greater activity prior to prosaccades. Finally, relatives exhibited reduced centro-parietal potentials for correct antisaccades compared to incorrect antisaccades, especially in the overlap fixation condition (Cz during ! 90 to ! 60 ms: t(18) 5 ! 3.14, p 5 .006; Pz during ! 90 to ! 60 ms: t(18) 5 ! 3.19, p 5 .005). These findings support the interpretation that the reduced potentials over parietal cortex in relatives reflect inhibition of disengagement from the fixation stimulus (i.e., improved fixation), thus preventing errant reflexive saccades. In other words, the diminished parietal site amplitudes in biological relatives could reflect a compensatory mechanism for executing correct antisaccades during overlap fixation trials. A tendency for relatives and schizophrenia patients to exhibit larger frontal polar potentials may also represent a dynamic response to failure in lateral prefrontal cortex. The frontal polar pre-saccadic potentials may reflect orbital frontal cortex or ventromedial PFC activity given a report of pre-saccadic activity of similar topography that was localized to the orbital frontal gyrus and was larger prior to antisaccades than prosaccades (Richards, 2003). Source analysis suggested that the orbital frontal cortex (Brodman area 11) may be a main contributor to the frontal polar activity prior to antisaccades observed in the current study (see Supplementary Materials). Recruitment of additional prefrontal regions in schizophrenia is also suggested by recent work that revealed schizophrenia patients to activate ventrolateral prefrontal cortex in performing an N-back working memory task while control subjects largely augmented dorsolateral PFC (Tan, Callicott, & Weinberger, 2007; Tan, Sust, Buckholtz, Mattay, MeyerLindenberg, et al., 2006). Neuroimaging studies of cognitive tasks requiring inhibitory control, such as go-no go and antisaccade tasks, suggest that the orbital frontal cortex assists with inhibiting unwanted behavioral responses (Aron et al., 2004; Chikazoe, Konishi, Asari, Jimura, & Miyashita, 2007). As a result of impaired dorsolateral PFC activity prior to antisaccades, relatives may rely more heavily upon orbital PFC than control subjects in
an attempt to inhibit prepotent reflexive saccades. Recent work has provided evidence that a failure of schizophrenia patients to inhibit reflexive saccades (Manoach, Lindgren, Cherkasova, Goff, Halpern, et al., 2002) and not deficient switching between tasks (Greenzang, Manoach, Goff, & Barton, 2007) leads to poor antisaccade performance. We found schizophrenia patients had larger frontal polar potentials than controls from 120 to 0 ms prior to step prosaccade potentials. Because prosaccades do not require inhibition of saccades and the topographies of potentials prior to prosaccades were different from those of antisaccades, future work is required to clarify the source and function of increased frontal polar potentials in schizophrenia patients. Although analyses focused on potentials locked to saccade onset (i.e., response-locked), one cannot rule out the possibility that neural activity reflecting processing of the fixation stimulus may contribute to the pre-saccadic potentials. Inspection of stimulus-locked potentials from 0 to 250 ms after stimulus onset (the time period largely before saccadic eye movements) revealed essentially no notable components at frontal sites (Fp1, Fp2, F3, F4) shortly after onset of the fixation stimulus; however, a broad shallow positive potential at central and parietal sites was observed from 50 to 150 ms after fixation onset. A recent report of a stimulus-locked P2 response recorded in two monkeys described a decrement in the potential at a parietal recording site for fixations prior to antisaccades (Sander, Soper, & Everling, 2010). This is consistent with the possibility that pre-saccadic potentials over parietal cortex reflect spatial attention modulation in preparation for saccade generation. Additionally, the potentials prior to visually triggered saccades in the present study are similar to a pre-saccadic positivity over the parietal region previously noted as diminished prior to antisaccades (Everling et al., 1997). Finally, a spike potential noted at or near saccade onset by other studies (Everling et al., 1997; Richards, 2003) was evident with the expected frontal-negative-positive-elsewhere distribution in data from the current investigation. In summary, electrophysiological abnormalities over lateral frontal cortex prior to saccades in an antisaccade task provide evidence suggestive of prefrontal cortical dysfunction preceding saccadic errors in schizophrenia patients and biological relatives of schizophrenia patients. Because anatomical locations of neural activity are not definitively identified and instead are estimated from scalp recordings, work remains to precisely isolate the location of pre-saccadic abnormalities associated with schizophrenia. Nevertheless, the present study provides evidence consistent with lateral frontal cortical dysfunction during volitional saccade generation being a marker of genetic liability for schizophrenia. Relatives also exhibited reduced potentials compared to control subjects over parietal cortex prior to saccade execution. Despite these electrophysiological abnormalities, the relatives successfully performed the antisaccade task. Reduced activity over parietal cortex may reflect suppression of attentional disengagement, leading to enhanced inhibition of reflexive saccades and normative performance in relatives. To conclude, the present study suggests that PFC and parietal cortex are elements of a neural substrate associated with poor saccadic control in schizophrenia and reflective of genetic liability for the disorder.
REFERENCES Aron, A. R., Robbins, T. W., & Poldrack, R. A. (2004). Inhibition and the right inferior frontal cortex. Trends in Cognitive Science, 8, 170– 177.
Braun, D., & Breitmeyer, B. G. (1988). Relationship between directed visual attention and saccadic reaction times. Experiments in Brain Research, 73, 546–552.
360 Brooker, B., & Cyr, J. (1986). Tables for clinicians to use to convert WAIS-R short forms. Journal of Clinical Psychology, 42, 983–986. Calkins, M. E., Curtis, C. E., Iacono, W. G., & Grove, W. M. (2004). Antisaccade performance is impaired in medically and psychiatrically healthy biological relatives of schizophrenia patients. Schizophrenia Research, 71, 167–178. Calkins, M. E., Dobie, D. J., Cadenhead, K. S., Olincy, A., Freedman, R., Green, M. F., et al. (2007). The Consortium on the Genetics of Endophenotypes in Schizophrenia: Model recruitment, assessment, and endophenotyping methods for a multisite collaboration. Schizophrenia Bulletin, 33, 33–48. Camchong, J., Dyckman, K. A., Austin, B. P., Clementz, B. A., & McDowell, J. E. (2008). Common neural circuitry supporting volitional saccades and its disruption in schizophrenia patients and relatives. Biological Psychiatry, 64, 1042–1050. Carter, C. S., Perlstein, W., Ganguli, R., Brar, J., Mintun, M., & Cohen, J. D. (1998). Functional hypofrontality and working memory dysfunction in schizophrenia. American Journal of Psychiatry, 155, 1285–1287. Chikazoe, J., Konishi, S., Asari, T., Jimura, K., & Miyashita, Y. (2007). Activation of right inferior frontal gyrus during response inhibition across response modalities. Journal of Cognitive Neuroscience, 19, 69–80. Clementz, B. A., McDowell, J. E., & Zisook, S. (1994). Saccadic system functioning among schizophrenia patients and their first-degree biological relatives. Journal of Abnormal Psychology, 103, 277–287. DeSouza, J. F., Menon, R. S., & Everling, S. (2003). Preparatory set associated with pro-saccades and anti-saccades in humans investigated with event-related FMRI. Journal of Neurophysiology, 89, 1016–1023. Dracheva, S., Marras, S. A., Elhakem, S. L., Kramer, F. R., Davis, K. L., & Haroutunian, V. (2001). N-methyl-D-aspartic acid receptor expression in the dorsolateral prefrontal cortex of elderly patients with schizophrenia. American Journal of Psychiatry, 158, 1400–1410. Ekselius, L., Lindstrom, E., von Knorring, L., Bodlund, O., & Kullgren, G. (1994). SCID II interviews and the SCID screen questionnaire as diagnostic tools for personality disorders in DSM-III-R. Acta Psychiatrica Scandinavica, 90, 120–123. Ettinger, U., Ffytche, D. H., Kumari, V., Kathmann, N., Reuter, B., Zelaya, F., et al. (2007). Decomposing the neural correlates of antisaccade eye movements using event-related fMRI. Cerebral Cortex, 18, 1148–1159. Evdokimidis, I., Liakopoulos, D., Constantinidis, T. S., & Papageorgiou, C. (1996). Cortical potentials with antisaccades. Electroencephalography and Clinical Neurophysiology, 98, 377–384. Evdokimidis, I., Smyrnis, N., Constantinidis, T. S., Gourtzelidis, P., & Papageorgiou, C. (2001). Frontal-parietal activation differences observed before the execution of remembered saccades: An event-related potentials study. Brain Research, 12, 89–99. Everling, S., & Fischer, B. (1998). The antisaccade: A review of basic research and clinical studies. Neuropsychologia, 36, 885–899. Everling, S., Krappmann, P., & Flohr, H. (1997). Cortical potentials preceding pro- and antisaccades in man. Electroencephalography and Clinical Neurophysiology, 102, 356–362. Everling, S., Spantekow, A., Krappmann, P., & Flohr, H. (1998). Event-related potentials associated with correct and incorrect responses in a cued antisaccade task. Experiments in Brain Research, 118, 27–34. First, M. B., Spitzer, R. L., Gibbon, M., & Williams, J. B. W. (1996). Structured Clinical Interview for DSM-IV Axis I Disorder (SCID-I, Research Version). New York: Biometric Research Department, New York State Psychiatric Institute. Fischer, B., & Weber, H. (1993). Express saccades and visual attention. Behavioral and Brain Sciences, 16, 553–610. Fischer, B., & Weber, H. (1997). Effects of stimulus conditions on the performance of antisaccades in man. Experiments in Brain Research, 116, 191–200. Franke, C., Reuter, B., Schulz, L., & Kathmann, N. (2007). Schizophrenia patients show impaired response switching in saccade tasks. Biological Psychology, 76, 91–99. Fukumoto-Motoshita, M., Matsuura, M., Ohkubo, T., Ohkubo, H., Kanaka, N., Matsushima, E., et al. (2009). Hyperfrontality in patients with schizophrenia during saccade and antisaccade tasks: A study with fMRI. Psychiatry and Clinical Neuroscience, 63, 209– 217. Fukushima, J., Fukushima, K., Miyasaka, K., & Yamashita, I. (1994). Voluntary control of saccadic eye movement in patients with frontal
S. S. Kang et al. cortical lesions and parkinsonian patients in comparison with that in schizophrenics. Biological Psychiatry, 36, 21–30. Gottesman, II & Gould, T. D. (2003). The endophenotype concept in psychiatry: Etymology and strategic intentions. American Journal of Psychiatry, 160, 636–645. Greenzang, C., Manoach, D. S., Goff, D. C., & Barton, J. J. (2007). Task-switching in schizophrenia: Active switching costs and passive carry-over effects in an antisaccade paradigm. Experiments in Brain Research, 181, 493–502. Guitton, D., Buchtel, H. A., & Douglas, R. M. (1985). Frontal lobe lesions in man cause difficulties in suppressing reflexive glances and in generating goal-directed saccades. Experiments in Brain Research, 58, 455–472. Hutton, S. B., & Ettinger, U. (2006). The antisaccade task as a research tool in psychopathology: A critical review. Psychophysiology, 43, 302–313. Kapoula, Z., Isotalo, E., Muri, R. M., Bucci, M. P., & Rivaud-Pechoux, S. (2001). Effects of transcranial magnetic stimulation of the posterior parietal cortex on saccades and vergence. NeuroReport, 12, 4041– 4046. Kendler, K. S., Lieberman, J. A., & Walsh, D. (1989). The structured interview for schizotypy (SIS): A preliminary report. Schizophrenia Bulletin, 15, 559–571. Kim, J. J., Kwon, J. S., Park, H. J., Youn, T., Kang, D. H., Kim, M. S., et al. (2003). Functional disconnection between the prefrontal and parietal cortices during working memory processing in schizophrenia: A [15(O)]H2O PET study. American Journal of Psychiatry, 160, 919–923. Klein, C., Heinks, T., Andresen, B., Berg, P., & Moritz, S. (2000). Impaired modulation of the saccadic contingent negative variation preceding antisaccades in schizophrenia. Biological Psychiatry, 47, 978–990. Kurtzberg, D., & Vaughan, H. G. Jr. (1982). Topographic analysis of human cortical potentials preceding self-initiated and visually triggered saccades. Brain Research, 243, 1–9. Levy, D. L., O’Driscoll, G., Matthysse, S., Cook, S. R., Holzman, P. S., & Mendell, N. R. (2004). Antisaccade performance in biological relatives of schizophrenia patients: A meta-analysis. Schizophrenia Research, 71, 113–125. Lynch, J. C., Mountcastle, V. B., Talbot, W. H., & Yin, T. C. (1977). Parietal lobe mechanisms for directed visual attention. Journal of Neurophysiology, 40, 362–389. MacDonald, A. W. III, & Carter, C. S. (2003). Event-related FMRI study of context processing in dorsolateral prefrontal cortex of patients with schizophrenia. Journal of Abnormal Psychology, 112, 689–697. Manoach, D. S., Ketwaroo, G. A., Polli, F. E., Thakkar, K. N., Barton, J. J., Goff, D. C., et al. (2007). Reduced microstructural integrity of the white matter underlying anterior cingulate cortex is associated with increased saccadic latency in schizophrenia. NeuroImage, 37, 599–610. Manoach, D. S., Lindgren, K. A., Cherkasova, M. V., Goff, D. C., Halpern, E. F., Intriligator, J., et al. (2002). Schizophrenic subjects show deficient inhibition but intact task switching on saccadic tasks. Biological Psychiatry, 51, 816–826. Maruff, P., Danckert, J., Pantelis, C., & Currie, J. (1998). Saccadic and attentional abnormalities in patients with schizophrenia. Psychological Medicine, 28, 1091–1100. McDowell, J. E., Brown, G. G., Paulus, M., Martinez, A., Stewart, S. E., Dubowitz, D. J., et al. (2002). Neural correlates of refixation saccades and antisaccades in normal and schizophrenia subjects. Biological Psychiatry, 51, 216–223. McDowell, J. E., & Clementz, B. A. (1997). The effect of fixation condition manipulations on antisaccade performance in schizophrenia: Studies of diagnostic specificity. Experiments in Brain Research, 115, 333–344. McGuffin, P., Farmer, A., & Harvey, I. (1991). A polydiagnostic application of operational criteria in studies of psychotic illness. Development and reliability of the OPCRIT system. Archives of General Psychiatry, 48, 764–770. Miller, E. K., & Cohen, J. D. (2001). An integrative theory of prefrontal cortex function. Annual Review of Neuroscence, 24, 167–202. Moon, S. Y., Barton, J. J., Mikulski, S., Polli, F. E., Cain, M. S., Vangel, M., et al. (2007). Where left becomes right: A magnetoencephalographic study of sensorimotor transformation for antisaccades. NeuroImage, 36, 1313–1323. Moster, M. L., & Goldberg, G. (1990). Topography of scalp potentials preceding self-initiated saccades. Neurology, 40, 644–648.
Mechanisms of antisaccade generation in schizophrenia Nurnberger, J. I. Jr., Blehar, M. C., Kaufmann, C. A., York-Cooler, C., Simpson, S. G., Harkavy-Friedman, J., et al. (1994). Diagnostic interview for genetic studies. Rationale, unique features, and training. NIMH Genetics Initiative. Archives of General Psychiatry, 51, 849–859; discussion 863–844. Ojeda, N., Ortuno, F., Arbizu, J., Lopez, P., Marti-Climent, J. M., Penuelas, I., et al. (2002). Functional neuroanatomy of sustained attention in schizophrenia: Contribution of parietal cortices. Human Brain Mapping, 17, 116–130. Pierrot-Deseilligny, C., Milea, D., & Muri, R. M. (2004). Eye movement control by the cerebral cortex. Current Opinions in Neurology, 17, 17–25. Pierrot-Deseilligny, C., Muri, R. M., Ploner, C. J., Gaymard, B., Demeret, S., & Rivaud-Pechoux, S. (2003). Decisional role of the dorsolateral prefrontal cortex in ocular motor behaviour. Brain, 126, 1460–1473. Pierrot-Deseilligny, C., Rivaud, S., Gaymard, B., & Agid, Y. (1991). Cortical control of reflexive visually-guided saccades. Brain, 114, 1473–1485. Pierrot-Deseilligny, C., Rivaud, S., Gaymard, B., Muri, R., & Vermersch, A. I. (1995). Cortical control of saccades. Annals of Neurology, 37, 557–567. Posner, M. I. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 3–25. Posner, M. I., Walker, J. A., Friedrich, F. J., & Rafal, R. D. (1984). Effects of parietal injury on covert orienting of attention. Journal of Neuroscience, 4, 1863–1874. Raemaekers, M., Ramsey, N. F., Vink, M., van den Heuvel, M. P., & Kahn, R. S. (2006). Brain activation during antisaccades in unaffected relatives of schizophrenic patients. Biological Psychiatry, 59, 530–535. Rafal, R. D. (2006). Oculomotor functions of the parietal lobe: Effects of chronic lesions in humans. Cortex, 42, 730–739. Raine, A. (1991). The SPQ: A scale for the assessment of schizotypal personality based on DSM-III-R criteria. Schizophrenia Bulletin, 17, 555–564. Reuter, B., Herzog, E., Endrass, T., & Kathmann, N. (2006). Brain potentials indicate poor preparation for action in schizophrenia. Psychophysiology, 43, 604–611. Reuter, B., Jager, M., Bottlender, R., & Kathmann, N. (2007). Impaired action control in schizophrenia: The role of volitional saccade initiation. Neuropsychologia, 45, 1840–1848. Richards, J. E. (2003). Cortical sources of event-related potentials in the prosaccade and antisaccade task. Psychophysiology, 40, 878–894. Rivaud, S., Muri, R. M., Gaymard, B., Vermersch, A. I., & PierrotDeseilligny, C. (1994). Eye movement disorders after frontal eye field lesions in humans. Experiments in Brain Research, 102, 110–120. Sander, V., Soper, B., & Everling, S. (2010). Nonhuman primate eventrelated potentials associated with pro- and anti-saccades. NeuroImage, 49, 1650–1658. Schulze, K., MacCabe, J. H., Rabe-Hesketh, S., Crawford, T., Marshall, N., Zanelli, J., et al. (2006). The relationship between eye movement and brain structural abnormalities in patients with schizophrenia and their unaffected relatives. Journal of Psychiatric Research, 40, 589–598. Schwartz, B. D., & Evans, W. J. (1999). Neurophysiologic mechanisms of attention deficits in schizophrenia. Neuropsychiatry and Neuropsychological Behavioral Neurology, 12, 207–220.
361 Servan-Schreiber, D., Cohen, J. D., & Steingard, S. (1996). Schizophrenic deficits in the processing of context. A test of a theoretical model. Archives of General Psychiatry, 53, 1105–1112. Sprague, R. L., Kalachnik, J. E., & Slaw, K. M. (1989). Psychometric properties of the Dyskinesia Identification System: Condensed User Scale (DISCUS). Mental Retardation, 27, 141–148. Sweeney, J. A., Mintun, M. A., Kwee, S., Wiseman, M. B., Brown, D. L., Rosenberg, D. R., et al. (1996). Positron emission tomography study of voluntary saccadic eye movements and spatial working memory. Journal of Neurophysiology, 75, 454–468. Tan, H. Y., Callicott, J. H., & Weinberger, D. R. (2007). Dysfunctional and compensatory prefrontal cortical systems, genes and the pathogenesis of schizophrenia. Cerebral Cortex, 17(Supp), 171–181. Tan, H. Y., Sust, S., Buckholtz, J. W., Mattay, V. S., Meyer-Lindenberg, A., Egan, M. F., et al. (2006). Dysfunctional prefrontal regional specialization and compensation in schizophrenia. American Journal of Psychiatry, 163, 1969–1977. Thickbroom, G. W., & Mastaglia, F. L. (1990). Premotor negativity associated with saccadic eye movement and finger movement: A comparative study. Brain Research, 506, 223–226. Tu, P. C., Yang, T. H., Kuo, W. J., Hsieh, J. C., & Su, T. P. (2006). Neural correlates of antisaccade deficits in schizophrenia, an fMRI study. Journal of Psychiatric Research, 40, 606–612. Ventura, J., Nuechterlein, K. H., Subotnik, K. L., Gutkind, D., & Gilbert, E. A. (2000). Symptom dimensions in recent-onset schizophrenia and mania: A principal components analysis of the 24-item Brief Psychiatric Rating Scale. Psychiatry Research, 97, 129–135. Weinberger, D. R., Berman, K. F., & Zec, R. F. (1986). Physiologic dysfunction of dorsolateral prefrontal cortex in schizophrenia. I. Regional cerebral blood flow evidence. Archives of General Psychiatry, 43, 114–124. Yang, Q., & Kapoula, Z. (2004). TMS over the left posterior parietal cortex prolongs latency of contralateral saccades and convergence. Investigative Ophthalmology and Visual Science, 45, 2231–2239. Zhang, M., & Barash, S. (2000). Neuronal switching of sensorimotor transformations for antisaccades. Nature, 408, 971–975.
SUPPORTING INFORMATION Additional Supporting Information may be found in the online version of this article: Figure S1: Grand Average Event Related Potentials (ERPs) of the Three Groups. Figure S2: Source Analysis Results of the ERPs. Appendix S1: Supplementary References. Please note: Wiley-Blackwell are not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article. (Received July 13, 2009; Accepted April 30, 2010)
Psychophysiology, 48 (2011), 362–369. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01063.x
EEG power changes reflect distinct mechanisms during long-term memory retrieval
PATRICK H. KHADER and FRANK RO¨SLER Institute of Psychology, Experimental and Biological Psychology, Philipps-University, Marburg, Germany
Abstract The roles of theta and alpha oscillations for long-term memory (LTM) retrieval are still under debate. Both are modulated by LTM retrieval demands, but it is unclear what specific LTM functions they are related to. Here, different oscillatory correlates of LTM retrieval could be obtained for theta and alpha with a paradigm that is suited to monitor the activation of a varying number of material-specific LTM representations. Both frequency bands responded parametrically to the number of retrieved items. However, only the alpha effect dissociated topographically for material type, indicating that the activation of material-specific representations became systematically modulated. For theta, this effect was material-unspecific with mid-frontal topography. These results suggest that alpha is functionally related to the activation of stored information, whereas theta is a sign of retrieval-related control processes. Descriptors: Alpha, Theta, Long-term memory, EEG
participants had to judge whether sequentially presented featureconcept pairs (such as ‘‘claws-eagle’’) are semantically congruent (Klimesch, Doppelmayr, Pachinger, & Ripper, 1997a; Klimesch, Doppelmayr, Pachinger, & Russegger, 1997b). These results, together with neurophysiological considerations that alpha power decreases are possibly generated by circumscribed cortical cell assemblies triggered by thalamo-cortical feedback loops (Nunez & Srinivasan, 2006; Steriade, 1999), led Klimesch (1996, 1999) to suggest that activity within the upper alpha band correlates with search and retrieval processes of semantic information stored in cortical associative networks. Together, the reviewed results suggest a functional dissociation for theta and upper alpha oscillations with respect to memory processes. Theta activity seems to be related to episodic LTM encoding and the maintenance of information in WM, while activity in the upper alpha seems to be related to semantic LTM retrieval. However, this interpretation is at variance with findings showing that theta activity can also be affected by LTM manipulations (Klimesch, Doppelmayr, Stadler, Pollhuber, Sauseng, & Ro¨hm, 2001a; Klimesch, Doppelmayr, Yonelinas, Kroll, Lazzara, et al., 2001b) and upper alpha activity by WM demands. For example, theta power has been found to be related to the strength of an episodic memory trace (Klimesch, Hanslmayr, Sauseng, Gruber, Brozinsky, et al., 2006) and to the difficulty of conceptual operations during memory search (Mecklinger, Kramer, & Strayer, 1992). Alpha power, like theta power, has been found to be sensitive to episodic encoding (Hanslmayr, Spitzer, & Ba¨uml, 2009; Khader, et al., 2010; Sederberg, Kahana, Howard, Donner, & Madsen, 2003; Weiss & Rappelsberger, 2000) and to increase with WM load (Jensen, Gelfand, Kounios, & Lisman, 2002). Taken together, it is unclear whether theta and upper alpha activity do actually reflect functionally distinct
Recent research suggests that event-related changes of electroencephalogram (EEG) oscillations, especially in the theta (5–7 Hz) and the upper alpha band (10–12 Hz), are functionally related to memory processes (for reviews, see Kahana, Seelig, & Madsen, 2001; Klimesch, 1996, 1999). Systematic changes of theta activity have most often been observed during working memory (WM) and long-term memory (LTM) encoding tasks. In particular, correlations were found between theta power and working memory load (Gevins, Smith, McEvoy, & Yu, 1997; Jensen & Tesche, 2002; McEvoy, Pellouchoud, Smith, & Gevins, 2001; Onton, Delorme, & Makeig, 2005), and between the length of theta episodes and the duration of WM demands (Raghavachari, Kahana, Rizzuto, Caplan, Kirschen, et al., 2001; Tesche & Karhu, 2000). Furthermore, theta activity recorded during encoding has been found to be more pronounced for words later remembered than for words later forgotten (Khader, Jost, Ranganath, & Ro¨sler, 2010; Klimesch, Doppelmayr, Russegger, & Pachinger, 1996; Klimesch, Doppelmayr, Schimke, & Ripper, 1997c), suggesting that high theta power indicates successful memory encoding. In contrast, activity within the upper alpha band has been functionally linked to retrieval from semantic LTM (for reviews, see Klimesch, 1996, 1999). For example, in two studies Klimesch and colleagues found that upper alpha power decreased when
This work was supported by Grant FOR254/2 and Grant KH235/1-1 of the German Research Foundation (DFG). We thank Kerstin Jost for valuable discussions and Stefan Debener, as well as two anonymous reviewers, for helpful comments on the manuscript. Address correspondence to: Dr. Patrick H. Khader, Department of Psychology, Philipps-University, 35032 Marburg, Germany. E-mail:
[email protected] 362
EEG power changes during LTM retrieval memory processes, as the available database does not provide a clear-cut picture about which memory function becomes manifest in which frequency band. The present study wants to disentangle the possible functions of theta and upper alpha activity during LTM retrieval by employing a retrieval paradigm that allows to systematically manipulate retrieval effort and the type of memory representation that has to be retrieved (Heil, Ro¨sler, & Hennighausen, 1994; Khader, Burke, Bien, Ranganath, & Ro¨sler, 2005; Khader, Heil, & Ro¨sler, 2005; Khader, Knoth, Burke, Bien, Ranganath, & Ro¨sler, 2007; Ro¨sler, Heil, & Hennighausen, 1995). The rationale of using these two manipulations rests on the assumption that brain measures showing material-specific activation patterns during memory retrieval do most likely reflect processes that are specific to a reactivation of stored engrams, while brain measures with material-unspecific but load-sensitive activation patterns do most likely indicate superordinate control mechanisms. In more detail: According to the ‘‘recapitulation’’ account of LTM representations (McClelland, McNaughton, & O’Reilly, 1995; O’Reilly & Rudy, 2001; Squire & Alvarez, 1995; for a recent review, see Khader & Ro¨sler, 2009), engrams are established within the neocortex due to permanent synaptic changes in distributed neural networks. These synaptic changes take place in the very same networks (anatomical areas) that are activated during encoding and other perceptual or motor processing episodes of the material, i.e., the very same areas that are in charge for on-line processing activities also contain the engram. This implies that different types of memories should be stored and reactivated in anatomically distinct areas. Reactivation of an engram, however, is not sufficient for successful memory retrieval. In addition, the retrieval process has to be controlled, i.e., among others, memory search has to be initiated, search results have to be evaluated, and possibly competing associations have to be blocked. Recent research has shown that structures of the prefrontal cortex are essential for such control processes that regulate the activity levels of the material-specific networks containing the engrams (e.g., Badre & Wagner, 2007). Due to their integrating function, these ‘‘control’’ areas are material-unspecific in that projections from different material-specific networks must project onto the same ‘‘control’’ structure, if competing activation levels have to be ‘‘negotiated’’ (see, e.g., O’Reilly, Noelle, Braver, & Cohen, 2002; Owen & Hampshire, 2009). Accordingly, we wanted to investigate whether theta and alpha power changes show load and material-specific effects that are either topographically distinct or not. To this end, participants learned associations between cues (abstract nouns) and either pictures of objects (drinking cups) or spatial locations (locations in a 5 ! 5 grid) (see Figure 1). During retrieval, two nouns were presented and participants had to recall the objects (or spatial positions) associated with these nouns in order to decide whether the nouns were both associated with the same object (or spatial position) or not (Figure 1C). At the heart of this design lies a parametric variation of the number of items associated with each cue. One cue (noun) was associated with either one or two objects (or positions) (Figures 1A, B). Accordingly, during retrieval participants had either to check two associations (if both words were associated with one item each, such as ‘‘STRUKTUR’’ [structure] and ‘‘KONZEPT’’ [concept] in Figure 1A), three associations (if one word was associated with one and the other one with two items, such as ‘‘STRUKTUR’’ and ‘‘AUFTAKT’’ [beginning]), or four (if both words were associated with two items). The number of associations to be
363 checked defines the level of ‘‘associative fan’’ (Anderson, 1974), which is systematically related to retrieval load, i.e., with increasing fan both response time and error rate increase. By means of this paradigm, both the quality and the quantity of activated associations are varied systematically without changing any other aspect of the retrieval situation. Most importantly, spatial and object associations are triggered by the same type of perceptual cue, i.e., an abstract noun (Figure 1C). Therefore, material-specific cortical activations can be genuinely related to the reactivation of stored representations rather than to processing different perceptual cues. Moreover, the amount of LTM scanning or retrieval effort is a priori defined by the controlled parametric manipulation of associative fan. Given that the retrieval of LTM representations triggers reverberating activity in material-specific cortical cell assemblies (McClelland et al., 1995; O’Reilly & Rudy, 2001), the fan manipulation should lead to a gradual activation of these assemblies, and these effects should become manifest in changes of specific frequency bands of the EEG signal. Power changes showing anatomically distinct, material-specific fan-related modulations are supposed to indicate the activation of LTM representations as such, while power changes showing materialunspecific modulations with fan level are supposed to mediate material-unspecific control processes during LTM retrieval.
Methods The present paper presents a frequency-domain analysis of data that were originally collected for a time-domain analysis of slow event-related potentials during memory retrieval (Khader et al., 2007). Participants The sample comprised 15 right-handed students of the University of Marburg (4 males and 11 females; mean age 23.8 years, range 20–28). They gave written consent to participate and received either monetary compensation or course credit. Material On the first day, associations between words and positions and words and objects were established and trained. Fifty-four abstract nouns were linked to 9 objects and 9 spatial positions. Half of the words (27 words) were associated with either one (18 words) or two (9 words) of the 9 spatial positions (see Figure 1A for examples), and the other half with either one or two of the 9 objects, respectively. The object stimuli were grayscale photographs of 9 cups (see Figure 1B for examples). The spatial positions were defined as fields in a 5 ! 5 matrix of dark gray squares, in which the position(s) relevant in a trial were highlighted in light gray (see Figure 1B for examples). On the following day, participants had to retrieve the learned stimuli from LTM while EEG was recorded. Here, word pairs were presented as retrieval cues and participants had to decide whether they were linked via a common object/position or not (see Figure 1C). Depending on the number of associations, two, three, or four activated items had to be compared in a given trial, i.e., either two (if both words are associated with one item each, such as ‘‘STRUKTUR’’ and ‘‘KONZEPT’’ in Figure 1A), three (if one word is associated with one and the other one with two items, such as ‘‘STRUKTUR’’ and ‘‘AUFTAKT’’ in Figure 1A), or
364
P. H. Khader & F. Ro¨sler and had to decide whether the words were linked via a common object/position or not. The words remained visible until the participant pressed a response button. Eight seconds after the presentation of the word pair the fixation cross disappeared, signaling the end of the retrieval period. Before the start of the next trial, a blank screen was shown for 2 s. Participants had to press either the left or right ‘‘Ctrl’’ key of a computer keyboard to respond ‘‘yes’’ or ‘‘no.’’ After every block of six trials, a message on the screen indicated the number of errors in that block. After every 36 trials, a pause of 10 s was enforced to enable a DC reset of the EEG.
Figure 1. (A) Three examples of to-be-learned associations between words and spatial positions, illustrating that one word could be associated to either one or two items. (B) Examples of objects and positions that had to be encoded. (C) Trial timing of the retrieval phase. During retrieval, probe stimuli comprising two nouns were presented. Participants had to decide whether the nouns were linked to each other via a commonly associated object or spatial position (the answer would be ‘‘yes’’ for the given example). Depending on the word pair’s level of associative fan, a parametrically varying number of items had to be retrieved from LTM.
four (if both words are associated with two items), resulting in three levels of associative fan (Fan1, Fan2, Fan3). By means of this paradigm, the quality and quantity of activated stimuli could be varied without changing any other aspect of the retrieval situation. Test probes were selected such that the factors of material type and level of fan were completely crossed, with 36 retrieval trials for each of the resulting 6 conditions (432 trials in total) (for a detailed description of Materials and Methods, see Khader et al., 2007). Procedure In each trial of the learning task, a noun was shown, followed after 6.5 s by one or two position(s) or object(s). All stimuli remained on the screen until the participant started the next trial by pressing the space key of a computer keyboard. Participants were instructed to encode the associations by forming an integrated picture of the display. All 54 trials were presented three times in randomized order. When the words were repeated, participants were told to anticipate the associated object(s) or position(s) during the interval between word onset and the presentation of the associated item(s). Immediately following the acquisition phase, participants had to work through a training phase, in which, on each trial, the 5 ! 5 matrix of squares or, alternatively, a 3 ! 3 grid depicting the 9 objects was presented, together with one out of the 54 words. The associated spatial position(s) or object(s) had to be selected by pressing one or two out of 25 black buttons arranged in a square. In case of an error, the correct position(s) or object(s) subsequently appeared on the screen. On average, participants had to complete 7–8 sets before they met the learning criterion, i.e., less than three errors in one complete set of 54 trials (93% correct). Trials of the retrieval phase (see Figure 1C) started with a fixation cross, which was shown for 2 s, followed by one word above and another one below, and participants had to recall the respective objects or spatial positions associated with the words
EEG Recording and Analysis The EEG was recorded in direct-current (DC) mode during the retrieval phase from 61 Ag/AgCl electrodes (Easycap, Falk Minow, Munich, Germany) referenced to one earlobe and rereferenced offline to averaged earlobes. The horizontal and vertical electrooculogram (EOG) was monitored with appropriate electrode pairs. The left or right mastoid (counterbalanced across participants) served as ground. Impedances were kept below 5 kO. Recording and digitization (sampling rate 5 500 Hz) was done with two 32-channel amplifiers (SYNAMPS, NeuroScan, Compumedics GmbH, Singen, Germany) and NeuroScan software Acquire. EEG analysis was done with the BrainVision Analyzer2 software (www.brainproducts.com). After correction of the DC drift (Hennighausen et al., 1993), independent component analysis (ICA) was used to detect and eliminate vertical and horizontal ocular artifacts. This was done with the ‘‘Ocular Correction ICA’’ procedure implemented in BrainVision Analyzer2. Here, recommended standard values were used (number of ICA steps: 512; ICA algorithm: Infomax Restricted). The blink and horizontal eye activity components were found by means of the relative vertical EOG variance, or horizontal EOG variance, respectively. An interval of 200 s was chosen for the ICA by visual inspection that contained a smooth raw EEG interspersed with pronounced blinks and horizontal eye movements. All channels were enabled for the ICA correction. The artifact-corrected raw data were bandpass-filtered between 10 and 12 Hz (upper alpha) and between 5 and 7 Hz (theta) and segmented into trials ranging from 500 ms preceding the onset of the words in the retrieval phase to 8000 ms poststimulus. Trials with residual artifacts were removed by applying the following criteria: (1) the maximal allowed absolute difference of two values in the segment was 200 mV, (2) the minimal absolute difference of two values in any 200 ms interval was 0.5 mV (‘‘dead-channel criterion’’), and (3) the maximal allowed voltage step between two adjacent sampling points was 20 mV. On average, about 50 trials in each experimental condition remained for subsequent analysis, i.e., fan1: 59 (41–71), fan2: 55(33–71), fan3: 48(26–66) for recalling positions, and fan1: 57(39–71), fan2: 49(26–63), fan3: 45(26–68) for recalling objects. After amplitude rectification, amplitudes were transformed into percentsignal-change values with respect to a prestimulus baseline from 500 to 0 ms preceding stimulus onset, yielding event-related power increases or decreases, i.e., ‘‘event-related (de)synchronization’’ (ERD/ERS; Pfurtscheller & Lopes da Silva, 1999). We used the classical formula, i.e., ERD/ERS 5 ((Reference Power– Test Power)/Reference Power) ! 100, in which ERD is associated with decreased power relative to the baseline, resulting in negative values, and ERS being associated with increased power relative to the baseline, resulting in positive values. These values were averaged across trials separately for participants,
EEG power changes during LTM retrieval electrodes, and experimental conditions. Only trials with correct responses were used for the average. Focusing on theta and upper alpha in the present study is based on the literature, showing that memory-related effects are most often found in these frequency bands. However, in order to determine whether the found effects are specific to these bands, the neighboring frequencies should also be analyzed. Looking for effects in the delta range (1–4 Hz) is always problematic with DC-recorded EEG data, as slow drifts that are preserved in the EEG signal are hard to disentangle from ‘‘real’’ oscillations. A bandpass filter for delta captures parts of these slow drifts, producing a distorted signal with very low signal-to-noise ratio. Furthermore, obtaining effects in the gamma range (425 Hz) with scalp-recorded EEG is generally a difficult enterprise, as signal-to-noise ratios are usually very low. Therefore, we additionally analyzed the lower alpha (8–10 Hz) and the beta (13–18 Hz) frequency ranges in order to cover all frequencies between 4 and 18 Hz. Average ERD/ERS amplitudes were computed for consecutive intervals of 500 ms from 0 to 8 s after stimulus onset. A subset of electrodes equivalent to the 19 standard electrodes of the 10–20 system entered the statistical analysis (Fp1, Fp2, F3, F4, F7, F8, Fz, C3, C4, Cz, T3, T4, T5, T6, P3, P4, Pz, O1, and O2). Effects of interest were tested with repeated-measures analyses of variance (ANOVAs) with factors electrodes, material type (objects or spatial positions), and level of fan (Fan1, Fan2, Fan3) for each time epoch and frequency band. Error probability was corrected according to Huynh and Feldt (1976) to account for nonsphericity.
Results Behavioral Data The level of associative fan had a strong effect on both error rates and response times. Both measures increased with increasing fan level (error rates: fan1 5 4.9%, fan2 5 7.6%, fan3 5 18.2% for recalling positions, and fan1 5 6.3%, fan2 5 18.8%, fan3 5 23.0% for recalling objects; response times: fan1 5 3314 ms, fan2 5 4383 ms, fan3 5 5603 ms for recalling positions, and fan1 5 3643 ms, fan2 5 5182 ms, fan3 5 5964 ms for recalling objects). The number of correct responses varied between 77% and 96% over participants (mean 5 87%, SD 5 6.3%), and an ANOVA revealed main effects of fan level, F(2,28) 5 23.53, po.0001, e(H-F) 5 .742, and stimulus type, F(1,14) 5 10.85, p 5 .0053, as well as a significant interaction, F(2,28) 5 5.67, p 5 .0129, e(H-F) 5 .843. All in all, one-way stimulus-specific ANOVAs with factor fan proved that both objects and positions showed a significant fan effect, albeit the difference between the lowest and the intermediate fan level was significant only for objects, whereas the intermediate and highest levels differed significantly only for positions. Furthermore, at the intermediate fan level, recalling objects produced significantly more errors. The response time (RT)-ANOVA revealed again main effects of fan level, F(2,28) 5 120.62, po.0001, e(H-F) 5 .870, and stimulus type, F(1,14) 5 19.71, p 5 .0006, as well as a significant interaction, F(2,28) 5 3.96, p 5 .0373, e(H-F) 5 .871. T-tests revealed that RTs increased significantly with increasing fan for both types of stimuli, and that recalling objects took longer at all fan levels. A detailed account of the behavioral data is provided in Khader et al. (2007).
365 EEG Power Changes As can be seen in Figures 2, 3, and 4, theta, alpha, and beta activity responded systematically to the number of retrieved items. However, it holds only for alpha and beta power, but not for theta power, that this effect dissociated topographically with respect to material type. The distinct topographies indicate that material-specific cortical cell assemblies became systematically modulated the more associations had to be accessed. For theta power, this load effect proved as material-unspecific with a midfrontal topography in both conditions (Figure 3). The statistical analysis aimed at substantiating these observations by detecting electrodes that show significant modulations of ERD/ERS by the level of associative fan, i.e., the number of long-term representations that are being retrieved. For the upper alpha band, the overall ANOVA revealed an effect of fan level being present during most of the post-stimulus period. Main effects of fan level proved as reliable between 500 and 1500 ms poststimulus, with F(2,28) 5 8.75, p 5 .0011, e(H-F) 5 .9700 and F(2,28) 5 4.62, p 5 .0215, e(H-F) 5 .8276, between 3000 and 4500 ms, with minF(2,28) 5 4.30, p 5 .0235, e(H-F) 5 .9472 and maxF(2,28) 5 12.41, po.0001, e(H-F) 5 .7975 (minF and maxF referring to the smallest and largest F values within the significant time interval), and between 5000 and 6500 ms, minF(2,28) 5 6.96, p 5 .0095, e(H-F) 5 .6637; maxF(2,28) 5 7.30, p 5 .0059, e(H-F) 5 .7302. These effects were accompanied by a significant interaction of Fan level ! Electrodes between 6000 and 6500 ms, F(36,504) 5 1.86, p 5 .0258, e(H-F) 5 .2009, and a significant interaction of Fan level ! Material Type ! Electrodes between 3500 and 4000 ms, F(36,504) 5 2.03, p 5 .0300, e(H-F) 5 .1645. Together, these results indicate a temporally stable and broadly distributed effect of fan level, which, as the three-way interaction between 3500 and 4000 ms confirms, has a material-specific topography. Electrodespecific t-tests of the fan effect, i.e., the difference between the highest and lowest fan condition, showed significant (all po.05) fan-related modulations at frontal (Fp1, Fp2, F7, F3, Fz, F4), central (C3, Cz, C4), left temporal (T3, T5), and parietal (P3, Pz, P4) electrodes for positions, and at frontal (F7, Fz, F8) and left temporal (T3) electrodes for objects (for an overview of those electrodes from the complete set of 61 scalp electrodes that show a significant effect of fan, significance maps are provided below the topographical maps in Figures 2, 3, and 4). As we were most interested in parametric signal modulations, i.e., monotonic decreases (ERD) or increases (ERS), linear and quadratic trends were computed for the significant electrodes. For both positions and objects, all of the electrodes listed above exhibited significant linear trends. For objects, none of the four electrodes showed a significant quadratic trend. For positions, electrode F7 also showed a significant quadratic trend. As can be seen in the bar graphs in Figure 2, the increasing number of to-be-reactivated associations became manifest in an increasing ERD of the upper alpha band. The topographical maps of the effect of fan level, i.e., the difference of the highest minus the lowest level, show that the effect has its maximum over the parietal scalp when positions are recalled, and a broader distributed maximum that extends over the frontal, parietal, and left temporal scalp when objects are recalled. After having determined these neural correlates of the effect of associative fan, we wanted to find out whether they are correlated with the behavioral effect of fan across participants. Therefore, we correlated (1) the differences between the highest and lowest fan levels for the response times when recalling objects
366
P. H. Khader & F. Ro¨sler
Figure 2. Effect of the number of retrieved stimuli (i.e., the level of associative fan) on event-related power changes in the upper and lower alpha band. The difference maps of ‘‘Fan3–Fan1’’ for recalling positions (left side) and objects (right side) between 3500 and 4000 ms after onset of the retrieval cue show that the most pronounced effect for positions is located at parietal electrodes, whereas the difference was more broadly distributed towards frontal electrodes when objects had to be retrieved. The bar plots of the mean ERD amplitudes between 3500 and 4000 ms poststimulus, as well as the ERD time courses shown below the bar plots, illustrate that at these electrode sites alpha power decreases with fan level. Topographic maps in this figure and the following figures are based on all 61 electrodes. Below the topographical maps, significance maps are shown in which black regions denote electrodes with significant (po.05) effects of fan.
with the corresponding difference of the upper-alpha power between 3500 and 4000 ms poststimulus at electrode Fz for recalling objects, and (2) the differences between the highest and lowest fan levels for the response times when recalling spatial positions with the corresponding difference of the upper-alpha power between 3500 and 4000 ms poststimulus at electrode Pz for recalling spatial positions. However, these correlations (.01 for objects and .29 for positions) were small and not significant (p 5 .987 for objects and p 5 .287 for positions). As can be seen in Figure 2 (lower panel), the lower alpha band responded very similarly to the upper alpha band. Main effects of fan level were found between 500 and 1500 ms, with F(2,28) 5 5.89, p 5 .0073, e(H-F) 5 1 and F(2,28) 5 6.21, p 5 .0059, e(H-F) 5 1, and between 3000 and 6500 ms, with minF(2,28) 5 3.67, p 5 .0431, e(H-F) 5 .1920 and maxF (2,28) 5 8.21, p 5 .0039, e(H-F) 5 .7790, accompanied by an interaction of Fan level ! Electrodes between 6000 and 6500 ms, F(36,504) 5 2.20, p 5 .0010, e(H-F) 5 .6886, and an interaction of Fan level ! Material Type ! Electrodes between 3500 and 4000 ms, F(36,504) 5 1.90, p 5 .0448, e(H-F) 5 .2966.
Fan-related modulations were found at frontal (Fp1, Fp2, F7, F3, F4), central (C3, Cz, C4), left temporal (T3, T5), and parietal (P3, Pz, P4) electrodes for positions, and at mid-frontal (Fz) and left temporal (T3) electrodes for objects. For both positions and objects, all of the electrodes listed above exhibited significant linear, but no quadratic trends. In contrast to activity in the upper alpha band, activity of the theta band increased with increasing fan level. Here, significant main effects of fan level were found between 2500 and 3500 ms with F(2,28) 5 3.40, p 5 .0476, e 5 1, and F(2,28) 5 4.16, p 5 .0378, e 5 .7818. This effect proved as material-unspecific since no interactions with factor material type was found. As can be seen in Figure 3, this fan-dependent theta power increase has maxima at anterior frontal and right temporal electrodes. Electrode-specific t-tests computed for the two significant time windows confirmed significant fan effects at frontal (Fz, F4) and right temporal-parietal (P4, T6) electrodes. Again, linear and quadratic trend components were computed for the significant electrodes. The linear trend was significant at electrode P4, F(1,14) 5 5.25; p 5 .0379, and marginally significant at Fz,
EEG power changes during LTM retrieval
Figure 3. Effect of the number of retrieved items (i.e., the level of associative fan) on event-related power changes in the theta band. The difference map of ‘‘Fan3–Fan1’’ between 2500 and 3500 ms after onset of the retrieval cue reveals that the most pronounced effect is located at midfrontal electrodes. The bar plots from electrodes Fz and T6 show that theta power increases monotonically with increasing fan level at Fz, but not at T6.
F(1,14) 5 3.35; p 5 .0884, F4, F(1,14) 5 3.34; p 5 .0890, and T6, F(1,14) 5 3.68; p 5 .0758. The quadratic trend proved as highly significant for the posterior electrodes P4, F(1,14) 5 9.04; p 5 .0094, and T6, F(1,14) 5 14.69; p 5 .0018, as marginally significant for F4, F(1,14) 5 3.99; p 5 .0657, and as insignificant for Fz, F(1,14) 5 1.23; p 5 .2852. The trend analysis indicates that a monotonic increase of theta power with an increasing number of retrieved LTM representations is present only over mid-frontal scalp (see the bar plots in Figure 3), whereas at more posterior electrodes the trend is non-monotonic (mean power values: P4: Fan1 5 ! 1.18, Fan2 5 14.50, Fan3 5 9.15; T6: Fan1 5 ! 2.64, Fan2 5 16.33, Fan3 5 8.85). As with the alpha band, the correlation of the difference between the highest and lowest fan levels for the response times and the corresponding difference of theta power between 2500 and 3500 ms poststimulus at electrode Fz was small (.16) and not significant (p 5 .571). Finally, for the beta band, a material-specific power decrease with fan level could be observed (see Figure 4), with main effects of fan level between 500 and 1500 ms, F(2,28) 5 7.32, p 5 .0030, e(H-F) 5 .9789 and F(2,28) 5 4.49, p 5 .0203, e(H-F) 5 1, between 3000 and 4500 ms, minF(2,28) 5 5.44, p 5 .0149, e(H-F) 5 .8429 and maxF(2,28) 5 11.79, p 5 .0004, e(H-F) 5 .8901, and between 5000 and 6000 ms, F(2,28) 5 5.04, p 5 .0170, e(H-F) 5 .8958 and F(2,28) 5 4.93, p 5 .0152, e(H-F) 5 .9816, accompanied by an interaction of Fan level " Material Type " Electrodes between 3500 and 4000 ms, F(36,504) 5 2.18, p 5 .0059, e(H-F) 5 .4556. In contrast to alpha, however, fanrelated modulations were found at more electrodes for objects compared to positions (see Figure 4), i.e., at frontal (Fp1, F3, F8), central (C3), left temporal (T3), and parietal (Pz) electrodes for positions, and at frontal (Fp1, Fp2, Fz, F4, F8), left temporal (T3, T5), central (C3, Cz, C4), and parietal (P3, Pz, P4) electrodes for objects. For both positions and objects, all of the
367
Figure 4. Effect of the number of retrieved stimuli (i.e., the level of associative fan) on event-related power changes in the beta band. The difference maps of ‘‘Fan3–Fan1’’ for recalling positions (left side) and objects (right side) between 3500 and 4000 ms after onset of the retrieval cue show that the most pronounced effect for positions is located at left temporal-parietal electrodes, whereas the difference was more broadly distributed towards right parietal and left frontal electrodes when objects had to be retrieved.
electrodes listed above exhibited significant linear, but no quadratic trends.
Discussion Employing a paradigm that enforces associative LTM retrieval without confounding recall with perceptual processing, we found evidence for material-specific and material-unspecific oscillatory dynamics during associative LTM retrieval. Both response times and error rates showed parametric increases with fan level, proving that the paradigm was successful in eliciting a parametrically controlled increase of retrieval load the more associations had to be checked. Correspondingly, EEG power measures responded to the number of retrieved items as well. For the alpha band, this effect dissociated topographically for the two types of material. As material-specificity is an essential feature of the distributed model of LTM representations (McClelland et al., 1995; O’Reilly & Rudy, 2001; Squire & Alvarez, 1995), this indicates that material-specific cortical cell assemblies became systematically modulated the more associations had to be accessed. For recalling positions, the strongest parametric modulation was found at parietal electrodes. This finding is in line with previous fMRI studies of spatial LTM retrieval using the same paradigm and material (Khader et al., 2005, 2007). There, the most pronounced hemodynamic responses were found in the superior and inferior parietal cortex when positions were recalled. For objects, the topography of the alpha modulation was distributed more broadly and extended towards frontal electrode sites. This again corresponds nicely with our previous fMRI results (Khader et al., 2007), where the most pronounced hemodynamic responses for recalling objects were found within the left middle/inferior frontal gyrus. The present data provide evidence for a material-specific rather than a material-unspecific involvement of alpha oscillations in LTM retrieval. This suggests that decreasing alpha power is functionally related to the activation of stored engrams in circumscribed cortical networks that are specialized for process-
368
P. H. Khader & F. Ro¨sler
ing the one or the other type of information. Previous findings on effects in the upper alpha band reviewed in the introduction suggested that these effects are in general related to semantic processing demands (Klimesch, Doppelmayr, Pachinger, & Ripper, 1997a, Klimesch, Doppelmayr, Pachinger, & Russegger, 2007b) without specifying whether alpha ERD indexes the activation of memory trace as such or more general control processes. The results of the present study complement and clarify these observations in three points. First, they show that upper alpha ERD is not restricted to memory retrieval in the semantic domain alone, but that the effect also accompanies retrieval in other material domains. Second, the clear topographic dissociation found in the present study supports the notion that alpha ERD is a sign of material-specific activation processes. Third, as the lower alpha band responded very similarly to the upper alpha band, the present data speak for a general role of the alpha band for the proposed processes, not only for the upper alpha band. In the beta band, we also found material-specific fan-related power decreases. However, we did not find the topographies obtained in the alpha band (see Figures 2 and 4), and, in contrast to alpha, fan-related modulations were found at more electrodes for objects compared to positions. As there is little motivation from the theoretical background for the beta band, we refrain from discussing the effect with respect to possible underlying mechanisms, but utilize this finding to show that the effects found in the alpha band are specific to the alpha band. Nevertheless, a closer look at the literature reveals that there are a few hints to the role of beta oscillations for LTM retrieval. For example, in Hanslmayr et al. (2009), successful LTM formation in a semantic encoding task was related to power decreases in the beta (12–20 Hz) band. This result, together with the present study, suggests that beta is related to LTM processes, but the exact nature of these processes have to be specified in future studies. In contrast to the power decreases, the event-related power increases in the theta band were material-unspecific and showed a monotonic increase over mid-frontal areas, i.e., scalp sites that were distinct from those where the monotonic alpha ERD increases appeared. This supports the notion that both the alpha power increase and the theta power decrease are functionally independent. Moreover, the specific topography at Fz and the material independence of the theta effect suggest that it is most likely associated with unspecific control processes during memory retrieval. The findings on theta power reviewed in the introduction suggest that changes in this frequency band are also functionally related to maintenance of information in WM. If this holds true, material-specific effects are possible as well because of two reasons. First, different WM buffers have been proposed for different materials (Baddeley, 2003; Jonides, Sylvester, Lacey, Wager, Nichols, & Awh, 2003). Second, there is evidence that WM is essentially a state of activated LTM and therefore should recruit the same neural networks in which enduring memory representations are stored (Postle, 2006; Ruchkin, Grafman, Cameron, & Berndt, 2003). According to the distributed model of LTM representations (McClelland et al., 1995; O’Reilly & Rudy, 2001; Squire & Alvarez, 1995), these representations should be housed by material-specific cortical
cell assemblies. However, the material-unspecific increase of theta power with increasing retrieval demands observed in the present study does not support the idea that this activity is a sign of material-specific WM buffers. Rather, it suggests that theta is a manifestation of more general control processes that are invoked during both LTM retrieval and WM maintenance. The mid-frontal topography of the theta power increase further supports the notion of general control processes. Generators of theta activity with this topography have been localized to the anterior cingulate cortex (Jensen & Tesche, 2002; Luu, Tucker, & Makeig, 2004), a brain area that is generally involved in executive control processes (Botvinick, Cohen, & Carter, 2004; MacDonald, Cohen, Stenger, & Carter, 2000). Furthermore, Klimesch et al. (2001a) investigated whether theta-power increases during memory retrieval are related to the access of a stored code or to more general processes. In a recognition task with pictures, they found that theta power was higher during encoding than retrieval, but did not differ between new and successfully retrieved old pictures. The authors suggested that the increase in theta during retrieval may reflect general processing demands of a complex episodic memory system. In the present study, we found neural correlates of retrieving a varying number of stimuli from LTM. Therefore, we expected that these effects are correlated with the systematic slowing of response times with an increasing number of retrieved stimuli. However, such correlations were small and not significant. Insignificant effects are always problematic to interpret, because it is not clear why there was no significant effect. Possible reasons range from a high variance across participants to other variables that could compromise the correlation. Therefore, we do not want to speculate too much on the absence of these correlations. However, it should always be considered that responses are given after information processing is completed, summing up the durations and interactions of all processes that take place from stimulus encoding to response execution. Electrophysiological measures, in contrast, capture transient neural activation elicited by specific processes on-line during the course of information processing. Therefore, one would not necessarily expect an exact match between these two measures. For the present study, one possible reason for this discrepancy could be that the response times are more related to post-retrieval comparison processes than to the reactivation process proper. After the stimuli have been retrieved (which is accompanied by alpha ERD reflecting the activation of the stimulus representations and by theta ERS reflecting control over the activation process), they have to be maintained in working memory and have to be compared for common associations. This process could be much more timeconsuming than the activation process and should therefore substantially affect the response times, but not the alpha and theta oscillations. To conclude, by employing a memory-retrieval paradigm that is suited to monitor the controlled activation of LTM representations, different oscillatory correlates of LTM retrieval could be delineated. Whereas alpha is supposed to reflect activation of stored engrams, theta seems to be involved in retrieval-related control processes that mediate the activation of specific engrams.
REFERENCES Anderson, J. (1974). Retrieval of propositional information from longterm memory. Cognitive Psychology, 6, 451–474.
Baddeley, A. (2003). Working memory: Looking back and looking forward. Nature Reviews Neuroscience, 4, 829–839.
EEG power changes during LTM retrieval Badre, D., & Wagner, A. D. (2007). Left ventrolateral prefrontal cortex and the cognitive control of memory. Neuropsychologia, 45, 28830–28901. Botvinick, M. M., Cohen, J. D., & Carter, C. S. (2004). Conflict monitoring and anterior cingulate cortex: An update. Trends in Cognitive Sciences, 8, 539–546. Gevins, A., Smith, M. E., McEvoy, L., & Yu, D. (1997). High-resolution EEG mapping of cortical activation related to working memory: Effects of task difficulty, type of processing, and practice. Cerebral Cortex, 7, 374–385. Hanslmayr, S., Spitzer, B., & Ba¨uml, K. H. (2009). Brain oscillations dissociate between semantic and nonsemantic encoding of episodic memories. Cerebral Cortex, 19, 1631–1640. Heil, M., Ro¨sler, F., & Hennighausen, E. (1994). Dynamics of activation in long-term memory: The retrieval of verbal, pictorial, spatial, and color information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 185–200. Hennighausen, E., Heil, M., & Ro¨sler, F. (1993). A correction method for DC drift artifacts. Electroencephalography and Clinical Neurophysiology, 86, 199–204. Huynh, H., & Feldt, L. S. (1976). Estimation of the Box correction for degrees of freedom from sample data in randomized block and splitplot designs. Journal of Educational Statistics, 1, 69–82. Jensen, O., Gelfand, J., Kounios, J., & Lisman, J. E. (2002). Oscillations in the alpha band (9–12 Hz) increase with memory load during retention in a short-term memory task. Cerebral Cortex, 12, 877–882. Jensen, O., & Tesche, C. D. (2002). Frontal theta activity in humans increases with memory load in a working memory task. European Journal of Neuroscience, 15, 1395–1399. Jonides, J., Sylvester, C. Y. C., Lacey, S. C., Wager, T. D., Nichols, T. E., & Awh, E. (2003). Modules of working memory. In R. H. Kluwe, G. Lu¨er, & F. Ro¨sler (Eds.), Principles of learning and memory (pp. 113– 134). Cambridge, MA: Birkha¨user. Kahana, M. J., Seelig, D., & Madsen, J. R. (2001). Theta returns. Current Opinion in Neurobiology, 11, 739–744. Khader, P., Burke, M., Bien, S., Ranganath, C., & Ro¨sler, F. (2005). Content-specific activation during associative long-term memory retrieval. NeuroImage, 27, 805–816. Khader, P., Heil, M., & Ro¨sler, F. (2005). Material-specific long-term memory representations of faces and spatial positions: Evidence from slow event-related brain potentials. Neuropsychologia, 43, 2109–2124. Khader, P., Jost, K., Ranganath, C., & Ro¨sler, F. (2010). Induced EEG oscillations reflect successful long-term memory encoding during working-memory maintenance. Neuroscience Letters, 468, 329–343. Khader, P., Knoth, K., Burke, M., Bien, S., Ranganath, C., & Ro¨sler, F. (2007). Topography and dynamics of associative long-term memory retrieval in humans. Journal of Cognitive Neuroscience, 19, 492–511. Khader, P., & Ro¨sler, F. (2009). Where memories are stored discloses how memories are formedFContent specificity of long-term memory representations. In F. Ro¨sler, C. Ranganath, B. Ro¨der, & R. H. Kluwe (Eds.), Neuroimaging of human memoryFLinking cognitive processes to neural systems. New York: Oxford University Press. Klimesch, W. (1996). Memory processes, brain oscillations and EEG synchronization. International Journal of Psychophysiology, 24, 61–100. Klimesch, W. (1999). EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Research Reviews, 29, 169–195. Klimesch, W., Doppelmayr, M., Pachinger, T., & Ripper, B. (1997a). Brain oscillations and human memory: EEG correlates in the upper alpha and theta band. Neuroscience Letters, 238, 9–12. Klimesch, W., Doppelmayr, M., Pachinger, T., & Russegger, H. (1997b). Event-related desynchronization in the alpha band and the processing of semantic information. Cognitive Brain Research, 6, 83–94. Klimesch, W., Doppelmayr, M., Russegger, H., & Pachinger, T. (1996). Theta band power in the human scalp EEG and the encoding of new information. NeuroReport, 7, 1235–1240. Klimesch, W., Doppelmayr, M., Schimke, H., & Ripper, B. (1997c). Theta synchronization and alpha desynchronization in a memory task. Psychophysiology, 34, 169–176. Klimesch, W., Doppelmayr, M., Stadler, W., Pollhuber, D., Sauseng, P., & Ro¨hm, D. (2001a). Episodic retrieval is reflected by a process specific increase in human electroencephalographic theta activity. Neuroscience Letters, 302, 49–52.
369 Klimesch, W., Doppelmayr, M., Yonelinas, A., Kroll, N. E., Lazzara, M., Ro¨hm, D., & Gruber, W. (2001b). Theta synchronization during episodic retrieval: Neural correlates of conscious awareness. Cognitive Brain Research, 12, 33–38. Klimesch, W., Hanslmayr, S., Sauseng, P., Gruber, W., Brozinsky, C. J., Kroll, N. E., et al. (2006). Oscillatory EEG correlates of episodic trace decay. Cerebral Cortex, 16, 280–290. Luu, P., Tucker, D. M., & Makeig, S. (2004). Frontal midline theta and the error-related negativity: Neurophysiological mechanisms of action regulation. Clinical Neurophysiology, 115, 1821–1835. MacDonald, A. W., Cohen, J. D., Stenger, V. A., & Carter, C. S. (2000). Dissociating the role of the dorsolateral prefrontal and anterior cingulate cortex in cognitive control. Science, 288, 1835–1838. McClelland, J. L., McNaughton, B. L., & O’Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102, 419–457. McEvoy, L. K., Pellouchoud, E., Smith, M. E., & Gevins, A. (2001). Neurophysiological signals of working memory in normal aging. Cognitive Brain Research, 11, 363–376. Mecklinger, A., Kramer, A. F., & Strayer, D. L. (1992). Event related potentials and EEG components in a semantic memory search task. Psychophysiology, 29, 104–119. Nunez, P., & Srinivasan, R. (2006). Electrical fields of the brain: The neurophysics of EEG (2nd edition). New York: Oxford University Press. Onton, J., Delorme, A., & Makeig, S. (2005). Frontal midline EEG dynamics during working memory. NeuroImage, 27, 341–56. O’Reilly, R. C., Noelle, D. C., Braver, T. S., & Cohen, J. D. (2002). Prefrontal cortex and dynamic categorization tasks: Representational organization and neuromodulatory control. Cerebral Cortex, 12, 246–257. O’Reilly, R. C., & Rudy, J. W. (2001). Conjunctive representations in learning and memory: Principles of cortical and hippocampal function. Psychological Review, 108, 311–345. Owen, A. M., & Hampshire, A. (2009). The mid-ventrolateral frontal cortex and attentional control. In F. Ro¨sler, C. Ranganath, B. Ro¨der, & R. H. Kluwe (Eds.), Neuroimaging of human memoryFLinking cognitive processes to neural systems. New York: Oxford University Press. Pfurtscheller, G., & Lopes da Silva, F. H. (1999). Event-related EEG/ MEG synchronization and desynchronization: Basic principles. Clinical Neurophysiology, 110, 1842–1857. Postle, B. R. (2006). Working memory as an emergent property of the mind and brain. Neuroscience, 139, 23–38. Raghavachari, S., Kahana, M. J., Rizzuto, D. S., Caplan, J. B., Kirschen, M. P., Bourgeois, B., et al. (2001). Gating of human theta oscillations by a working memory task. Journal of Neuroscience 21, 3175–3183. Ruchkin, D. S., Grafman, J., Cameron, K., & Berndt, R. S. (2003). Working memory retention systems: A state of activated long-term memory. Behavioral and Brain Sciences, 26, 709–728; discussion 728–777. Ro¨sler, F., Heil, M., & Hennighausen, E. (1995). Distinct cortical activation patterns during long-term memory retrieval of verbal, spatial, and color information. Journal of Cognitive Neuroscience, 7, 51–65. Sederberg, P. B., Kahana, M. J., Howard, M. W., Donner, E. J., & Madsen, J. R. (2003). Theta and gamma oscillations during encoding predict subsequent recall. Journal of Neuroscience, 23, 10809–10814. Squire, L. R., & Alvarez, P. (1995). Retrograde amnesia and memory consolidation: A neurobiological perspective. Current Opinion in Neurobiology, 5, 169–177. Steriade, M. (1999). Cellular substrates of brain rhythmus. In E. Niedermeyer (Ed.), Electroencephalography: Basic principles, clinical applications, and related fields (pp. 27–59). Philadelphia: Lippincott Williams & Wilkins. Tesche, C. D., & Karhu, J. (2000). Theta oscillations index human hippocampal activation during a working memory task. Proceedings of the National Academy of Sciences, USA, 97, 919–924. Weiss, S., & Rappelsberger, P. (2000). Long-range EEG synchronization during word encoding correlates with successful memory performance. Cognitive Brain Research, 9, 299–312.
(Received November 2, 2009; Accepted April 7, 2010)
Psychophysiology, 48 (2011), 370–376. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01067.x
The effects of EEG data transformations on the solution accuracy of principal component analysis
JAMES E. ARRUDA,a HEATHER A. MCGEE,a HONGMEI ZHANG,b and CLAUDIA J. STANNYa a
Department of Psychology, University of West Florida, Pensacola, Florida, USA Department of Epidemiology and Biostatistics, University of South Carolina, Columbia, South Carolina, USA
b
Abstract Principal component analysis (PCA) is a commonly used multivariate procedure that reduces the dimensionality of a data set. When applied to quantitative electroencephalogram (qEEG) data, PCA produces components that may represent functional systems within the brain. Unfortunately, qEEG, like many other physiological measures, produce distributions that are positively skewed. In response, researchers often transform qEEG data prior to conducting a PCA, which does not require univariate or multivariate normality. Despite this, researchers continue to transform qEEG data with limited knowledge of how such transformations will affect the accuracy (precision) of their component solutions. The purpose of the present investigation was to examine the effects of several commonly used data transformation procedures on PCA solution accuracy. Descriptors: Electroencephalogram, EEG, Principal component analysis, PCA, Factor analysis, FA, Transformation, Absolute power, Relative power, Log
(qEEG) data, PCA may be the more desirable extraction method to use with qEEG data (Green & Salkind, 2005). The process of conducting a PCA on qEEG data requires several important decision-making steps that will have a profound effect on a final PCA solution. Arruda et al. (1996) documented these decision-making steps and established guidelines for the application of PCA to qEEG data, which include the choice of matrix, the choice of rotation procedure, and the choice of extraction method. Unfortunately, the effects of many commonly employed qEEG data transformation procedures have yet to be investigated. qEEG, like many other physiological measures, produces distributions of scores that deviate from normal (Osborne, 2002; Riniolo & Porges, 2000). In fact, positively skewed distributions are so frequently obtained for qEEG variables that they are considered to be a natural characteristic of qEEG data. Examples of skewed distributions are prevalent throughout the qEEG literature and include distributions obtained for all frequency bands and for a variety of different population types (e.g., adults, children, clinical, and nonclinical; Fein et al., 1983; Galbraith & Wong, 1993; Gasser et al., 1982; John, Ahn, & Prichep, 1980; Oken & Chiappa, 1988; Pollock, Schneider, & Lyness, 1990; Weiler, 1993). Positively skewed distributions, unlike normal distributions, are asymmetric and possess means that are larger than either the median or the mode of the distribution from which they come. Unfortunately, nonnormal distributions, such as those seen in qEEG research, can be an obstacle for researchers who plan to perform a variety of parametric analyses (Albada & Robinson, 2007; Dunlap, Chen, & Greer, 1994; Levine & Dunlap, 1982; Weiler, 1993).
Recent advances in technology have allowed for the use of an increased number of electrode recording sites in electroencephalogram (EEG) studies. In fact, some systems have up to 256 leads, making both data analysis and model building an extremely cumbersome task (Cacioppo, Tassinary, & Bernston, 2000). The result of such technological advances has been an increase in the number of dependent variables measured and an increase in the frequency with which data reduction procedures, including principal components analysis (PCA) and factor analysis (FA), are used (Arruda et al., 1996; Duffy, Jones, Bartels, McAnulty, & Alber, 1992; Gasser, Ba¨cher, & Mo¨cks, 1982; Schenk et al., 1982).1 Empirical investigations comparing PCA with FA have reported few practical differences in the solutions produced by the two techniques (Fava & Velicer, 1992b; Schenk et al., 1982; Velicer & Fava, 1987; Velicer, Peacock, & Jackson, 1982). Nonetheless, PCA is more commonly used by researchers (Guadagnoli & Velicer, 1988; Velicer & Fava, 1998) and, given the distributional (nonnormal) nature of quantitative electroencephalogram Address correspondence to: James E. Arruda, Ph.D., Associate Professor, Department of Psychology, 11000 University Parkway, Pensacola, FL 32514, USA. E-mail:
[email protected] 1 Although PCA and FA have many similarities, the two methods differ in the way components or factors, respectively, are derived. For example, whereas a PCA derives components by performing an eigen decomposition on either a correlation or variance–covariance matrix that is comprised of both shared and unique variance, FA derives factors by performing an eigen decomposition on either a correlation or variance– covariance matrix that is comprised of only shared varianceFunique variance is estimated and excluded prior to eigen decomposition (Tabachnick & Fidell, 2001). 370
Effects of EEG data transformations PCA, however, is a descriptive procedure that does not require either univariate or multivariate normality. Despite this, researchers continue to transform qEEG data prior to conducting PCAs with the belief that such transformations are necessary (Tabachnick & Fidell, 2001). Unfortunately, researchers who choose to transform qEEG data prior to performing a PCA do so with limited knowledge of how such transformations will affect the accuracy (precision) of their component solutions. For example, although it is known that transformations change the metric of the data that are transformed, it is unclear how such changes, including the alteration of correlations between measures, might affect the solution accuracy of a PCA. To illustrate, Tenke and Kayser (2005) have reported that the log power spectrum may misrepresent the underlying latent structure, particularly when used in concert with predetermined frequency bands. This uncertainty has led some researchers to analyze and report the results associated with both untransformed and transformed qEEG data, which can sometimes produce different component solutions (Arruda et al., 1996). The purpose of the present investigation was to examine the effects of several commonly used data transformation procedures on PCA solution accuracy, with solution accuracy being defined as the degree of fit between a population and a sample component solution. Although it has been suggested that nonnormal distributions may degrade PCA solutions and that component solution accuracy may improve when data are normally distributed (Tabachnick & Fidell, 2001), there have been no published reports, to our knowledge, that have empirically supported this assumption.
Method In this investigation, we attempted to create conditions that are often encountered by investigators who employ qEEG as a method of studying neurocognitive processes. As such, the population component patterns used in the present investigation were based on data collected by our laboratory, which represent complex, real-world conditions. A description of the process by which actual data were sampled and simulated data were generated can be seen below. Additional discussion pertaining to the choice of qEEG measures, decisions pertaining to the implementation of the PCA, and the choice of data transformation procedures also follow. Actual Data Participants. Comprising the sample were 65 healthy men (n 5 20) and women (n 5 45) with no self-reported history of psychiatric or neurological illness. The ages of participants ranged from 18 to 26 years (M 5 19.30, SD 5 1.53), and all participants were self-reported to have either normal or correctedto-normal hearing. This research received prior approval from the responsible Institutional Review Board, and all participants provided written informed consent prior to their participation. Apparatus. A computer, employing Neuroscan System 4 software, was used to time-lock qEEG recordings to synchronized pulses denoting the presentation of stimuli and the recording of responses. A separate computer, employing Neuroscan STIM2 software, was used to present auditory stimuli and to record performance. qEEG data were collected using a 64channel electrode cap. EEG was sampled from 58 active scalp
371 electrodes at 500 Hz and band passed 0.3–70 Hz ( ! 12 dB/octave 160 Hz notch) through two 32-channel Synamp DC amplifiers. In addition to the 58 active scalp electrodes, a reference electrode was attached to each earlobe (A1 and A2) and a ground electrode was situated between sites Fz and Fpz. All impedances were kept below 5 kO. Trials contaminated by eye-movement artifact were rejected using an off-line artifact rejection algorithm (Semlitsch, Anderer, Schuster, & Presslich, 1986). DC corrections were performed when appropriate, and a Hanning window was used to reduce edge effects. Procedure. Upon arrival, each participant was given a packet containing a brief demographic questionnaire. Once it was completed, each participant was seated in a comfortable chair while electrodes were placed according to an expanded version of the International 10–20 System of Electrode Placement. A 30-min, auditory continuous performance test (CPT) was then administered. Participants were instructed to listen, with eyes closed, to a digital recording containing pseudorandomly arranged letters of the alphabet (Valentino, Arruda, & Gold, 1993). Participants were instructed to press a button using the right hand each time a double-letter target was detected. Stimuli were presented every 2000 ms and double-letter targets occurred, on average, 1.21 times per minute. Scan analyses. EEG recordings were low-pass filtered at 50 Hz and epoched from ! 1024 ms before the target stimulus to 0 ms after the target stimulus (i.e., target delivery). Epochs were spline fit from 513 to 512 points and averaged in the frequency domain to extract absolute power values for each of the following four frequency bands: delta (0–3.90 Hz), theta (3.91–7.80 Hz), alpha (7.81–11.70 Hz), and beta (11.71–24.37 Hz). qEEG was collected from correct trials with the goal of activating neurophysiological processes that would drive the underlying latent structure of the PCA solution (Arruda et al., 1999). The mean absolute power for delta, theta, alpha, and beta were 23.90 (SD 5 20.21), 11.21 (SD 5 7.56), 22.65 (SD 5 19.35), and 1.48 (SD 5 0.76), respectively. Skewness and kurtosis values for each frequency band are reported in Table 1. Simulated Data: Generation Data were simulated using a multivariate normal distribution. Normalization was accomplished by transforming each original qEEG variable using a log base a transformation.2 Once normalized, a vector of means (l) and a matrix of covariances (R) were derived and used to simulate samples of data. Each sample of data was then transformed back to the original metric using a aY transformation so that the simulated data would possess the same characteristics (e.g., skewness and kurtosis) as the actual data. A comparison of the simulated (sample) and actual (population) distributions can be seen in Table 1. A more complete description of the data generation process can be seen in the appendix. Solution Accuracy: Summary Statistic (g2) Component solution accuracy was assessed using the g2 statistic (Guadagnoli & Velicer, 1988). The g2 statistic provides a measure of solution accuracy by comparing the pattern (solution) matri2 Transformations were performed because many of the qEEG variables were skewed and/or kurtotic. Indeed, many of the variables possessed skewness or kurtosis that was greater than 1.0 (Tabachnick & Fidell, 2001).
372
J.E. Arruda et al.
Table 1. Average Skewness and Kurtosis of Actual and Simulated Data Skewness
Kurtosis
2.08 1.29 1.64 1.06
5.14 1.42 3.84 1.42
1.84 1.09 1.47 1.01
5.46 2.09 3.47 1.76
a
Actual data Delta Theta Alpha Beta Simulated datab Delta Theta Alpha Beta
ces produced by sample (simulated) and population (actual) data. More formally, g2 is derived by averaging the squared deviations between corresponding loadings of two pattern matrices: ðg2 ¼ traceðE 0 EÞ=ðpmÞ; where E ¼ A $ A% Þ: The terms p, m, A, and An represent the number of variables, the number of components, the sample pattern matrix, and the population pattern matrix, respectively. Smaller values of g2 were indicative of greater solution accuracy, with g2 values less than .01 representing a good fit between a sample and a population pattern matrix (Guadagnoli & Velicer, 1988). Guadagnoli and Velicer established the .01 criterion while examining the fit between sample and population pattern matrices in 1988. However, in that investigation, Guadagnoli and Velicer generated population pattern matrices that were much simpler than the population pattern matrices used in the present investigation. Hence, it is likely that values larger than .01 may represent an acceptable fit when complex, real-world conditions are considered. Quantitative Electroencephalogram Measures: Choice of Absolute Power or Relative Power Two commonly used qEEG measures are absolute power (AP) and relative power (RP). Each of these measures can be calculated for a frequency band measured at a single electrode-recording site. Researchers disagree as to which measure provides a better index of brain function (Caldwell & Roberts, 1997; Fein et al., 1983; Gasser et al., 1982; John et al., 1977, 1980; John, Prichep, Fridman, & Easton, 1988; Pollock et al., 1990; Weiler, 1993). Despite this disagreement, there appears to be a preference for AP measures. This may be due to the difficulty investigators experience while interpreting RP change, which is often dependent upon RP measures obtained from the same electrode recording site. Regardless, a clear majority of applied qEEG literature recommends the use of AP measures (Arruda et al., 1996; Caldwell & Roberts, 1997; Pivik et al., 1993; Weiler, 1993). Therefore, AP in the following four frequency bands was examined in the present investigation: delta (0.00–3.90 Hz), theta (3.91–7.80 Hz), alpha (7.81–11.70 Hz), and beta (11.71–24.37 Hz). Principal Component Analysis Choice of correlation or variance–covariance matrix. The process of conducting a PCA on raw data begins with the creation of either a variance–covariance matrix or a correlation matrix. A variance–covariance matrix should be used only if all variables are in the same metric or have approximately equal variances. In cases where variables are in different metrics or have unequal variancesFa situation that commonly occurs when multiple
frequency bands and electrode recording sites are entered into a PCAFthe use of a correlation matrix is recommended (Arruda et al., 1996; Weiler, 1993). Therefore, correlation matrices were used in the present investigation, with each correlation matrix being comprised of absolute power (i.e., delta, theta, alpha, or beta) sampled from each of the 58 active electrode-recording sites. All correlation matrices were based on a sample size of 100. Choice of retention method. Retention refers to the process by which the number of components in a PCA solution is restricted (Green & Salkind, 2005). Several empirical decision methods are available for determining the number of components to retain, including the Kaiser eigenvalue-greater-than-one rule (Kaiser, 1960), Cattell’s scree test (Cattell, 1966), and Horn’s parallel analysis (Horn, 1965; Lautenschlager, 1989). In the present investigation, the number of components was restricted using the Kaiser eigenvalue-greater-than-one rule. The decision to use the eigenvalue-greater-than-one rule was based on the feasibility of incorporating the restriction criterion into a SAS program. Implementation of the alternative methods (i.e., scree test, parallel analysis) would have been nearly impossible, given the large number of PCA analyses planned for this study. More importantly, the Kaiser rule is the most commonly used criterion for component retention; in fact, most statistical software packages use the Kaiser rule as the default method for selecting the number of components to retain (Costello & Osborne, 2005). Choice of rotation method. Component rotation, which can be either oblique or orthogonal in nature, can be performed to improve the interpretability of an obtained component solution. The choice between oblique or orthogonal rotations is often based on theory and the expected future use of component scores (Arruda et al., 1996). Given that a major goal of using PCA with EEG data is to return a set of uncorrelated measures, orthogonal rotations are the most frequently reported rotation procedures in the qEEG literature. Therefore, all PCAs performed in the present investigation were orthogonally rotated using a VARIMAX rotation procedure. Choice of Transformations A number of transformations can normalize positively skewed distributions. Such transformations work by ‘‘pulling down’’ observations that comprise the longer tail. Typically, transformations performed on AP qEEG variables are variants of root, log, and inverse transformations (Weiler, 1993). In general, root transformations are the least robust in terms of normalization, followed by log, and then inverse transformations (Osborne, 2002). Specific examples from EEG literature include square root, cube root, natural log (ln), log base 10 (log10), log10(11AP), and inverse square root (Weiler, 1993). The log10 transformation is by far the most popular transformation for AP variables (Pivik et al., 1993). Empirical research comparing the ability of different transformations to normalize AP distributions has repeatedly found the log transformations to be superior to all other transformations (Davidson, Chapman, Chapman, & Henriques, 1990; Fein et al., 1993; Gasser et al., 1982; John et al., 1980; Pollock et al., 1990; Weiler, 1993). Some of the most convincing evidence comes from comprehensive studies, such as those conducted by Gasser et al. and Weiler, which included a variety of transformations, multiple frequency bands, multiple recording sites, and numerous recording conditions.
Effects of EEG data transformations
373
To illustrate, Gasser et al. (1982) examined the distributional effects of transformations on a variety of AP variables collected during a resting state. Gasser and colleagues reported that the log10 transformation improved the normality of AP variables better than all other candidate transformations, which included log(11AP), square root, cube root, and inverse square root. In the case of delta, a square root transformation most normalized the distribution. Weiler (1993) provides further evidence that log10 transformations normalize AP measures better than other competing transformations. Weiler reported that the log10 transformation not only improved the normality of AP variables, but did so regardless of frequency band, recording site, or recording condition (i.e., resting, sustained attention). In the present investigation, AP data were transformed using all six data transformation procedures: square root(AP), cube root(AP), log10(AP), log10(11AP), and inverse square root(AP). The solution accuracy associated with each transformation was then compared to the solution accuracy associated with untransformed AP data. Finally, an eighth condition was added to provide a ‘‘worst case’’ circumstance to which all other conditions, including both transformed and untransformed data, could be compared. This was accomplished by randomly rearranging the AP scores comprising each variable prior to performing an eigen decomposition. This was done to remove natural correlations between qEEG variables while still preserving the statistical characteristics (e.g., mean, standard deviation) of each qEEG measure. The component solutions obtained from the random-untransformed data provided a worst case value of g2, to which all other comparisons could be referred. Results A two-way, between-subjects analysis of variance revealed a statistically significant interaction effect involving the type of frequency band (i.e., delta, theta, alpha, and beta) and the type of transformation method used (i.e., random-untransformed, untransformed, square root, cube root, log10, log10(11AP), and inverse square root), F(18,2772) 5 17.93, po.01. Cell means and standard deviations pertaining to the interaction effect can be seen in Table 2 and in Figure 1. The results of the follow-up, simple main effects tests are presented below. A significant simple main effect of transformation method was found for the theta frequency band, F(6,693) 5 57.73, po.01, the alpha frequency band, F(6,693) 5 36.32, po.01, and the beta frequency band, F(6,693) 5 36.92, po.01. Post hoc, pairwise comparisons using the Tukey HSD test revealed statistically significant differences in solution accuracy between the randomTable 2. Mean g2 Values Associated with Each Frequency Band and Transformation Method Combination Delta M Randomuntransformed Untransformed Square root Cube root Log10 Log10 (11AP) Inverse square root
SD
Theta M
SD
Alpha M
SD
Beta M
SD
.0663 .0020 .0678 .0021 .0677 .0020 .0675 .0023 .0723 .0723 .0744 .0764 .0761 .0752
.0145 .0169 .0178 .0179 .0182 .0175
.0403 .0387 .0388 .0394 .0395 .0397
.0145 .0141 .0163 .0161 .0162 .0142
.0326 .0323 .0299 .0293 .0287 .0289
.0296 .0268 .0242 .0234 .0236 .0245
.0353 .0318 .0303 .0279 .0294 .0295
.0287 .0260 .0244 .0238 .0232 .0237
untransformed condition and each of the remaining transformation conditions, including untransformed, square root, cube root, log10, log10(11AP), and inverse square root (po.01). All six of the aforementioned transformation methods produced solution accuracies that were significantly higher (i.e., smaller g2) than the solution accuracy produced by the random-untransformed condition. There were no statistically significant differences in solution accuracy between the untransformed, square root, cube root, log10, log10(11AP), and inverse square root transformation methods. A significant simple main effect of transformation method was also found for the delta frequency band, F(6,693) 5 4.77, po.01. However, post hoc pairwise comparisons revealed a slightly different pattern of solution accuracy differences. As can be seen in Table 2 and in Figure 1, the random-untransformed method produced solution accuracies that were significantly higher (i.e., smaller g2) than the cube root, log10, log10(11AP), and inverse square root transformation methods (po.01). There were, however, no significant differences in solution accuracy between the random-untransformed condition and the untransformed and square root transformation methods. No other significant pairwise differences were observed. One-sample t tests were performed to determine if the solution accuracies obtained for each transformation method and frequency band combination differed significantly from the .01 criterion level established by Guadagnoli and Velicer (1988). As might be expected, the mean solution accuracy associated with each transformation method and frequency band combination were significantly lower (i.e., larger g2 values) than the solution accuracy established by Guadagnoli and Velicer (p4.05). As can be seen in Table 2, the solution accuracies (i.e., g2) associated with the transformed and the untransformed theta, alpha, and beta frequency bands ranged from .03 to .04 (M 5 .03, SD 5 .01). The mean solution accuracy associated with the random-untransformed data (delta, theta, alpha, and beta) was .07 (SD 5 .00).
Discussion The purpose of the present investigation was to determine if six commonly used data transformation procedures, including square root(AP), cube root(AP), ln(AP), log10(AP), log10(11AP), and inverse square root(AP), would significantly affect the solution accuracy of a PCA. Arguments for data transformation generally suggest that the normalization of data will increase the reliability and the validity of results produced by a PCA (Tabachnick & Fidell, 2001). In contrast, arguments against data transformation suggest that a transformation may change the characteristics of the data and degrade a PCA solution (Arruda et al., 1996). Interestingly, neither assumption was supported by the findings of the present investigation. PCA solutions based on transformed qEEG data were as accurate as PCA solutions based on the untransformed qEEG data, and, with the exception of the delta frequency band, PCA solutions produced by both transformed and untransformed data were more accurate than the PCA solutions produced by the random-untransformed data. Results associated with the delta frequency band, on the other hand, were quite different. Whereas the solution accuracies of PCAs performed on the remaining frequency bands appeared to be quite good, the solution accuracies of PCAs performed on the delta frequency band were not. In fact, the use of some transformation methods, including the log10, the log10(11AP), and the inverse square root,
374
Figure 1. Interaction effect involving frequency band and transformation method. Smaller g2 values are indicative of greater solution accuracy or fit between a population component solution and a sample component solution.
produced solution accuracies that were even lower than the random-untransformed method. A likely explanation for these findings may be related to the nature of the delta frequency band and to the experimental conditions used to record the qEEG data. Previous research by our laboratory (Arruda et al., 1999) and by the laboratories of others (Valentino et al., 1993) have demonstrated that absolute power in the delta frequency range declines during periods of increased vigilance. Given that the qEEG collected in the present investigation was collected solely from correct trials, absolute power within the delta frequency range may have had a diminished presence in the electroencephalographic record. This, in turn, would have caused an attenuation of any underlying latent structure within the delta frequency band. Finally, the correction of eye-movement artifact performed in the present investigation would have further diminished the presence of any remaining latent structure (Semlitsch et al., 1986). Taken together, it is likely that the delta frequency band had very little latent structure to identify and even less latent structure to drive a valid component solution. Under these conditions, the accuracy of a PCA solution would be quite poor, bringing into question the validity of any PCA solution based on the delta frequency band when the delta frequency band is measured during a highly vigilant state. Results associated with the remaining frequency bands, which include theta, alpha, and beta, suggest a number of important implications for the application of PCA with qEEG data. First, PCA solution accuracy was not enhanced by any of the commonly used qEEG data transformation procedures. This finding clearly contradicts the commonly held but unsubstantiated belief that PCA solution accuracy improves with the transformation of qEEG data (Tabachnick & Fidell, 2001). Therefore, the transformation of qEEG data prior to performing a PCA should not be viewed as either necessary or beneficial to the accuracy of a PCA solution. Second, given that solution accuracy was not degraded by the use of a data transformation procedure, investigators should feel free to transform qEEG when subsequent analyses require the use of normally distributed data. The findings reported here are consistent with the results reported by other investigators that have supported the use of the log10 transformation procedure when inferential tests are to be performed (Davidson et al., 1990; Fein et al., 1993; Gasser et al., 1982; John et al., 1980; Pollock et al., 1990; Weiler, 1993). This
J.E. Arruda et al. finding is both consistent with a general trend among qEEG researchers to choose the log10 transformation over other transformation procedures and indirectly validates the method by which the data were simulated in the present investigation. Although the results of the current study are compelling and the implications of these findings are practical, the research reported here should be viewed as a small step toward a more comprehensive understanding of the effects of commonly used qEEG data transformation procedures. Indeed, there are a few limitations associated with the present investigation that must be addressed. First, the generalizability of the present findings may have been limited by the use of AP measures, correlation matrices, the eigenvalue greater-than-one criterion, and the varimax rotation. The extent to which the current results may generalize to different qEEG measures, variance–covariance matrices, the parallel analysis criterion, or oblique rotations, for example, is unknown. However, decisions concerning the choice of measure, matrix retention rule, and rotation were based on research published by our laboratory (Arruda et al., 1996) and by the laboratories of others (Fava & Velicer, 1992a, b; Velicer & Fava, 1987, 1998; Velicer et al., 1982; Velicer, Eaton, & Fava, 2000). Therefore, the findings obtained in the present investigation should generalize to those investigations that employ what most would consider to be a set of best practices for the application of PCA. Second, all sample data sets generated in the present investigation were comprised of 100 cases. Unfortunately, sample sizes of 100 or more cases are uncommon in many qEEG investigations, and the present investigation did not address whether solution accuracy would change with smaller sample sizes. Research conducted by our laboratory will soon address the issue of sample size and PCA solution accuracy. Finally, although the solution accuracies produced by each transformation method and frequency band combination were significantly lower than the .01 criterion level established by Guadagnoli and Velicer (1988), the population component patterns used in the present investigation were based on actual qEEG data that represent complex, real-world conditions. By contrast, Guadagnoli and Velicer used population component patterns that were contrived and much simpler than would be expected under real-world conditions. In fact, each pattern matrix constructed by Guadagnoli and Velicer was comprised of variables that loaded on a single component, and an equal number of variables defined each component. Further, all nonzero loadings were equal. Hence, the solution accuracies obtained in the present investigation (i.e., .03 to .04) may be more appropriate for applied qEEG research. In summary, the findings presented here suggest that PCA solution accuracy is unaffected by the transformation of qEEG data. Solutions associated with six of the most commonly used data transformation procedures, including square root(AP), cube root(AP), ln(AP), log10(AP), log10(11AP), and inverse square root(AP), were as accurate as the solutions associated with the untransformed qEEG data. This was true for all frequency bands with the exception of the delta frequency band. Solution accuracies associated with the delta frequency band, including those produced by both transformed and untransformed data, were either equal to or lower than the solution accuracy obtained with the random-untransformed qEEG data. This finding was attributed to a lack of underlying latent structure in the delta frequency band in the present investigation.
Effects of EEG data transformations
375 REFERENCES
Albada, S. J., & Robinson, P. A. (2007). Transformation of arbitrary distributions to the normal distribution with application to EEG testretest reliability. Journal of Neuroscience Methods, 161, 205–211. Arruda, J. E., Walker, K. A., Weiler, M. D., & Valentino, D.A. (1999). Validation of a right hemisphere vigilance system as measured by principal component and factor analyzed quantitative electroencephalogram. International Journal of Psychophysiology, 32, 119–128. Arruda, J. A., Weiler, M. D., Valentino, D., Willis, W. G., Rossi, J. S., Stern, R. A., et al. (1996). A guide for applying principal-components analysis and confirmatory factory analysis to quantitative electroencephalogram data. International Journal of Psychophysiology, 23, 63–81. Cacioppo, J. T., Tassinary, L. G., & Bernston, G. G. (Eds.) (2000). Handbook of psychophysiology (2nd ed). New York: Cambridge University Press. Caldwell, J. A. Jr., & Roberts, K. A. (1997). The practical effect of routine data transformations on absolute EEG power derived from spectral analysis (USAARL Rep. No. 98-04; NTIS No. AD-A332 880/4). Fort Rucker, AL: Army Aeromedical Research Laboratory. Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245–276. Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment Research & Evaluation, 10. Retrieved September 10, 2007, from http://pareonline.net/getvn. asp?v=10&n=7. Davidson, R. J., Chapman, J. P., Chapman, L. J., & Henriques, J. B. (1990). Asymmetrical brain electrical activity discriminates between psychometrically matched verbal and spatial cognitive tasks. Psychophysiology, 27, 528–543. Duffy, F., Jones, K., Bartels, P., McAnulty, G., & Alber, M. (1992). Unrestricted principal components analysis of brain electrical activity: Issues of data dimensionality, artifact, and utility. Brain Topography, 4, 291–306. Dunlap, W. P., Chen, R., & Greer, T. (1994). Skew reduces test-retest reliability. Journal of Applied Psychology, 79, 310–313. Fava, J. L., & Velicer, W. F. (1992a). The effects of overextraction on factor and component analysis. Multivariate Behavioral Research, 27, 387–415. Fava, J. L., & Velicer, W. F. (1992b). An empirical comparison of factor, image, component, and scale scores. Multivariate Behavioral Research, 27, 301–322. Fein, G., Galin, D., Johnstone, J., Yingling, C. D., Marcus, M., & Kiersch, M. E. (1983). EEG power spectra in normal and dyslexic children: I. Reliability during passive conditions. Electroencephalography and Clinical Neurophysiology, 55, 399–405. Galbraith, G. C., & Wong, E. H. (1993). Moment analysis of EEG amplitude histograms and spectral analysis: Relative classification of several behavioral tasks. Perceptual and Motor Skills, 76, 859–866. Gasser, T., Ba¨cher, P., & Mo¨cks, J. (1982). Transformations towards the normal distribution of broad band spectral parameters of the EEG. Electroencephalography and Clinical Neurophysiology, 53, 119–124. Green, S. A., & Salkind, N. J. (2005). Using SPSS for Windows and Macintosh: Analyzing and understanding data (4th ed.). Upper Saddle River, NJ: Pearson Prentice Hall. Guadagnoli, E., & Velicer, W. F. (1988). Relation of sample size to the stability of component patterns. Psychological Bulletin, 103, 265–275. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185. John, E. R., Ahn, H., & Prichep, L. (1980). Developmental equations for the electroencephalogram. Science, 210, 1255–1258. John, E. R., Karmel, B. Z., Corning, W. C., Easton, P., Brown, D., Ahn, H., et al. (1977). Neurometrics. Science, 196, 1393–1410. John, E. R., Prichep, L. S., Fridman, J., & Easton, P. (1988). Neurometrics: Computer-assisted differential diagnosis of brain dysfunctions. Science, 239, 162–169.
Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141–151. Lautenschlager, G. J. (1989). A comparison of alternatives to conducting Monte Carlo analyses for determining parallel analysis criteria. Multivariate Behavioral Research, 24, 365–395. Levine, D. W., & Dunlap, W. P. (1982). Power of the F test with skewed data: Should one transform or not? Psychological Bulletin, 92, 272– 280. Oken, B. S., & Chiappa, K. H. (1988). Short-term variability in EEG frequency analysis. Electroencephalography and Clinical Neurophysiology, 69, 191–198. Osborne, J. W. (2002). Normalizing data transformations. ERIC Digest (ERIC No. ED470204). College Park, MD: ERIC Clearinghouse on Assessment and Evaluation. Pivik, R. T., Broughton, R. J., Coppola, R., Davidson, R. J., Fox, N., & Nuwer, M. R. (1993). Guidelines for the recording and quantitative analysis of electroencephalographic activity in research contexts. Psychophysiology, 30, 547–558. Pollock, V. E., Schneider, L. S., & Lyness, S. A. (1990). EEG amplitudes in healthy, late-middle-aged and elderly adults: Normality of the distributions and correlations with age. Electroencephalography and Clinical Neurophysiology, 75, 276–288. Riniolo, T. C., & Porges, S. W. (2000). Evaluating group distributional characteristics: Why psychologists should be interested in qualitative departures from the normal distribution. Psychophysiology, 37, 21– 28. Schenk, G. K., Filler, W., Ranft, W., Zerbin, D., Dokk, D., Haverkorn, J., et al. (1982). Factor-analytical reliability studies with quantitative EEG parameters. In W. M. Herrmann (Ed.), Electroencephalography in drug research (pp. 209–225). Stuttgart: Fischer. Semlitsch, H. V., Anderer, P., Schuster, P., & Presslich, O. (1986). A solution for reliable and valid reduction of ocular artifacts, applied to the P300 ERP. Psychophysiology, 23, 695–703. Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics. New York: Harper Collins. Tenke, C. E., & Kayser, J. (2005). Reference-free quantification of EEG spectra: Combining current source density (CSD) and frequency principal components analysis (fPCA). Clinical Neurophysiology, 116, 2826–2846. Valentino, D. A., Arruda, J. E., & Gold, S. M. (1993). Comparison of QEEG and response accuracy in good vs poorer performers during a vigilance task. International Journal of Psychophysiology, 15, 123– 133. Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment: Honoring Douglas N. Jackson at seventy (pp. 41– 71). New York: Kluwer Academic/Plenum Publishers. Velicer, W. F., & Fava, J. L. (1987). An evaluation of the effects of variables sampling on component, image, and factor analysis. Multivariate Behavioral Research, 22, 193–210. Velicer, W. F., & Fava, J. L. (1998). Effects of variable and subject sampling on factor pattern recovery. Psychological Methods, 3, 231– 251. Velicer, W. F., Peacock, A. C., & Jackson, D. N. (1982). A comparison of component and factor pattern: A Monte Carlo approach. Multivariate Behavioral Research, 17, 371–388. Weiler, M. D. (1993). A measurement model for spectral analyzed EEG. Doctoral dissertation, University of Rhode Island. Dissertation Abstracts International, 54, 1715.
(Received June 24, 2009; Accepted April 13, 2010)
376
J.E. Arruda et al.
APPENDIX i. Following lists the steps used to generate one data set:
1. Evaluate univariate kurtosis and skewness of each variable (58 variables in total) 2. If kurtosis and skewness are within the range of the requirements for normal distributions (i.e. normality assumption is likely to be valid), then 3. Calculate the mean of each variable in the population data to form a vector of means, l 4. Calculate the covariance matrix of the population data, S 5. Simulate a data set from a multivariate normal distribution with mean l and covariance matrix R.
ii. If kurtosis and skewness are outside the range of the requirements for normal distributions (i.e. normality assumption is likely to be violated):
1. Perform log-transformation with base a on the population data. The base a is selected such that kurtosis and skewness of the transformed population data are both within the range for normal distributions.
2. Calculate the mean of each variable in the transformed population data to form a vector of means, denoted as l’. 3. Calculate the covariance matrix from the transformed population data, denoted as R’. 4. Simulate a data set from a multivariate normal distribution with mean l’ and covariance matrix R’. We denote the data as Y. 5. Transformation Y back to the original scale using the formula aY.
iii. The kurtosis and skewness of the simulated data were calculated and compare with the kurtosis and skewness of the original data to evaluate the similarity of the distribution patterns between the simulated data and the true data. iv. The above scenarios were applied to all the frequency bands considered in our study. Following the above scenario, for each frequency band, we generated 100 data sets such that each data set had 100 observations and each observation had 58 realizations of the 58 variables. All the simulated data were in original scale, i.e. un-transformed data.
Psychophysiology, 48 (2011), 377–384. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01073.x
Electrophysiological evidence for the hierarchical organization of auditory change detection in the human brain
SABINE GRIMM, CARLES ESCERA, LAVINIA SLABU, and JORDI COSTA-FAIDELLA Institute for Brain, Cognition and Behavior (IR3C), University of Barcelona, Barcelona, Spain
Abstract Auditory change detection has been associated with mismatch negativity (MMN), an event-related potential (ERP) occurring at 100–250 ms after the onset of an acoustic change. Yet, single-unit recordings in animals suggest much faster novelty-specific responses in the auditory system. To investigate change detection in a corresponding early time range in humans, we measured the Middle Latency Response (MLR) and MMN during a controlled frequency oddball paradigm. In addition to MMN, an early effect of change detection was observed at about 40 ms after change onset reflected in an enhancement of the Nb component of the MLR. Both MMN and the Nb effect were shown to be free from confounding influences such as differences in refractoriness. This finding implies that early change detection processes exist in humans upstream of MMN generation, which supports the emerging view of a hierarchical organization of change detection expanding along multiple levels of the auditory pathway. Descriptors: Middle-latency response, Auditory processing, Deviance detection, MMN
Alain, Otten, Ritter, & Achim, 2000). Thereby, an MMN response is generated at 100–250 ms from deviance onset by sources located bilaterally in the supratemporal brain region in the vicinity of the auditory cortex (Alho, 1995; Maess, Jacobsen, Schro¨ger, & Friederici, 2007; Na¨a¨ta¨nen & Alho, 1995). Additional prefrontal contributions have been reported in several studies (Deouell, 2007). The underlying processes involve the modeling and storage of the acoustic regularities and cannot merely be explained by different states of refractoriness of feature-specific neurons responding to the standard or deviant (Na¨a¨ta¨nen, Jacobsen, & Winkler, 2005; see also Ja¨a¨skela¨inen, Ahveninen, Bonmassar, Dale, Ilmoniemi, et al., 2004). This has been shown by means of a controlled oddball paradigm for a variety of deviant types including location (Schro¨ger & Wolff, 1996), frequency (Jacobsen & Schro¨ger, 2001), and duration (Jacobsen & Schro¨ger, 2003) deviants. In the controlled paradigm, the deviant stimulus from the oddball block is compared to a physically identical sound occurring with the same probability as the deviant in a context of different randomly presented equiprobable stimuli. Thus, the differential response is ensured not to be due to the differences in stimulus probability and associated differences in the state of refractoriness of neural populations, but is reflecting ‘‘true’’ deviance detection based on a regularity representation stored in auditory sensory memory. Nevertheless, our understanding of the neural mechanisms underlying auditory deviance detection is still fragmentary. A step forward in this direction has been recently provided by studies of single-unit recordings in anesthetized animals. Indeed, the majority of neurons of the cat’s primary auditory cortex
The ability to detect new events in the acoustic environment is vitally important, as they might call for a prompt adaptive response. This requires that regularities in the acoustic input are modeled and kept in memory, so that deviant or contextually new stimuli violating the regularity representations can be detected. Processes of deviance detection have traditionally been associated with a particular component, the mismatch negativity (MMN; Escera, 2007; Na¨a¨ta¨nen, 2007; Na¨a¨ta¨nen, Gaillard, & Ma¨ntysalo, 1978) of the human event-related potential (ERP). This brain response is usually obtained with the auditory oddball paradigm comparing activity elicited by a frequently repeated stimulus (standard) to that elicited by an interspersed rare stimulus containing a feature variation (deviant). In that way, MMN can be obtained for violations of simple feature rules, as for example in the case of frequency, location, or intensity deviants, but it is also elicited for sounds violating more complex regularities (e.g., phonetic contrasts, abstract regularities defining the relationship between sounds, etc.; for an overview, see Picton, This work was supported by the program Consolider-Ingenio 2010 (grant number CDS2007-00012), the National Program for Fundamental Research (grant number PSI2009-08063), a Juan de la Cierva grant to Sabine Grimm (grant number JCI-2009-04401) by the Spanish Ministry of Science and Innovation, and the ERANET NEURON project PANS (grant number EUI2009-04086). The authors thank Marc Recasens for his help during data acquisition. Address correspondence to: Carles Escera, PhD, Professor, Department of Psychiatry & Clinical Psychobiology, Faculty of Psychology, University of Barcelona, P. Vall d’Hebron 171, 08035 Barcelona, Catalonia, Spain. E-mail:
[email protected] 377
378
S. Grimm et al.
a
b
Figure 1. Experimental design. Stimuli were presented in three types of blocks: (a) oddball blocks; (b) reversed oddball blocks interchanging the roles of deviant and standard frequencies; and (c) control blocks containing equiprobable tones of five different frequencies. The squares symbolize single tones. Sound duration was 50 ms, the stimulus onset-to-onset interval was set to 293 ms.
c
(PAC) exhibit a property termed stimulus-specific adaptation (SSA), that is, they reduce significantly their discharge rate after a few repetitions of the standard tone, but show fast robust responses to novel1 stimuli that slightly differ in their feature properties from the standard (Ulanovsky, Las, Farkas, & Nelken, 2004; Ulanovsky, Las, & Nelken, 2003). Despite sharing similar characteristics with the human MMN, the early latency of these novelty-specific neural responses (circa 20 ms) suggests that they are not directly equivalent, but rather lie upstream of MMN generation (von der Behrens, Ba¨uerle, Ko¨ssl, & Gaese, 2009). Furthermore, very recent single-unit studies have shown that neurons in the inferior colliculus in the rat (Malmierca, Cristaudo, Pe´rez-Gonza´lez, & Covey, 2009; Pe´rez-Gonza´lez, Malmierca, & Covey, 2005) and in the barn owl (Reches & Gutfreund, 2008) and neurons in the medial geniculate body of the thalamus (Anderson, Christianson, & Linden, 2009; Antunes, Covey, & Malmierca, 2010) exhibit similar SSA to that found in the PAC of the cat, suggesting that deviance detection in the auditory system can be found even before the information reaches the auditory cortex. A comprehensive interpretation of these animal and human results is suggestive of two important aspects regarding the auditory system: (1) that deviance detection is a key principle expanding along the auditory pathway from the lower levels of the brainstem to high-order areas of the cerebral cortex; (2) that the generation of the MMN recorded from the human scalp is the consequence of a cascade of deviance detection processes at these different levels. Yet, a unified picture of these two lines of research, in humans and animals, is missing. In the present study, we aimed to test the hypothesis of a ‘‘pervasive auditory novelty system’’ by investigating processes of auditory change detection in humans on multiple time scales
1 Note that the term novel is used here in a more general meaning referring to contextually new or deviant information.
including the time range corresponding to the novelty responses observed in the auditory cortex of animals. Therefore, we measured, aside from the MMN component, an earlier portion of the ERP, the human Middle Latency Response (MLR) during a controlled oddball paradigm. The MLR is characterized by a sequence of waveforms in the range of 12–50 ms from sound onset, labeled as P0, Na, Pa, and Nb (Picton, Hillyard, Krausz, & Galambos, 1974), composed of activation in subcortical, and primary and secondary auditory cortices (Deiber, Ibanez, Fischer, Perrin, & Mauguiere, 1988; Liegeois-Chauvel, Musolino, Badier, Marquis, & Chauvel, 1994; Yvert, Crouzeix, Bertrand, Seither-Preisler, & Pantev, 2001; Yvert, Fischer, Guenot, Krolak-Salmon, Isnard, & Pernier, 2002). The MLR and MMN responses were measured to frequency deviants (800 Hz and 3730 Hz, in separate conditions) occurring in an oddball block, and the responses were compared to those elicited by physically identical stimuli when they had the role of a standard in a ‘‘reversed’’ oddball block, and when occurring equiprobably amongst four other low-probability tones (control condition; Figure 1). The set-up of the electroencephalogram (EEG) recordings was tailored to provide the possibility to extract the MLR components P0, Na, Pa, Nb, and the longlatency component MMN in parallel analyses. Our results revealed ‘‘true’’ deviance detection at a latency of 40 ms in humans, i.e., by the Nb waveform of the MLR, supporting the idea of a multistage comparison system for change detection along the auditory pathway in humans.
Methods Participants Twenty healthy, normal-hearing students (18–31 years, 11 female) participated in the experiment for payment (h6 per hour). All participants had normal hearing at both ears with a
Hierarchical organization of auditory change detection mean hearing threshold below 25 dB tested for the five frequencies used in the experiment. The experimental protocol was approved by the Ethical Committee of University of Barcelona, and was in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). Participants gave written informed consent before the experiment. Materials Auditory sequences were composed of pure sine wave sounds of 50 ms duration including a 5-ms rise and a 5-ms fall time. Sounds were presented binaurally via headphones at an intensity level of 50 dB above the individual hearing threshold as measured at the beginning of each experiment. The stimulus onset-to-onset interval was set to 293 ms. The oddball sequence contained a frequent standard sound occurring with a probability of .80 and a rare frequency deviant occurring randomly with a probability of .20. In one of two conditions, the standard frequency was 1,200 Hz and the deviant frequency was 800 Hz (low frequency condition). In the second high frequency condition, the standard frequency was 2,580 Hz and the deviant frequency, 3,730 Hz. Corresponding to each of the oddball blocks, a reversed sequence was introduced in which the roles of deviant and standard stimuli were switched. Additionally, a control condition was presented randomly intermixing tones of five different frequencies (800, 1,200, 1,780, 2,580, 3,730 Hz), each occurring with a probability of .20 from which the two extreme stimuli were taken as control tones for the low and high frequency condition, respectively (see Figure 1). This was done to preclude refractoriness confounds, i.e., the assumption being that the deviant stimulus will elicit a stronger response per se, as it occurs with a much smaller probability than the standard and, therefore, would find feature-specific neural populations in a less refractory state than those responding to the standard stimulus (Schro¨ger & Wolff, 1996). Procedure During the experiment, subjects were seated comfortably in an electrically shielded and sound-attenuated chamber. They were instructed to relax and to watch a silent movie with subtitles, ignoring the auditory stimulation. In total, 920 trials per deviant, standard, and control stimuli were delivered. The three conditions (oddball, reversed oddball, and control) were split into a total number of 16 blocks of approximately 5 min each, which were presented in random order. After each block, subjects had a short break allowing for movements, after every fourth block a 5-min break for rest was introduced. Electroencephalographic Recording The EEG was recorded continuously from 9 tin scalp electrodes referenced to an electrode placed on the tip of the nose. Electrodes were mounted according to the 10–20 system using an elastic cap (ECI Electro-Cap, Electro-Cap International, Inc., Eaton, OH) at the positions F3, F4, Fz, C3, C4, Cz, Pz, and the left and right mastoids. Additionally, eye movements were measured bipolarly by two electrodes placed above and below the right eye (vertical electrooculogram, VEOG) and two electrodes placed at the outer canthi of each eye (horizontal electrooculogram, HEOG). The electrode signals were amplified using a SynAmps amplifier (NeuroScan, Compumedics, Charlotte, NC), online bandpass-filtered from 0.05 to 500 Hz, and digitized with a sampling rate of 2500 Hz. Off-line data were re-referenced to the left mastoid.
379 EEG Analysis For the analysis in the long-latency range of the ERP, data was filtered off-line with a Kaiser-windowed sinc bandpass filter (beta 5 5.658) from 0.6 to 35 Hz. Epochs of 400 ms including a 100-ms pre-stimulus baseline were averaged separately for the deviant, standard, and control stimulus in the two frequency conditions. Any trial with an amplitude variation larger than 80 mV was excluded from further analysis. Mean amplitudes of MMN were extracted at the electrode Cz from a 30-ms time window around the grand-average peak latency ranging from 90 to 120 ms as derived from the difference waveforms between deviant and standard, and deviant and control stimuli, respectively. A repeated measures analysis of variance (ANOVA) including the factors Stimulus Type (deviant, standard, control) and Tone Frequency (low, high) was calculated on the mean amplitudes in the MMN time window. If appropriate, pairwise differences between single levels of factors were tested applying repeated measures contrasts. For the MLR analysis, data was filtered with a Kaiser-windowed sinc bandpass filter (beta 5 5.658) from 15 to 200 Hz. Epochs of 150 ms including a 50-ms pre-stimulus baseline were averaged. Any trial with an amplitude variation larger than 80 mV was excluded from further analysis. The components P0, Na, Pa, and Nb of the MLR were extracted for the three stimulus types (deviant, standard, controls) and the two frequency conditions. Individual peak latencies were derived from the largest peak in the time windows 7–17 ms (P0), 19–29 ms (Na), 26–36 ms (Pa), and 37–47 ms (Nb), respectively. As mean amplitudes are known to be more reliable than peak amplitudes (Picton, Bentin, Berg, Donchin, Hillyard, et al., 2000), analyses of the components’ amplitudes were based on the mean voltage measured in a 4-ms time window centered on the respective mean grand-average peak latency elicited by the deviant, standard, and control stimuli at the electrode Cz. For the high frequency stimuli, P0 peaked in the grand-average waveforms at 10 ms, Na at 23 ms, Pa at 28 ms, and Nb at 38 ms. For the low frequency stimuli, the respective grand-average peak latencies were at 12 ms (P0), 24 ms (Na), 32 ms (Pa), and 42 ms (Nb) after tone onset. For each component, a repeated measures ANOVA including the factors Stimulus Type (deviant, standard, control) and Tone Frequency (low, high) was calculated on MLR peak latencies and mean amplitudes. The Greenhouse-Geisser (G-G) correction was applied if the assumption of sphericity was violated. If appropriate, pairwise differences between single levels of Stimulus Type were tested applying repeated measures contrasts. A result was considered significant when po.05 using a twotailed analysis. Bonferroni correction was used to adjust p-values for all multiple pairwise contrasts as well as for testing on multiple MLR components. The effect size (partial eta squared, Z2p ) is reported in addition to F- and p-values.
Results ERPs recorded in 20 participants displayed the typical series of components in response to the standard, deviant, and control tones in the low (Figure 2a) and in the high (Figure 2b) frequency conditions. Focusing on the long-latency range of the ERPs, a prominent MMN was obtained peaking at about 105 ms after tone onset. In the MLR, the characteristic P0-Na-Pa-Nb complex was displayed in all experimental conditions (Figure 2, middle column). MLR peak latencies are given in Table 1. Table 2 shows the mean
380
S. Grimm et al.
a MLR
LLR −3
−3
µV
µV
STD DEV CON −3
−100
100
µV 2
−100
100
2
µV
−0.4
Nb*
2
−3
Na
−0.2
-50
50
ms
0.4
Pa
MLR
LLR −3
−3
µV
µV
STD DEV CON µV
−100
−100
100 2
−100
100 2
P0
−3
MMN*
µV
−100
0.2
b
100
100 −0.4
2
−0.2
2
µV Na
−3
Nb*
−50
50 0.2
100
P0
Pa
ms
−100
µV
MMN*
100 2
0.4 Figure 2. Grand-average evoked potentials of 20 subjects (a) for the low frequency tones and (b) for the high frequency tones. On the left of each panel, the auditory evoked potentials for deviant (grey solid), standard (black dashed), and control sounds (black solid) are shown including middle and long latency portions of the ERP applying a 0.6 Hz high-pass filter. In the middle column of each panel, the data filtered in the MLR range for the three stimulus conditions are presented. A statistical difference on the Nb component peaking at about 40 ms was observed as indicated by the asterisks in the zoom below. In the right column of each panel, data filtered in the long-latency range (LLR) of the ERP are shown. In the difference waveforms below, a clear MMN is present for the deviant and standard (black dashed) as well as for the deviant and control comparison (black solid).
amplitude values for each MLR component in the 4-ms latency windows around the grand-average peaks as given above. Long-Latency Components of the ERP First, data were analyzed with respect to the long-latency range of ERP contrasting evoked responses to standard, deviant, and
control stimuli for both the low and high frequency tones. In the time window around the MMN peak from 90 to 120 ms, deviant ERPs displayed a sharp negative potential, this being of less negative amplitude for control ERPs and even positive for standard ones. Amplitude differences for the three stimulus types were statistically significant in this latency window
Hierarchical organization of auditory change detection
381
Table 1. Mean of Individual Peak Latencies for the MLR Components P0, Na, Pa, and Nb P0 Freq 800 Hz Sta 13.35 Dev 13.05 Con 12.45 Freq 3730 Hz Sta 11.90 Dev 13.15 Con 12.05
Na
Pa
Nb
(0.49) 24.05 (0.33) 24.50 (0.49) 24.65
(0.44) 32.50 (0.48) 43.35 (0.29) 33.00 (0.36) 43.20 (0.40) 31.70 (0.48) 43.60
(0.45) (0.58) (0.55)
(0.66) 23.30 (0.60) 22.60 (0.59) 22.85
(0.51) 30.50 (0.69) 42.10 (0.47) 29.35 (0.44) 40.85 (0.51) 30.50 (0.59) 40.80
(0.70) (0.65) (0.67)
Note: Standard Errors of Mean are given in parentheses.
(F(2,38) 5 74.15, po.001, Z2p ¼ 0:80). The repeated measures ANOVA revealed no main effect of Tone Frequency (F(1,19) 5 3.51, p 5 .077, Z2p ¼ 0:16), but an interaction of Tone Frequency and Stimulus Type (F(2,38) 5 11.99, po.001, Z2p ¼ 0:39). Tone differences between the deviant and standard (repeated measures contrasts for low frequency: F(1,19) 5 124.11, corrected po.001, Z2p ¼ 0:87; for high frequency: F(1,19) 5 45.32, corrected po.001, Z2p ¼ 0:71) and somewhat smaller differences between deviant and control (repeated measures contrasts for low frequency: F(1,19) 5 49.71, corrected po.001, Z2p ¼ 0:72; for high frequency: F(1,19) 5 24.47, corrected po.001, Z2p ¼ 0:56) were present. The interaction resulted from the fact that, in the low frequency condition, control and deviant responses showed more negative mean amplitudes than in the high frequency condition, whereas mean amplitudes for standard ERP had more positive values than in the high frequency condition.
MLR Components of the ERP Second, we examined data with respect to the MLR of the ERP. Peak latencies and main amplitudes of the components P0, Na, Pa, Nb were contrasted for the three stimulus types (deviant, standard, control) and the two frequencies (low, high). Peaks elicited by the high frequency tones showed shorter latencies than the low frequency ones (see Table 1) as reflected in a main effect of Tone Frequency for the components Na (F(1,19) 5 9.81, corrected p 5 .022, Z2p ¼ 0:34), Pa (F(1,19) 5 17.01, corrected p 5 .002, Z2p ¼ 0:47), and Nb (F(1,19) 5 13.502, corrected p 5 .006, Z2p ¼ 0:42), but not for P0 (F(1,19) 5 1.83, corrected p 5 .77, Z2p ¼ 0:09). Neither latency differences between standard, deviant, and control stimuli, nor any interaction
Table 2. Mean Amplitudes of the MLR Components P0, Na, Pa, Nb Derived from a 4-ms Latency Windows Centered Around the Peak Latency in the Grand-Average Waveforms P0 Freq 800 Hz Sta 0.122 Dev 0.182 Con 0.171 Freq 3730 Hz Sta 0.027 Dev 0.125 Con 0.082
Na
Pa
Nb
(0.042) " 0.198 (0.038) 0.237 (0.045) " 0.203 (0.055) (0.040) " 0.178 (0.038) 0.292 (0.061) " 0.285 (0.051) (0.047) " 0.195 (0.043) 0.233 (0.062) " 0.253 (0.062) (0.044) " 0.079 (0.037) 0.067 (0.043) " 0.083 (0.046) (0.024) " 0.069 (0.033) 0.101 (0.039) " 0.222 (0.046) (0.036) " 0.143 (0.029) 0.042 (0.032) " 0.149 (0.036)
Note: Standard Errors of Mean are given in parentheses.
between Stimulus Type and Tone Frequency were observed (F’s o3.5, corrected p’s 4.17, Z2p < 0:16). A main effect of Tone Frequency on component’s mean amplitudes, in terms of larger mean amplitudes elicited by the low compared to the high frequency tones, was found for the components Na (F(1,19) 5 8.11, corrected p 5 .040, Z2p ¼ 0:30), Pa (F(1,19) 5 22.30, corrected p 5 .003, Z2p ¼ 0:54), and Nb (F(1,19) 5 11.41, corrected p 5 .012, Z2p ¼ 0:38) but not for P0 (F(1,19) 5 4.10, corrected p 5 .23, Z2p ¼ 0:18; see Table 2). Modulations through novelty, in terms of a main effect of Stimulus Type on the components’ mean amplitude, were observed for the time window of the Nb component (F(2,38) 5 6.84, corrected p 5 .028 (G-G), Z2p ¼ 0:27), with this main effect of Stimulus Type being absent for the earlier components (P0: F(2,38) 5 2.31, corrected p 5 .512 (G-G), Z2p ¼ 0:11, Na: F(2,38) 5 1.11, corrected p 5 1.000, Z2p ¼ 0:06, Pa: F(2,38) 5 1.35, corrected p 5 1.000, Z2p ¼ 0:07; cf. Figure 3). Post-hoc repeated measures contrasts confirmed differences in the Nb latency window between deviant and standard responses (F(1,19) 5 9.98, corrected p 5 .010, Z2p ¼ 0:34) and between deviant and control responses (F(1,19) 5 6.57, corrected p 5 .038, Z2p ¼ 0:26). Amplitude modulations through Stimulus Type did not interact with the effects of Tone Frequency for any of the components (Fso1.1, corrected p’s 5 1.000, Z2p < ; 0:06).
Discussion The results of the present study have revealed that ‘‘true’’ auditory deviance detection can take place in humans as early as 40 ms after the onset of a deviant feature presented in an otherwise repetitive sequence of standard stimuli. This was supported by the fact that a characteristic amplitude modulation of the MLR component Nb (peaking at about 38–42 ms) was obtained depending on the deviant status of a stimulus. The Nb response was larger for a stimulus when it occurred as a frequency deviant than the response elicited by the same stimulus when it occurred in the role of a standard or a control tone. In the later portion of the ERP, we additionally obtained a clear MMN peaking at about 105 ms after sound onset when contrasting deviant against standard and control responses. The MMN is the typical marker of deviance detection known to reflect a memory-based process of comparing incoming stimuli with an internal model derived from the regularities in the previous stimulation. A respective control condition is required (Schro¨ger & Wolff, 1996) to ensure that any deviance-related modulation obtained in an oddball paradigm is truly memory-based and not merely reflecting differences in the response strength of featurespecific neural populations (that are more refractory in case of a repeatedly presented standard frequency than in the case of a rarely presented deviant one). Of similar importance is the control for physical stimulus properties. Particularly the MLR is a sequence of components whose latencies and amplitudes are systematically influenced by physical stimulus characteristics (Picton et al., 1974); as reflected in the latency and amplitude differences for the MLR components Pa, Na, and Nb between low and high frequency tones in the present study, which is congruent with previous reports in the literature (Kraus & McGee, 1988; Thornton, Heneghan, James, & Jones, 1984). Taking into account these points, the MMN and the enhancement of Nb obtained in the present study are due to the informational status carried by a stimulus in its respective
382
S. Grimm et al.
Figure 3. Bar charts showing mean amplitudes of the MLR components (left side) and MMN (right side). Each chart plots the mean values and standard errors of mean for the three stimulus types (standard in light grey, deviant in grey, and control in dark grey) for the 800 Hz and 3730 Hz frequencies. Note that, in accordance with Figure 2, positive values are pointing downwards.
context and thus reflect true deviance detection rather than confounds by stimulus properties or refractoriness. Thus, it can be concluded that, besides the typical MMN component, the middle-latency portion of the ERP (which has so far barely been analyzed in studies applying the oddball paradigm) is also sensitive to stimulus novelty. This is in agreement with a concurrent study of our group (Slabu, Escera, Grimm, & Costa-Faidella, in press) and a few earlier results challenging the long-held belief that early auditory processing reflected in the MLR solely depends on the physical properties of an incoming stimulus. Sonnadara, Alain, and Trainor (2006) reported an enhanced Na component of the MLR peaking at 25 ms after sound onset for location deviants compared to standard click sounds, thus indicating an early effect of stimulus rareness. Yet, the lack of a respective control condition does not ascertain conclusively whether those results can be attributed to ‘‘true’’ deviance detection. Furthermore, modulations of MLR amplitudes by sensory gating (Mu¨ller, Keil, Kissler, & Gruber, 2001), self-initiation of a stimulus (Baess, Widmann, Roye, Schro¨ger, & Jacobsen, 2009), task requirements (Woldorff & Hillyard, 1991) and even by sound segregation processes (Dyson & Alain, 2004) have been shown, altogether underlining the complex nature of auditory processing already in its initial phase. From animal studies, we have indication that deviance detection can arise already at these early steps of auditory processing. For the first time, Ulanovsky et al. (2003, 2004) comprehensively described the activity of novelty neurons in the primary auditory cortex (A1) of the cat. These neural responses share a variety of properties with the MMN, for which they have
been regarded as its single-neuron correlate. Both the firing of novelty neurons and MMN are pre-attentive responses whose magnitude is inversely related with the deviant probability and positively related with the degree of deviance; both show already local sequence effects, their latencies are similarly influenced by deviant probability, and both responses are localized to the auditory cortex (Nelken & Ulanovsky, 2007). On the other hand, however, there are remarkable differences in timing between the firing onset of novelty units (at about 20 ms from stimulus onset; Pe´rez-Gonza´lez et al., 2005; Ulanovsky et al., 2004) and the peak latency of the MMN, which is contradicting the view that the first directly accounts for the latter (von der Behrens et al., 2009). Therefore the activity of novelty neurons has been interpreted as a change detection process in PAC that lies upstream of later MMN generation. The modulation of Nb by stimulus deviance found here exemplifies that those ‘‘upstream’’ activities also exist in the human auditory system and that they can be identified in the ERP with a respectively tailored set-up. It remains open whether the deviance-related Nb enhancement is more directly linked to the activity of novelty neurons in PAC. At least the two share partly similar origins as the transition of components Pa to Nb is supposed to be generated by cortico-cortical connections mediating auditory information from PAC to the superior temporal gyrus (STG; Yvert et al., 2002). Yet, only the simultaneous use of the different techniques in future studies might permit one to disambiguate the temporal relationship between single-cell firing and the potentials measured over larger auditory fields.
Hierarchical organization of auditory change detection
383
Eventually, the present study confirms that deviance detection is implemented on multiple levels during auditory processing in the human brain. In order to integrate these results into a conceptual framework, we here propose that deviance detection and the underlying processes of modeling invariant input are a pervasive property of the auditory system, expanding from lower levels along the auditory pathway to high-order areas of the cerebral cortex. This property allows us to react quickly to new events in our environment and crucially shapes our perception by sharpening its sensitivity to changes in incoming information. Within this framework, it can be assumed that the generation of MMN is a consequence of a cascade of deviance detection processes occurring at hierarchically lower levels. Accordingly, we can hypothesize that this function is organized in a hierarchical manner, so that deviance-related responses to simple-feature changes as used here are detected at the lower levels of the novelty system’s hierarchy, whereas more complex levels of regularity will be encoded in higher levels and thus in the latency range of MMN only. Alternatively, one could speculate about more profound differences in the
functional significance of the two levels of deviance processing, with the earlier possibly being related to a mechanism of auditory predictions (compare Bendixen, Schro¨ger, & Winkler 2009; Winkler, Denham, & Nelken, 2009), and the later level being related to regularity updating, which has been proposed previously as one of the functional roles of MMN (Winkler, Karmos, & Na¨a¨ta¨nen, 1996; Winkler, 2007). Without question, future studies are needed to clarify the functional significance of the different levels of deviance processing. To summarize, we report here a new electrophysiological marker of auditory deviance detection that indicates the auditory system’s rapid response to rare and unexpected sounds. This response very much resembles in origin and latency the recently described responses of so-called novelty neurons found in the cat’s auditory cortex and complements our picture of the functional organization of the auditory system. The presence of different markers of deviant processing in two time ranges of the ERP in the present study strongly supports the idea of a hierarchically organized system serving auditory deviance detection.
REFERENCES Alho, K. (1995). Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes. Ear and Hearing, 16, 38–51. Anderson, L. A., Christianson, G. B., & Linden, J. F. (2009). Stimulusspecific adaptation occurs in the auditory thalamus. The Journal of Neuroscience, 29, 7359–7363. Antunes, F. M., Covey, E., & Malmierca, S. M. (2010). Is there stimulusspecific adaptation in the auditory thalamus? In E. A. Lopez-Poveda, A. R. Palmer, & R. Meddis (Eds.), The neurophysiological bases of auditory perception. New York: Springer. Baess, P., Widmann, A., Roye, A., Schro¨ger, E., & Jacobsen, T. (2009). Attenuated human auditory middle latency response and evoked 40-hz response to self-initiated sounds. The European Journal of Neuroscience, 29, 1514–1521. Bendixen, A., Schro¨ger, E., & Winkler, I. (2009). I heard that coming: Event-related potential evidence for stimulus-driven prediction in the auditory system. The Journal of Neuroscience, 29, 8447–8451. Deiber, M. P., Ibanez, V., Fischer, C., Perrin, F., & Mauguiere, F. (1988). Sequential mapping favours the hypothesis of distinct generators for Na and Pa middle latency auditory evoked potentials. Electroencephalography and Clinical Neurophysiology, 71, 187–197. Deouell, L. Y. (2007). The frontal generator of the mismatch negativity revisited. Journal of Psychophysiology, 21, 188–203. Dyson, B. J., & Alain, C. (2004). Representation of concurrent acoustic objects in primary auditory cortex. The Journal of the Acoustical Society of America, 115, 280–288. Escera, C. (2007). The mismatch negativity 30 years later: How far have we come? Journal of Psychophysiology, 21, 129–132. Ja¨a¨skela¨inen, I. P., Ahveninen, J., Bonmassar, G., Dale, A. M., Ilmoniemi, R. J., Levanen, S., et al. (2004). Human posterior auditory cortex gates novel sounds to consciousness. Proceedings of the National Academy of Sciences of the United States of America, 101, 6809–6814. Jacobsen, T., & Schro¨ger, E. (2001). Is there pre-attentive memory-based comparison of pitch? Psychophysiology, 38, 723–727. Jacobsen, T., & Schro¨ger, E. (2003). Measuring duration mismatch negativity. Clinical Neurophysiology, 114, 1133–1143. Kraus, N., & McGee, T. (1988). Color imaging of the human middle latency response. Ear and Hearing, 9, 159–167. Liegeois-Chauvel, C., Musolino, A., Badier, J. M., Marquis, P., & Chauvel, P. (1994). Evoked potentials recorded from the auditory cortex in man: Evaluation and topography of the middle latency components. Electroencephalography and Clinical Neurophysiology, 92, 204–214. Maess, B., Jacobsen, T., Schro¨ger, E., & Friederici, A. D. (2007). Localizing pre-attentive auditory memory-based comparison: Magnetic mismatch negativity to pitch change. NeuroImage, 37, 561–571.
Malmierca, M. S., Cristaudo, S., Pe´rez-Gonza´lez, D., & Covey, E. (2009). Stimulus-specific adaptation in the inferior colliculus of the anesthetized rat. The Journal of Neuroscience, 29, 5483–5493. Mu¨ller, M. M., Keil, A., Kissler, J., & Gruber, T. (2001). Suppression of the auditory middle-latency response and evoked gamma-band response in a paired-click paradigm. Experimental Brain Research, 136, 474–479. Na¨a¨ta¨nen, R. (2007). The mismatch negativity. Where is the big fish? Journal of Psychophysiology, 21, 133–137. Na¨a¨ta¨nen, R., & Alho, K. (1995). Generators of electrical and magnetic mismatch responses in humans. Brain Topography, 7, 315–320. Na¨a¨ta¨nen, R., Gaillard, A. W., & Ma¨ntysalo, S. (1978). Early selectiveattention effect on evoked potential reinterpreted. Acta Psychologica, 42, 313–329. Na¨a¨ta¨nen, R., Jacobsen, T., & Winkler, I. (2005). Memory-based or afferent processes in mismatch negativity (MMN): A review of the evidence. Psychophysiology, 42, 25–32. Nelken, I., & Ulanovsky, N. (2007). Mismatch negativity and stimulusspecific adaptation in animal models. Journal of Psychophysiology, 21, 214–223. Pe´rez-Gonza´lez, D., Malmierca, M. S., & Covey, E. (2005). Novelty detector neurons in the mammalian auditory midbrain. The European Journal of Neuroscience, 22, 2879–2885. Picton, T. W., Alain, C., Otten, L., Ritter, W., & Achim, A. (2000). Mismatch negativity: Different water in the same river. Audiology & Neuro-Otology, 5, 111–139. Picton, T. W., Bentin, S., Berg, P., Donchin, E., Hillyard, S. A., Johnson, R. Jr., et al. (2000). Guidelines for using human event-related potentials to study cognition: Recording standards and publication criteria. Psychophysiology, 37, 127–152. Picton, T. W., Hillyard, S. A., Krausz, H. I., & Galambos, R. (1974). Human auditory evoked potentials. I. Evaluation of components. Electroencephalography and Clinical Neurophysiology, 36, 179–190. Reches, A., & Gutfreund, Y. (2008). Stimulus-specific adaptations in the gaze control system of the barn owl. The Journal of Neuroscience, 28, 1523–1533. Schro¨ger, E., & Wolff, C. (1996). Mismatch response of the human brain to changes in sound location. NeuroReport, 7, 3005–3008. Slabu, L., Escera, C., Grimm, S., & Costa-Faidella, J. (in press). Early change detection in humans as revealed by auditory brainstem and middle-latency evoked potentials. European Journal of Neuroscience. doi: 10.1111/j.1460-9568.2010.07324.x. Sonnadara, R. R., Alain, C., & Trainor, L. J. (2006). Occasional changes in sound location enhance middle latency evoked responses. Brain Research, 1076, 187–192.
384 Thornton, C., Heneghan, C. P., James, M. F., & Jones, J. G. (1984). Effects of halothane or enflurane with controlled ventilation on auditory evoked potentials. British Journal of Anaesthesia, 56, 315–323. Ulanovsky, N., Las, L., Farkas, D., & Nelken, I. (2004). Multiple time scales of adaptation in auditory cortex neurons. The Journal of Neuroscience, 24, 10440–10453. Ulanovsky, N., Las, L., & Nelken, I. (2003). Processing of low-probability sounds by cortical neurons. Nature Neuroscience, 6, 391–398. von der Behrens, W., Ba¨uerle, P., Ko¨ssl, E. H., & Gaese, B. H. (2009). Correlating stimulus-specific adaptation of cortical neurons and local field potentials in the awake rat. The Journal of Neuroscience, 29, 13837–13849. Winkler, I. (2007). Interpreting the mismatch negativity (MMN). Journal of Psychophysiology, 21, 147–163. Winkler, I., Denham, S., & Nelken, I. (2009). Modeling the auditory scene: Predictive regularity representations and perceptual objects. Trends in Cognitive Sciences, 13, 532–540.
S. Grimm et al. Winkler, I., Karmos, G., & Na¨a¨ta¨nen, R. (1996). Adaptive modeling of the unattended acoustic environment reflected in the mismatch negativity event-related potential. Brain Research, 742, 239–252. Woldorff, M. G., & Hillyard, S. A. (1991). Modulation of early auditory processing during selective listening to rapidly presented tones. Electroencephalography and Clinical Neurophysiology, 79, 170–191. Yvert, B., Crouzeix, A., Bertrand, O., Seither-Preisler, A., & Pantev, C. (2001). Multiple supratemporal sources of magnetic and electric auditory evoked middle latency components in humans. Cerebral Cortex, 11, 411–423. Yvert, B., Fischer, C., Guenot, M., Krolak-Salmon, P., Isnard, J., & Pernier, J. (2002). Simultaneous intracerebral EEG recordings of early auditory thalamic and cortical activity in human. The European Journal of Neuroscience, 16, 1146–1150. (Received March 15, 2010; Accepted April 21, 2010)
Psychophysiology, 48 (2011), 385–392. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01064.x
Peripheral responses to attended and unattended angry prosody: A dichotic listening paradigm
TATJANA AUE,a,b CAROLINE CUNY,c DAVID SANDER,a,d and DIDIER GRANDJEANa,e a
Swiss Center for Affective Sciences, University of Geneva, Geneva, Switzerland Laboratory for Neurology and Imaging of Cognition, Department of Neurosciences, University of Geneva, Geneva, Switzerland c Grenoble Ecole de Management, Grenoble, France d Laboratory for the Study of Emotion Elicitation and Expression (E3 Lab), Department of Psychology, University of Geneva, Geneva, Switzerland e Neuroscience of Emotion and Affective Dynamics Lab (NEAD), Department of Psychology, University of Geneva, Geneva, Switzerland b
Abstract We investigated the effects of angry prosody, varying focus of attention, and laterality of presentation of angry prosody on peripheral nervous system activity. Participants paid attention to either their left or their right ear while performing a sex discrimination task on dichotically presented pseudo-words. These pseudo-words were characterized by either angry or neutral prosody and presented stereophonically (anger/neutral, neutral/anger, or neutral/neutral, for the left/ right ear, respectively). Reaction times and physiological responses (heart period, skin conductance, finger and forehead temperature) in this study were differentially sensitive to the effects of anger versus neutral prosody, varying focus of attention, and laterality of presentation of angry prosody. Descriptors: Emotion, Attention, Lateralization, Prosody, Anger, Dichotic listening, Peripheral nervous system, Heart period, Skin conductance, Forehead temperature, Finger temperature
tion is initially devoted to something else. Emotional stimulus material in the visual domain has been shown to capture attention (e.g., Bradley & Lang, 2000; Lipp & Waters, 2007; see Vuilleumier, 2005, for a review)Fwith and without the involvement of conscious or voluntary processesFand to initiate remarkable mobilization for action preparation (O¨hman & Mineka, 2001). For example, O¨hman, Esteves, and Soares (1995) demonstrated elevated electrodermal activity to reflect the preattentive processing of masked fear-conditioned faces. For the auditory domain, Grandjean, Sander, Lucas, Scherer, and Vuilleumier (2008) found that auditory extinction in right-hemisphere patients with left spatial neglect syndrome can be reduced when contralesional auditory stimuli are emotionally significant. Thus, emotional stimuli have been successfully demonstrated to capture attention (exogenous attention). In turn, the deployment of voluntary (or endogenous) attention has been repeatedly shown to modulate central nervous system activity during the processing of emotional facial expressions (e.g., Holmes, Vuilleumier, & Eimer, 2003; Pourtois, Grandjean, Sander, & Vuilleumier, 2004; Vuilleumier, Armony, Driver, & Dolan, 2001). However, few studies have explicitly examined the combined effects of voluntary attention and emotion in the auditory domain (e.g., Grandjean et al., 2005; Sander et al., 2005; Schirmer, Kotz, & Friederici, 2005). Moreover, to the best of our knowledge, no single study has investigated the effects of these manipulations on peripheral nervous system activity. Grandjean and collaborators (2005; see also Sander et al., 2005) investigated the neural correlates of attended versus unattended angry prosody in a dichotic listening paradigm. In this study, male and female voices were presented to the left and right
Although the effects of other persons’ facial emotional expressions on peripheral responding have been widely studied (e.g., Dimberg & Petterson, 2000), research investigating the effects of other persons’ vocal emotional expressions on peripheral responding has remained remarkably sparse (e.g., Mitchell, 2006). This is surprising because emotional prosody, defined as supra-segmental and segmental modulations of acoustical parameters of speech related to emotional processes (Grandjean, Ba¨nziger, & Scherer, 2006), plays a critical role in everyday life (Russell, Bachorowski, & Fernandez-Dols, 2003; Scherer, 1986, 2003). Moreover, several disorders such as schizophrenia and autism have been related to impaired perception and expression of emotional prosody (e.g., Bach et al., 2009; Hoekert, Kahn, Pijnenborg, & Aleman, 2007; Peppe´, McCann, Gibbon, O’Hare, & Rutherford, 2007), emphasizing its importance for social interactions. The study of angry prosody is of particular importance because angry prosody signals potential danger for individual well-being, even if face-to-face contact is prohibited. Orientation, rapid physiological adaptation, and response preparation for such threatening social situations have been prerequisites for survival and social adaptation, both phylogenetically and ontogenetically (O¨hman & Wiens, 2003; Panksepp, 1982; Plutchik, 1980). In threatening situations, it can be essential to automatically orient and prepare physiological responses that support the accomplishment of adaptive behavioral actions even if, or especially when, voluntary attenAddress correspondence to: Tatjana Aue, Swiss Center for Affective Sciences, University of Geneva, 7, rue des Battoirs, 1205 Geneva, Switzerland. E-mail:
[email protected] 385
386 ears of participants. These voice stimuli were characterized by either angry prosody or neutral prosody and presented stereophonically (anger/neutral, neutral/anger, or neutral/neutral, for the left/right ear, respectively). Participants were instructed to attend to either their left or their right ear and to denominate the sex of the voice presented to the attended ear. Thus, emotional prosody was not explicitly judged in the task to be performed (sex discrimination task). Voice-selective areas in both hemispheres (Belin, Zatorre, Lafaille, Ahad, & Pike, 2000), especially the bilateral superior temporal sulcus and the right amygdala, displayed greater activation for angry as compared with neutral prosody, irrespective of whether the ear receiving the angry prosody was attended to or not. Such an observation suggests once more that significant signals such as angry prosody can capture and direct attention. Furthermore, the medial portion of the orbitofrontal cortex, the cuneus in the medial occipital cortex, and the bilateral ventrolateral prefrontal cortex displayed greater activation when angry voice prosody was presented to the attended ear than when it was presented to the unattended ear (Sander et al., 2005; cf. data from the facial domain: Pourtois et al., 2004; Vuilleumier, 2002). Together, these results favor a model of multiple levels of processing of angry prosody, with areas sensitive to voluntary attention (as compared with areas insensitive to voluntary attention) reflecting more integrative stages of affective evaluation. Reaction times in Grandjean et al.’s (2005) and Sander et al.’s (2005) sex discrimination task were slowed when an angry voice was presented to the left ear, but only when this ear had to be attended to. The authors interpreted this as a sign for the superiority of the left ear/right hemisphere for the processing of auditory emotional stimuli (cf. Borod & Madigan, 2000; Ja¨ncke, Buchanan, Lutz, & Shah, 2001), which allows the processing of emotional prosody even if prosody is task irrelevant. In the literature, right-hemisphere superiority has repeatedly been reported for the processing of facial emotional expressions and also for prosodic emotional expressions (Dimberg & Petterson, 2000; Kucharska-Pietura, Phillips, Gernand, & David, 2003; Ross, Edmondson, Seibert, & Homan, 1988). Rodway and Schepman (2007) observed a right-hemisphere advantage for genuine emotional prosody (cf. Herrero & Hillix, 1990; Ley & Bryden, 1982; Schmitt, Hartje, & Willmes, 1997), but not for morphed emotional prosody (fundamental frequency of emotional utterances taken and imposed on initially neutral utterances) in a dichotic listening paradigm. They concluded that the ‘‘laterality effect may be a useful tool for the detection of fake emotions’’ (Rodway & Schepman, 2007, p. 31). Thus, for angry prosody processing, the right hemisphere might play a pivotal role in the detection of real threatFbut not necessarily a unique role. Processes related to fine temporal discrimination have been associated with the left hemisphere and are also important in emotional prosody processing (Schirmer & Kotz, 2006). The latter example shows that the hypothesis of a right-hemisphere advantage for the processing of emotional events has not remained unchallenged. Brosch, Grandjean, Sander, and Scherer (2008) stereophonically presented utterances with angry/neutral spatially lateralized prosody (with angry prosody being presented either to the left or the right space by means of a headrelated transfer function) or with neutral/neutral prosody to participants (cf. Grandjean et al., 2005) who were engaged in a visual dot probe paradigm. Shortly after the presentation of the utterances, the participants saw a dot appearing on a computer screen and had to say whether the dot had been displayed in their
T. Aue et al. right or left hemifield. Because angry prosody is supposed to trigger exogenous attention, valid trials were those in which the hemifield of the dot corresponded to the hemispace in which the angry prosody had been presented. Conversely, invalid trials were those in which the presentation sides of angry prosody and the dot diverged. Brosch et al. observed a facilitation effect (mirrored in a shortening of response times) of the valid cueing by angry prosody for the right hemifield only. Therefore, these data might indicate an advantage of the left hemisphere for the processing of angry prosody. Finally, unlateralized effects can be found in the literature as well (e.g., Caltagirone et al., 1989; Ethofer, Van de Ville, Scherer, & Vuilleumier, 2009; Kotz et al., 2003; Kowner, 1995). A prominent model in research on the hemispheric lateralization of emotion (cf. Davidson, Abercrombie, Nitschke, & Putnam, 1999; Harmon-Jones & Allen, 1998) suggests that emotions generally associated with approach tendencies, such as anger, rely on left-anterior-hemispheric processing, and emotions generally associated with withdrawal tendencies, such as fear, rely on right-anterior-hemispheric processing. Applying the model to the perception of angry prosody is, however, complicated because, depending on their own subjective coping potential and environmental standards, listeners may respond to the perceived threat by either approach or withdrawal tendencies. Together, the current state of research in the area demonstrates that the role of hemispheric lateralization in the processing of emotional, especially angry, prosody needs further investigation. Because research on emotional prosody for peripheral nervous system activity is virtually nonexistent, the aims of the current study were threefold. First, we investigated the general effects of angry versus neutral prosody on peripheral responding. Second, we examined whether variations in voluntary attention modulate the effects of angry prosody on peripheral responding. Third, we investigated a potential ear/hemisphere advantage for the processing of angry prosody and whether such an advantage is also reflected in peripheral nervous system activity. Our participants were instructed to perform a sex discrimination task for pseudo-words verbalized by male and female speakers in a dichotic listening paradigm (cf. Grandjean et al., 2005; Sander et al., 2005). The voice stimuli used in this sex discrimination task were characterized by either angry or neutral prosody and presented stereophonically (anger/neutral, neutral/ anger, or neutral/neutral, for the left/right ear, respectively). Specifically, participants attended to either their left or their right ear and decided on the sex of the voice presented to the attended ear. Thus, emotional prosody was literally irrelevant to the behavior task. Heart period, mean skin conductance, forehead temperature, and finger temperature were continuously measured during the task. The following questions and derived hypotheses were investigated: Question 1: How is angry prosody reflected in behavioral data and peripheral nervous system activity? We expected shorter reaction times in the sex discrimination task for the neutral/neutral as compared with the angry/neutral and neutral/angry prosody trials. This is because less attention should be directed to the less threatening neutral prosody. Effects of exogenous attention toward angry prosody should also be reflected in heart periods. Because bradycardia has been observed in previous studies investigating attentional processes toward threat (cf. Bradley & Lang, 2000), we expected increased heart periods for angry as compared with neutral utterances. Because both attentional processes and increased response mobilization have been linked
Peripheral responses to angry prosody to skin conductance (e.g., Bradley, Codispoti, Cuthbert, & Lang, 2001; Gomez, Stahel, & Danuser, 2004), a higher mean skin conductance was predicted when participants listened to angry prosody than when they listened to neutral prosody. As a result of rudimentary body preparation for potential fight, we further anticipated heightened finger temperature in response to angry versus neutral prosody (cf. Levenson, Ekman, & Friesen, 1990). Finally, a more elevated forehead temperature was expected for angry prosody than for neutral prosody on the basis of observations made by Zajonc and collaborators (McIntosh, Zajonc, Vig, & Emerick, 1997; Zajonc, Murphy, & McIntosh, 1993) for negative subjective feeling states. Question 2: Are the behavioral data and the peripheral efference of angry prosody modulated by the focus of voluntary attention? This question addressed whether our participants displayed particularly slow reaction times in the sex discrimination task when the ear receiving angry prosody was the focus of voluntary attention (cf. Grandjean et al., 2005; Sander et al., 2005). Similarly, we investigated whether the physiological effects of angry prosody varied as a function of focus of voluntary attention. Question 3: Is there a right- or left-hemisphere advantage for the processing of angry prosody (i.e., is it reflected in behavior and physiological responding)? Because results in the literature are contradictory regarding a potential hemispheric advantage for the processing of angry prosody, we had no a priori hypotheses for this question.
Method Participants Forty-two healthy female undergraduate students at the University of Geneva, aged between 19 and 34 years (M 5 22.7, SD 5 3.12) and without any history of audiological illness, took part in this study. They were all right-handed and recruited in an introductory psychology course. Participants were paid 15 Swiss francs each. Exclusion criteria for participation were (a) medical treatment, (b) pregnancy, (c) drug abuse, and (d) age below 18 or above 35 years. Stimuli The auditory stimuli were produced by actors and taken from a database previously acquired and analyzed by Banse and Scherer (1996). We used three different tokens of nonsense syllable sequences (pseudo-words: ‘‘goster,’’ ‘‘niuvenci,’’ and ‘‘figotleich’’) extracted from meaningless sentence-like utterances. These voices were previously judged to express anger or neutral prosody, as validated in earlier behavioral studies (Banse & Scherer, 1996), showing an average accuracy of 75% for anger recognition (see also Brosch et al., 2008). Male and female speakers were equally distributed across conditions. The stimuli were matched for duration (750 ms). The mean acoustic energy was also counterbalanced across stimuli to avoid loudness effects. Setting and Apparatus Participants sat comfortably in a reclining position. Their arms were placed on an armrest to prevent fatigue to the largest possible extent. The pseudo-words were presented over headphones. Physiological data acquisition was performed continuously with the Biopac TEL 100 Remote Monitoring System (Santa Barbara, CA). There were different settings for the electrocardiogram, temperature, and skin conductance channels (see section ‘‘Dependent Variables’’ for details). Signals were transferred from the experimental room to the MP 100 Acquisition Unit (16
387 bit A/D conversion) in the control room and stored on computer hard disk. A digital channel received inputs from the presentation computer and recorded on- and offset of presented pseudowords. Experimental control, such as sound presentation and computer synchronization, was performed by e-prime 1.1. A hidden camera (Sony EVI-D31) permitted the detection of larger body movements impinging on physiological responses. Procedure Participants were told that they were taking part in a study examining the effects of male and female voices on physiological responding. Upon participants’ arrival at the laboratory, the nature of the experiment was explained and written informed consent was obtained in accordance with the Helsinki Declaration of Human Rights (1991). After sensor placement, a 5-min relaxation period began, allowing the participants to become familiar with the experimental setting and to establish a physiological baseline. Voluntary attention (left vs. right ear) was manipulated orthogonally to emotional prosody in a dichotic listening paradigm, in which two stimuli were simultaneously presented, one to each ear (anger/neutral [AN], neutral/anger [NA], and neutral/ neutral [NN], on the left/right side). Every experimental trial consisted of one female and one male voice pronouncing the same pseudo-word. Participants were instructed to selectively attend to the voice presented to either the left or the right ear and to decide on the sex of the speaker uttering the pseudo-words in the attended ear. Participants revealed their decision by pressing one of two buttons of a button box. Sex-button contingencies were counterbalanced across participants. A total of 120 trials were presented to each participant. In one block (60 trials), the students focused on the voices presented to their right ear; in another block (60 trials), they attended to the voices presented to their left ear. The block sequence was counterbalanced across participants. The intertrial interval (ITI) was between 8 and 10 s (jittered presentation). Participants listened to the utterances (750 ms) and then gave their behavioral response while waiting for the next utterance. We chose this rather long ITI to account for the type of physiological responses investigated in this study (rather long latency signals). In a postexperiment interview, participants were asked about their physical and psychological wellbeing. None of the participants reported having been disturbed. Before leaving the laboratory, participants were debriefed. Dependent Variables Behavioral data. Reaction times in the sex discrimination task were measured for each trial and participant. Physiological data. The following physiological signals were recorded continuously with a sampling rate of 1000 Hz. Parameterization was performed with the program PPP 7.12 (2005; Extra Quality Measurement Systems, Frankfurt am Main, Germany). Heart period. Heart period (in seconds) was assessed by the use of Biopac pre-gelled disposable Ag/AgCl electrodes (10-mm sensor diameter). Electrodes were fixed according to Einthoven II, one below the right clavicle and another below the left lateral margin of the chest. Amplification was set to 500 and filters were set to 1 and 45 Hz. Mean skin conductance. Electrodermal activity was measured with a constant voltage of 0.5 V, using the SS 3A Biopac electrodermal response transducer filled with Biopac GEL 101
388 electrode paste (formulated with 0.5% saline in a neutral base). The transducer was placed at the volar surfaces of the medial phalanges of the index and ring fingers of the left hand. Amplification was set to 500 (corresponding to a sensitivity of 20 mS/V), and filters were set to DC and 10 Hz. The signal was smoothed by a moving average (length: ! 200 ms). Forehead and finger temperature. A Biopac temperature probe (SS 7) was fixed on the forehead to measure forehead temperature in degrees Fahrenheit. Finger temperature was measured with a Biopac fast response temperature probe (SS6) placed on the participants’ little finger. Amplification was set to 500 (corresponding to a sensitivity of 101F/V), and filters were set to DC and 10 Hz. The signal was smoothed by a moving average (length: ! 200 ms). Physiological responses during the 2 s before voice onset served as baseline and were subtracted from responses estimated for the 5 s following the pseudo-words (for skin conductance, the interval comprising 1–7 s following stimulus onset was considered). The resultant difference scores represented the change provoked by a particular combination of pseudo-words (AN, NA, or NN). Data Analysis Angry versus neutral prosody. In a first step, a paired t test contrasting angry versus neutral prosody was calculated for each variable (one-tailed testing was due to a priori hypotheses, addressing Question 1). Effects of attention focus and presentation laterality for angry prosody. In a second step, responses to neutral prosody (NN) trials attended to with the right (left) ear were subtracted from responses to angry prosody (AN and NA) trials attended to with the right (left) ear to reduce global sensitivity differences between the two ears. Then, an analysis of variance (ANOVA) with the withinparticipants factors attention focus (voluntary attention to left vs. right ear) and presentation laterality (anger presented to left vs. right ear; AN vs. NA) was calculated (addressing Questions 2 and 3). Reaction times, heart period, mean skin conductance, forehead temperature, and finger temperature were entered as dependent variables in all analyses. Results Behavioral Data Participants displayed a high level of accuracy in the sex discrimination task (Mcorrect 5 90.0%). The proportion of correct responses did not vary across the experimental conditions (w2 5 9.23, p 5 .10, Wilcoxon test). Incorrect trials and outliers (deviating more than 3 SD from the average individual reaction time across all experimental conditions; 2.5%) were eliminated from reaction times and substituted with a participant’s average reaction time for the respective condition. Angry versus neutral prosody. Consistent with our hypotheses, participants displayed longer reaction times for angry prosody than they did for neutral prosody, t(41) 5 2.57, p 5 .007 (one-tailed, Ms 5 729.20 ms and 703.19 ms, respectively). Effects of attention focus and presentation laterality for angry prosody. The ANOVA performed on reaction time data for angry prosody only (including the AN and NA presentations) failed to demonstrate a significant effect of attention focus,
T. Aue et al.
60.00
∆ Reaction Time in ms
40.00
Attention left right
20.00 0.00 AN
NA
Figure 1. Reaction time as a function of attention focus and presentation laterality. Error bars depict standard errors. Reaction times for neutral (NN) prosody attended to with the left (right) ear have been subtracted from reaction times for angry (AN and NA) prosody attended to with the left (right) ear to reduce global sensitivity differences between the two ears.
F(1,41) 5 0.21, n.s., partial Z2 5 .00, and presentation laterality, F(1,41) 5 0.48, n.s., partial Z2 5 .01. A trend was observed for their interaction, F(1,41) 5 2.93, p 5 .09, partial Z2 5 .07 (Figure 1). Contrary to earlier research, confidence intervals (CIs) for the interaction showed a slowing of reaction times in the unattended anger conditions as compared with the NN condition (recall that the NN reaction times had been subtracted before entering the AN and NA trials into the ANOVA; CIs: 8.20oAN, attention righto61.42; 0.93oNA, attention lefto69.53). Reaction times in the attended anger conditions were comparable to those in the NN condition (CIs: " 7.29oAN, attention lefto57.33; " 23.35oNA, attention righto41.32). Physiological Data Heart period data for one participant were excluded because of arrhythmia. In addition, trials in which there were incorrect responses in the sex discrimination task and outliers (deviating more than 3 SD from the average individual response across all experimental conditions; approximately 2.8% of all responses) were removed from all physiological responses and substituted with a participant’s average response for the respective condition. Angry versus neutral prosody. Consistent with our hypotheses, angry prosody in comparison with neutral prosody was associated with a higher mean skin conductance, t(41) 5 2.68, p 5 .005 (Ms 5 0.02 mrho and 0.01 mrho, respectively) and a higher forehead temperature, t(41) 5 2.35, p 5 .01 (Ms 5 0.0081F and 0.0071F). Contrasts for heart period, t(40) 5 1.34, n.s. (Ms 5 " 0.007 s and " 0.009 s), and finger temperature, t(41) 5 1.16, n.s. (Ms 5 0.0041F and 0.0021F), failed to reach significance. Effects of attention focus and presentation laterality for angry prosody. Table 1 displays the results of the ANOVAs performed for the four physiological measures. Longer heart periods were observed when angry prosody was presented to the right ear than when it was presented to the left ear, irrespective of focus of attention (Figure 2). Moreover, CIs revealed that the AN trials did not significantly differ from the NN trials (which had been subtracted before we entered the angry-prosody trials into the ANOVA; CI: " 0.004oANo0.003). By contrast, the NA trials did differ significantly from the NN trials (CI: 0.001oNAo0.007). Our heart period data thus are not supportive of the hypothesis of a right-hemisphere advantage for the processing of angry prosody.
Peripheral responses to angry prosody
389 ! 0.0055oAN, attention lefto0.0065; ! 0.0029oAN, attention righto0.0074). The ANOVA for forehead temperature did not reveal any significant effect.
Table 1. Effects of Attention Focus (Attention) and Laterality of Presentation (Presentation) for Angry Prosody Variable Heart period Mean skin conductance Finger temperature Forehead temperature
Effect
df
F
p
Partial Z2
Attention Presentation Attention " Presentation Attention Presentation Attention " Presentation Attention Presentation Attention " Presentation Attention Presentation Attention " Presentation
1.40 1.40 1.40 1.41 1.41 1.41 1.41 1.41 1.41 1.41 1.41 1.41
0.00 4.68 1.07 0.00 0.54 6.73 0.17 3.19 5.09 2.43 0.81 0.21
n.s. .04 n.s. n.s. n.s. .01 n.s. .08 .03 n.s. n.s. n.s.
.00 .10 .03 .00 .01 .14 .00 .07 .11 .06 .02 .01
Discussion
Mean skin conductance varied as a combined function of attention focus and presentation laterality of angry prosody. Figure 2 reveals that the attended angry prosody conditions were associated with a higher mean skin conductance than were the unattended angry prosody conditions. However, only attended anger presented to the left ear was significantly different from the neutral (NN) utterances (CIs: 0.001oAN, attention lefto0.032; ! 0.001oNA, attention righto0.024;–0.006oAN, attention righto0.027; ! 0.009oNA, attention lefto0.015). The marginally significant main effect of presentation laterality for finger temperature was qualified by the significant interaction of Attention Focus " Presentation Laterality. Our participants were characterized by a high finger temperature when they did not attend to angry prosody presented to their right ear. CIs revealed that only angry prosody presented to the right ear that was not attended to (CI: 0.0020;oNA, attention lefto0.0129) significantly differed from the NN utterances (all other CIs: ! 0.0033oNA, attention righto0.0080;
0.008
∆ Heart period in s
Effects of Angry Versus Neutral Prosody In the present study, participants displayed a general slowing of their responses in the sex discrimination task when they were confronted with an angry as compared with a neutral prosodic utterance. This finding is in line with the idea of increased exogenous attention toward the more survival-relevant angry prosody. It has been shown that emotional activation (e.g., evoked by emotional stimuli and emotional music) can modulate the excitability of the primary motor cortex, particularly so in the case of negative emotional reactions (Baumgartner, Willi, & Ja¨ncke, 2007). Therefore, the longer reaction times for angry as compared with neutral prosody in our study may have been a direct consequence of such changed motor cortex activation. The physiological data also speak to increased exogenous attention and mobilization when participants are confronted with angry as compared with neutral prosody, which was expressed in increases in mean skin conductance and forehead temperature. These results are in accordance with earlier studies on anger in which there were a variety of different experimental tasks and stimuli other than prosody. For instance, Levenson et al. (1990) reported heightened electrodermal activity, and Zajonc and collaborators (McIntosh et al., 1997; Zajonc et al., 1993) identified increased forehead temperature as a specific feature of negative emotions such as fear and anger. It might be argued, however, that we cannot be sure that differences between angry and neutral prosody in our study really are a result of angry versus neutral prosody. They could, alternatively, be explained by the fact that, in the anger conditions, angry prosody was presented to only one ear while the other
0.03
∆ Mean skin conductance in mrho
0.02
0.004
0.01 0.000
0.00 –0.01
–0.004 AN
0.010
AN
NA
∆ Finger temperature in °F
0.004
NA
∆ Forehead temperature in °F Attention
0.005
0.002
0.000
0.000
–0.005
left right
–0.002 AN
NA
AN
NA
Figure 2. Physiological changes as a function of attention focus and presentation laterality. Error bars depict standard errors. Responses to neutral (NN) prosody attended to with the left (right) ear have been subtracted from responses to angry (AN and NA) prosody attended to with the left (right) ear to reduce global sensitivity differences between the two ears.
390 received neutral prosody, whereas the neutral condition consisted of neutral prosody being presented to both ears. The angry conditions, therefore, may have been characterized by higher ambiguity or conflict. Ambiguity versus emotion effects can be distinguished by the inclusion of an AA condition in future studies. Given that our results are in line with earlier studies on emotion, we nonetheless think that ambiguity cannot be held responsible for the entire effect size that we obtained for angry versus neutral prosody. For instance, if the AN and NA combinations really had only provoked greater ambiguity than the NN combinations, the Brosch et al. (2008) cross-modal study should have found a general slowing of reaction times for the dot detection in the AN and NA voice conditions in comparison with the NN condition (or no difference at all). However, that dot detection was facilitated in exactly those cases in which the dot location corresponded to the space region where the angry prosody had been presented before speaks to the attraction of attention to angry prosody. Ambiguity in AN and NA combinations should have counteracted such an effect because one would expect attention to oscillate between left and right regions of the space. Importantly, the results discussed in the following section are unchallenged by a potential confound of ambiguity because only the supposedly ambiguous conditions AN and NA were compared. Modulation of Responses to Angry Prosody by Varying Attention Focus and Presentation Laterality We found a trend for a slowing of reaction times when angry prosody was unattended (Figure 1). This trend conflicts with our expectations and with the results in a previous study (Grandjean et al., 2005; Sander et al., 2005) that observed longer reaction times when angry prosody presented to the left ear was attended to. The discrepancy in behavioral responses between the two studies can possibly be attributed to differences in ITIs, (approximately 9 s in the current study and 5 s in the Grandjean et al., 2005, and Sander et al., 2005, study) and needs further investigation. Because in the current study there was only a trend for the interaction Attention Focus ! Presentation Laterality, we will abstain from overinterpreting this effect and the discrepancy with previous studies. Both mean skin conductance and finger temperature responses to angry prosody varied as a function of attention focus. Higher skin conductance was observed for attended as compared with unattended angry prosody. These data conform to activity observed in the medial portion of the orbitofrontal cortex in the Sander et al. (2005) study with the same stimulus material. The authors of the latter study found higher activation in this area when angry prosody was attended to than when it was not, irrespective of laterality of presentation. Such an observation strengthens, although indirectly, the idea of a strong link between skin conductance and the medial orbitofrontal cortex (OFC), as suggested by lesion studies (Bechara, Damasio, & Damasio, 2000). Finger temperature, in contrast to mean skin conductance, was further sensitive to laterality of presentation of angry prosody. We observed an increase in finger temperature when angry prosody was presented to the right ear, but only when this ear was outside the focus of voluntary attention. At first glance, this seems surprising, because one would expect the greatest change in finger temperature when angry prosody is in the center of attention. However, it is well possible that finger temperature changes resulting from peripheral vasodilatation in the service of fight preparation (Levenson et al., 1990) are most pronounced
T. Aue et al. when something is perceived outside the focus of endogenous attention. Voluntary attention to the angry utterances may well have initiated adaptation processes in order to modify such upcoming response preparation according to existent norms. Societal behavior standards often request the suppression of aggression. Voluntary attention could thus activate control mechanisms with the goal of down-regulating and counteracting automatic response tendencies, as possibly reflected in the unattended condition. An alternative explanation could be that angry prosody, when voluntarily attended to, is perceived as more threatening, consequently changing an initial fight tendency into a withdrawal tendency in some participants. Withdrawal motivation has been associated with decreases in finger temperature (Levenson et al., 1990). Because research on the psychological significance of finger temperature is relatively sparse, each interpretation remains premature and the effect needs further investigation. However, the fact that the effect was restricted to the right ear questions the hypothesis of a general right-hemisphere advantage for the processing of angry prosody. Heart period data also contradict the view of a general righthemisphere advantage for the processing of angry prosody because longer heart periods were observed when angry prosody was presented to the right ear than when it was presented to the left ear. Our heart period data can thus be linked to research conducted by Harmon-Jones and collaborators (e.g., HarmonJones & Allen, 1998), who associate the left frontal cortex with anger and associated response tendencies. Our results for heart period are further in line with the observed left-hemisphere advantage reported by Brosch et al. (2008). These left-lateralized effects may also be related to the linguistic structure of the pseudo-words. Indeed, the pseudo-words are linguistically structured exactly as usual semantic language but without the semantic dimension. Such linguistic structure might have induced high temporal resolution auditory processes more related to the left than to the right hemisphere (see Schirmer & Kotz, 2003, for a review). Interestingly, a left-hemisphere advantage was not irrevocably reflected in all dependent variables. Skin conductance and forehead temperature did not show laterality effects at all (cf. Ethofer, Van de Ville, et al., 2009). This is not surprising because other work on emotion has also demonstrated that different somatovisceral variables may code different components of an emotional stimulus or an emotional episode (e.g., Stemmler, Aue, & Wacker, 2007). Similarly, earlier studies (Grandjean et al., 2005; Sander et al., 2005) revealed that some brain areas such as the amygdala and the superior temporal sulcus respond to angry prosody irrespective of voluntary attention, whereas the effect of angry prosody on activity in other areas such as the orbitofrontal cortex was modulated by focus of voluntary attention. Consistent with these observations, the physiological responses studied here were differentially sensitive to effects of angry versus neutral prosody, attentional modulations for the processing of angry prosody, and laterality of presentation of angry prosody. Forehead temperature, for instance, was sensitive to the emotional content of the pseudo-words, but was insensitive toward the attention and laterality of presentation manipulations for angry prosody. Heart period, in contrast, was sensitive to the laterality of angry prosody presentation only. Skin conductance was modulated by angry versus neutral prosody and endogenous versus exogenous attentional processes to angry prosody, and, finally, finger temperature was modulated by an interaction of endogenous versus exogenous attentional processes and laterality of presentation.
Peripheral responses to angry prosody
391
Future Directions Our results do not allow for inferences, whether the demonstrated results are specific for angry voice prosody or whether they can also be generalized to cues associated with other emotions, particularly positive emotional prosodies. We also have no indication of whether our results in the auditory domain generalize to visual material such as facial expressions. Because of the necessity of a sufficiently high number of trials per condition, however, we were unable to test all of these questions in the current study. Future studies could address these questions by substituting neutral prosody with happy prosody or by substituting auditory with visual material. Given that sex differences have been demonstrated in previous studies on the perception of emotional prosody (e.g., Schirmer et al., 2005), it also remains to be investigated whether the effects observed in our study generalize to male participants. Participants in our study performed a sex discrimination task and were not explicitly asked to judge emotional prosody. It is possible that the results would be different if individuals were asked to tell the type of emotional prosody presented to either the left or the right ear. Modulation of behavioral and brain responses to emotional prosody by task instructions has been investigated (e.g., Ethofer, Kreifelts, et al., 2009; Grimshaw, 1998; Kitayama & Ishii, 2002; Schirmer & Kotz, 2003). Mitchell, Elliott, Barry, Cruttenden, and Woodruff (2003), for example, demonstrated that the right middle temporal gyrus was more strongly activated when the emotional content rather than the semantic content of vocal stimuli was attended to (cf. Ethofer et al., 2006). Comparably, Bach et al. (2008) reported stronger amygdala activation during sex labeling than during emotion labeling, but stronger prefrontal and anterior cingulate activation during emotional labeling than during sex labeling. Because the amygdala can innervate the autonomic nervous system via the hypothalamus and the brain stem (e.g., LeDoux, 2000; Yang
et al., 2007), changing task instructions may well be able to alternate the physiological responses that we observed in the current study. It would thus be interesting to study the influence of varying task instructions on our observed effects. Factors such as emotional valence (Davidson et al., 1999; Rosadini & Rossi, 1967), duration, rhythm, and intensity of emotional prosody (Sidtis & Van Lancker-Sidtis, 2003; Van Lancker & Sidtis, 1992), or linguistic load (Mitchell & Ross, 2008; Ross, Thompson, & Yenkosky, 1997; Van Lancker, 1980) have further been suggested to moderate hemispheric lateralization (see Kotz, Meyer, & Paulmann, 2006, for an overview and further considerations). The systematic manipulation of these factors and the investigation of their effects on peripheral autonomic responses may be addressed in future studies.
Conclusions Our data show differential sensitivities of the examined physiological measures for effects of angry voice prosody, voluntary attention, and laterality of presentation of angry prosody. The observed pattern of responses suggests that angry prosody attracts attention and provokes marked behavioral and physiological changes. Importantly, voluntary attention does not seem to be a prerequisite for some bodily changes (such as heart period and finger and forehead temperature) to occur, thus proposing that rudimentary analysis of the stimulus and adaptive response preparation take place even when the source of threat is outside the focus of voluntary attention. Together, our physiological data also clearly demonstrate that large parts of the concrete pattern of an effectuated mobilization may depend on the focus of voluntary attention and/or on which ear receives the prosodic threat information. Finally, our results are not supportive of the idea of a general right hemisphere-advantage for the processing of angry prosody.
REFERENCES Bach, D. R., Grandjean, D., Sander, D., Herdener, M., Strik, W. K., & Seifritz, E. (2008). The effect of appraisal level on processing of emotional prosody in meaningless speech. NeuroImage, 42, 919–927. Bach, D. R., Herdener, M., Grandjean, D., Sander, D., Seifritz, E., & Strik, W. K. (2009). Altered lateralization of emotional prosody processing in schizophrenia. Schizophrenia Research, 110, 180–187. Banse, R., & Scherer, K. R. (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70, 614–636. Baumgartner, T., Willi, M., & Ja¨ncke, L. (2007). Modulation of corticospinal activity by strong emotions evoked by pictures and classical music: A transcranial magnetic stimulation study. NeuroReport, 18, 261–265. Bechara, A., Damasio, H., & Damasio, A. R. (2000). Emotion, decision making and the orbitofrontal cortex. Cerebral Cortex, 10, 295–307. Belin, P., Zatorre, R. J., Lafaille, P., Ahad, P., & Pike, B. (2000). Voiceselective areas in human auditory cortex. Nature, 403, 309–312. Borod, J. C., & Madigan, N. K. (2000). Neuropsychology of emotion and emotional disorders: An overview. In J. C. Borod (Ed.), The neuropsychology of emotion (pp. 3–30). New York: Oxford University Press. Bradley, M. M., Codispoti, M., Cuthbert, B. N., & Lang, P. J. (2001). Emotion and motivation I: Defensive and appetitive reactions in picture processing. Emotion, 1, 276–298. Bradley, M. M., & Lang, P. J. (2000). Affective reactions to acoustic stimuli. Psychophysiology, 37, 204–215. Brosch, T., Grandjean, D., Sander, D., & Scherer, K. R. (2008). Behold the voice of wrath: Cross-modal modulation of visual attention by anger prosody. Cognition, 106, 1497–1503.
Caltagirone, C., Ekman, P., Friesen, W., Gainotti, G., Mammucari, A., Pizzamiglio, L., et al. (1989). Posed emotional expression in unilateral brain damaged patients. Cortex, 25, 653–663. Davidson, R., Abercrombie, H., Nitschke, J., & Putnam, K. (1999). Regional brain function, emotion and disorders of emotion. Current Opinion in Neurobiology, 9, 228–234. Dimberg, U., & Petterson, M. (2000). Facial reactions to happy and angry facial expressions: Evidence for right hemisphere dominance. Psychophysiology, 37, 693–696. Ethofer, T., Anders, S., Erb, M., Herbert, C., Wiethoff, S., Kissler, J., et al. (2006). Cerebral pathways in processing of affective prosody: A dynamic causal modeling study. NeuroImage, 30, 580–587. Ethofer, T., Kreifelts, B., Wiethoff, S., Wolf, J., Grodd, W., Vuilleumier, P., et al. (2009). Differential influences of emotion, task, and novelty on brain regions underlying the processing of speech melody. Journal of Cognitive Neuroscience, 21, 1255–1268. Ethofer, T., Van de Ville, D., Scherer, K., & Vuilleumier, P. (2009). Decoding of emotional information in voice-sensitive cortices. Current Biology, 19, 1028–1033. Gomez, P., Stahel, W. A., & Danuser, B. (2004). Respiratory responses during affective picture viewing. Biological Psychology, 67, 359–373. Grandjean, D., Ba¨nziger, T., & Scherer, K. R. (2006). Intonation as an interface between language and affect. Progress in Brain Research, 156, 235–268. Grandjean, D., Sander, D., Lucas, N., Scherer, K. R., & Vuilleumier, P. (2008). Effects of emotional prosody on auditory extinction for voices in patients with spatial neglect. Neuropsychologia, 46, 487–496. Grandjean, D., Sander, D., Pourtois, G., Schwartz, S., Seghier, M. L., Scherer, K. R., et al. (2005). The voices of wrath: Brain responses to
392 angry prosody in meaningless speech. Nature Neuroscience, 8, 145–146. Grimshaw, G. M. (1998). Integration and interference in the cerebral hemispheres: Relations with hemispheric specialization. Brain and Cognition, 36, 108–127. Harmon-Jones, E., & Allen, J. B. B. (1998). Anger and frontal brain activity: EEG asymmetry consistent with approach motivation despite negative affective valence. Journal of Personality and Social Psychology, 74, 1210–1316. Herrero, J. V., & Hillix, W. A. (1990). Hemispheric performance in detecting prosody: A competitive dichotic listening task. Perceptual and Motor Skills, 71, 479–486. Hoekert, M., Kahn, R. S., Pijnenborg, M., & Aleman, A. (2007). Impaired recognition and expression of emotional prosody in schizophrenia: Review and meta-analysis. Schizophrenia Research, 96, 135–145. Holmes, A., Vuilleumier, P., & Eimer, M. (2003). The processing of emotional facial expression is gated by spatial attention: Evidence from event-related brain potentials. Brain Research. Cognitive Brain Research, 16, 174–184. Ja¨ncke, L., Buchanan, T. W., Lutz, K., & Shah, N. J. (2001). Focused and nonfocused attention in verbal and emotional dichotic listening: An fMRI study. Brain and Language, 78, 349–363. Kitayama, S., & Ishii, K. (2002). Word and voice: Spontaneous attention to emotional utterances in two languages. Cognition and Emotion, 16, 29–59. Kotz, S. A., Meyer, M., Alter, K., Besson, M., von Cramon, D. Y., & Friederici, A. D. (2003). On the lateralization of emotional prosody: An event-related functional MR investigation. Brain and Language, 86, 366–376. Kotz, S. A., Meyer, M., & Paulmann, S. (2006). Lateralization of emotional prosody in the brain: An overview and synopsis on the impact of study design. Progress in Brain Research, 156, 285–294. Kowner, R. (1995). Laterality in facial expressions and its effect on attributions of emotion and personality: A reconsideration. Neuropsychologia, 33, 539–559. Kucharska-Pietura, K., Phillips, M. L., Gernand, W., & David, A. S. (2003). Perception of emotions from faces and voices following unilateral brain damage. Neuropsychologia, 41, 1082–1090. LeDoux, J. E. (2000). Emotion circuits in the brain. Annual Review of Neuroscience, 23, 155–184. Levenson, R. W., Ekman, P., & Friesen, W. V. (1990). Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology, 27, 363–384. Ley, R. G., & Bryden, M. P. (1982). A dissociation of right and left hemisphere effects for recognizing emotional tone and verbal content. Brain and Cognition, 1, 3–9. Lipp, O. V., & Waters, A. M. (2007). When danger lurks in the background. Attentional capture by animal fear-relevant distractors is specific and selectively enhanced by animal fear. Emotion, 7, 192–200. McIntosh, D. N., Zajonc, R. B., Vig, P. S., & Emerick, S. W. (1997). Facial movement, breathing, temperature, and affect: Implications of the vascular theory of emotional efference. Cognition and Emotion, 11, 171–195. Mitchell, R. L. C. (2006). Does incongruence of lexicosemantic and prosodic information cause discernable cognitive conflict? Cognitive, Affective, and Behavioral Neuroscience, 6, 298–305. Mitchell, R. L. C., Elliott, R., Barry, M., Cruttenden, A., & Woodruff, P. W. R. (2003). The neural response to emotional prosody, as revealed by functional magnetic resonance imaging. Neuropsychologia, 41, 1410–1421. Mitchell, R. L. C., & Ross, E. D. (2008). FMRI evidence for the effect of verbal complexity on lateralization of the neural response associated with decoding prosodic emotion. Neuropsychologia, 46, 2880–2887. O¨hman, A., Esteves, F., & Soares, J. J. F. (1995). Preparedness and preattentive associative learning: Electrodermal conditioning to masked stimuli. Journal of Psychophysiology, 9, 99–108. O¨hman, A., & Mineka, S. (2001). Fears, phobias, and preparedness: Toward an evolved module of fear and fear learning. Psychological Review, 108, 483–522. O¨hman, A., & Wiens, S. (2003). On the automaticity of autonomic responses in emotion: An evolutionary perspective. In R. J. Davidson, K. R. Scherer, & H. H. Goldsmith (Eds.), Handbook of affective sciences (pp. 256–275). Oxford, England: Oxford University Press.
T. Aue et al. Panksepp, J. (1982). Toward a general psychobiological theory of emotions. Behavioral and Brain Sciences, 5, 407–467. Peppe´, S., McCann, J., Gibbon, F., O’Hare, A., & Rutherford, M. (2007). Receptive and expressive prosodic ability in children with high-functioning autism. Journal of Speech, Language, and Hearing Research, 50, 1015–1028. Plutchik, R. (1980). Emotion. A psychoevolutionary synthesis. New York: Harper & Row. Pourtois, G., Grandjean, D., Sander, D., & Vuilleumier, P. (2004). Electrophysiological correlates of rapid spatial orienting towards fearful faces. Cerebral Cortex, 14, 619–633. Rodway, P., & Schepman, A. (2007). Valence specific laterality effects in prosody: Expectancy account and the effects of morphed prosody and stimulus lead. Brain and Cognition, 63, 31–41. Rosadini, G., & Rossi, G. F. (1967). On the suggested cerebral dominance for consciousness. Brain, 90, 101–112. Ross, E., Edmondson, J. A., Seibert, G. B., & Homan, R. W. (1988). Acoustic analysis of affective prosody during ride-sided Wada test: A within-subjects verification of the right hemisphere’s role in language. Brain and Language, 33, 128–145. Ross, E. D., Thompson, R. D., & Yenkosky, J. (1997). Lateralization of affective prosody in brain and the callosal integration of hemispheric language functions. Brain and Language, 56, 27–54. Russell, J. A., Bachorowski, J. A., & Fernandez-Dols, J. M. (2003). Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329–349. Sander, D., Grandjean, D., Pourtois, G., Schwartz, S., Seghier, M. L., Scherer, K. R., et al. (2005). Emotion and attention interactions in social cognition: Brain regions involved in processing anger prosody. NeuroImage, 28, 848–858. Scherer, K. R. (1986). Vocal affect expression: A review and a model for future research. Psychological Bulletin, 99, 143–165. Scherer, K. R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40, 227–256. Schirmer, A., & Kotz, A. (2003). ERP evidence for a sex-specific Stroop effect in emotional speech. Journal of Cognitive Neuroscience, 15, 1135–1148. Schirmer, A., & Kotz, S. A. (2006). Beyond the right hemisphere: Brain mechanisms mediating vocal emotional processing. Trends in Cognitive Sciences, 10, 24–30. Schirmer, A., Kotz, S. A., & Friederici, A. D. (2005). On the role of attention for the processing of emotions in speech: Sex differences revisited. Brain Research. Cognitive Brain Research, 24, 442–452. Schmitt, J. J., Hartje, W., & Willmes, K. (1997). Hemispheric asymmetry in the recognition of emotional attitude conveyed by facial expression, prosody and propositional speech. Cortex, 33, 65–81. Sidtis, J. J., & Van Lancker-Sidtis, D. (2003). A neurobehavioral approach to dysprosody. Seminars in Speech and Language, 24, 93–105. Stemmler, G., Aue, T., & Wacker, J. (2007). Anger and fear: Separable effects of emotion and motivational direction on somatovisceral responses. International Journal of Psychophysiology, 66, 141–153. Van Lancker, D. (1980). Cerebral lateralization of pitch cues in the linguistic signal. International Journal of Human Communication, 13, 101–109. Van Lancker, D., & Sidtis, J. (1992). The identification of affective-prosodic stimuli by left- and right-hemisphere-damaged subjects: All errors are not created equal. Journal of Speech and Hearing Research, 35, 963–970. Vuilleumier, P. (2002). Facial expression and selective attention. Current Opinion in Psychiatry, 15, 291–300. Vuilleumier, P. (2005). How brains beware: Neural mechanisms of emotional attention. Trends in Cognitive Sciences, 9, 585–594. Vuilleumier, P., Armony, J. L., Driver, J., & Dolan, R. J. (2001). Effects of attention and emotion on face processing in the human brain: An event-related fMRI study. Neuron, 30, 829–841. Yang, T. T., Simmons, A. N., Matthews, S. C., Tapert, S. F., BischoffGrethe, A., Frank, G. K., et al. (2007). Increased amygdala activation is related to heart rate during emotion processing in adolescent subjects. Neuroscience Letters, 428, 109–114. Zajonc, R. B., Murphy, S. T., & McIntosh, D. N. (1993). Brain temperature and subjective emotional experience. In M. Lewis & J. M. Haviland (Eds.), Handbook of emotions (pp. 209–220). New York: Guilford Press. (Received December 4, 2009; Accepted April 8, 2010)
Psychophysiology, 48 (2011), 393–396. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01076.x
BRIEF REPORT
Threat of suffocation and defensive reflex activation
PETER J. LANG,a BETHANY C. WANGELIN,a MARGARET M. BRADLEY,b FRANCESCO VERSACE,b PAUL W. DAVENPORT,c and VINCENT D. COSTAb a
Department of Clinical and Health Psychology, University of Florida, Gainesville, Florida Department of Psychology, University of Florida, Gainesville, Florida Department of Physiological Sciences, University of Florida, Gainesville, Florida
b c
Abstract The current study examined emotional reflex reactions of participants threatened with respiratory distress caused by imposing a resistive load at inspiration. Cues signaling threat (breathing MAY be difficult) and safe periods were intermixed while startle reflexes, heart rate, skin conductance, and facial EMG activity were measured. Compared to safe cues, threat cues elicited significant startle potentiation, enhanced skin conductance, heightened corrugator EMG changes, and pronounced ‘‘fear bradycardia’’ consistent with defensive activation in the context of threatened respiratory dysfunction. These data indicate that anticipating respiratory resistance activates defensive responding, which may mediate symptomatology in patients with panic and other anxiety disorders. Descriptors: Respiratory distress, Threat versus safe, Fear, Startle
participants first experience one trial of specified inspiratory resistance, followed by an assessment of defensive reflex reactivity across multiple periods of anticipation and safety. The present research methodology is modeled on recent studies of threat of receiving a painful electric shock (e.g., Bradley, Silakowski, & Lang, 2008; Bradley, Moulder, & Lang, 2005). Somatic and autonomic defense reactions were monitored in the context of visual cues signaling periods when it might be difficult to breathe (‘‘threat’’) and periods when breathing would be unimpeded (‘‘safe’’). Throughout the experiment, the participant wore a nose clip and breathed through a mouthpiece equipped with a device that controlled inspiratory resistive load. When load was increased, breathing became more difficult. Participants initially experienced this sensation during a 60-s exposure trial. Then, blue or red screens signaled periods when the inspiratory load could occur again (i.e., ‘‘threat’’) or periods where breathing would not be difficult (‘‘safe’’). The same inspiratory load was presented again halfway through the experiment. Heart rate, skin conductance, and facial frowning (corrugator electromyogram [EMG]) activity were continuously collected, and eyeblink reflexes elicited by acoustic startle probes were measured during threat and safe periods. Previous studies assessing defensive reactivity under threat of shock found potentiated startle reflexes, heightened skin conductance activity, ‘‘fear bradycardia,’’ and increases in facial frowning (Bradley et al., 2005, 2008; Grillon, Ameli, Woods, Merikangas, & Davis, 1991; Grillon & Davis, 1995) during threat, compared to safe, periods. In the current study, considering that obstructed breathing represents a comparable threat, we expect that anticipating impeded respiration will prompt a similar profile of defensive activation.
Breathing is fundamental to survival, and the occurrence of abrupt respiratory blockages that augur suffocation can be frightening events. Patients suffering from asthma or chronic obstructive pulmonary disease frequently call for medical assistance with persistent fears of respiratory dysfunction that adds significantly to disease morbidity (Davenport & Vovk, 2009). Suffocation fears are also predominant in anxiety disorders, most notably claustrophobia, agoraphobia, and panic (Klein, 1993; Roth, 2005; Timmons & Ley, 1994). Furthermore, comorbid anxiety disorders occur at higher rates in individuals with respiratory disease than in the general population (Brenes, 2003), emphasizing the survival threat implicit in respiratory distress and its exaggeration in clinically anxious individuals. In the current study, we measured emotional reflex reactions of unselected participants under threat of impeded breathing. Considering its diagnostic relevance for many anxiety disorders, a paradigm measuring reactivity in anticipation of respiratory resistance could be a useful tool for assessing anxiety disorder risk and resilience. Previous respiration studies examined defensive reflex reactions associated with CO2 inhalation (e.g., Fannes, van Diest, Meulders, de Peuter, Vansteenwegen, & van den Bergh, 2008) or when anticipating a paced hyperventilation task (Melzig, Michalowski, Holtz, & Hamm, 2008). The present study employs a more easily controlled challenge in which This research was supported in part by NIMH grant P50 MH07285004 to Peter J. Lang. Francesco Versace is now at the MD Anderson Cancer Center, Houston, Texas. Address correspondence to: Peter J. Lang, NIMH-CSEA, PO Box 112766, University of Florida, Gainesville, FL 32611. E-mail:
[email protected] 393
394 Method Participants Fifteen (6 female) University of Florida psychology students received course credit for participation in this study, which was approved by the University of Florida Institutional Review Board. Stimulus Materials Threat and safety cues. Cues were simple red or blue backgrounds projected on a screen in front of the participant. For half of the participants, a red background indicated that breathing might be difficult (‘‘threat’’) and a blue background indicated that breathing would be unimpeded (‘‘safe’’); for the other half of the participants, the conditions were reversed. Inspiratory load. Throughout the experiment, participants breathed through a mouthpiece attached to a non-rebreathing valve and pneumotachograph. The inspiratory portion of the mouthpiece was connected to a tube equipped with a manually operated 3-way valve to which the experimenter added resistance (50 cm H2O/L/s) by applying a stopcock. Inspiratory flow was continuously recorded from the pneumotachograph (Validyne MP45, Validyne Engineering Corp., Northridge, CA) and the signal was integrated (Validyne FV156). The experimenter applied the stopcock at the start of an exhalation to instate the inspiratory load. The load was instated for 60 s in the presence of the threat cue on two occasions: once during an initial exposure trial prior to the start of the experiment, and once halfway through the experimental session. Trial Structure Throughout the session, a white fixation cross continuously appeared centrally on the projector screen. Each trial began with a 3-s baseline in which a black background appeared. Threat and safety cues appeared for 12 s, followed by a variable (10–15 s) intertrial interval (black background). Startle probes occurred on each trial at one of four counterbalanced latencies following cue onset (4, 5, 6, or 7 s). After the initial exposure to the load, the session consisted of 32 trials (16 threat, 16 safe), with the resistive load applied once again halfway through the session. Procedure The participant sat in a comfortable chair while the experimenter attached the sensors and mouthpiece. The experimenter then explained that the projector would periodically display a red or blue screen, and that any time the screen was red (or blue, depending on group assignment), it was possible that it would become more difficult to breathe. During intertrial intervals, the participant was instructed to relax and wait for the next color cue. The initial exposure trial was then presented, in which the threat cue appeared while the inspiratory load was applied for 60 s. Instructions were then summarized, and the experiment began. At the end of the session, the participant made separate ratings of the level of fear and suffocation experienced when breathing under the resistive load, using a 10-point scale (0 5 no fear/suffocation, 9 5 severe fear/suffocation). Apparatus VPM (version 11.7; Cook, 2001) controlled data acquisition and E-prime (Psychology Software Tools, Inc., Sharpsburg, PA) controlled events. Cues were displayed via an LCD projector on
P. J. Lang et al. a screen 3 m from the participant. Acoustic startle stimuli (50 ms, 95 dB) were presented over headphones. In addition to inspiratory flow (described above), facial EMG was recorded at 20 Hz from 4 mm electrodes placed over the left corrugator supercilii. Raw signals were amplified by 30,000, filtered (13–1000 Hz, Coulbourn V75-01, Coulbourn Instruments, Whitehall, PA) and integrated (V76-23A, 500 ms time constant). Skin conductance was recorded from 8 mm electrodes filled with NaCl paste on the hypothenar eminence of the left palm. Heart rate was recorded from 8 mm electrodes on each forearm. A Schmitt trigger detected R-waves; inter-beat intervals were reduced to half-second bins. For startle, 4 mm Ag-AgCl electrodes were placed over the left orbicularis oculii (Fridlund & Cacioppo, 1986). Raw signals were amplified by 30,000 (Coulbourn V75-02), filtered from 8 to 1000 Hz (V75-48), sampled at 2000 Hz, and integrated (V-7623A, 20-ms time constant). Offline, blinks were filtered between 28 Hz and 500 Hz, rectified, and integrated using a 20-ms time constant. Data Reduction and Analysis Blink data were reduced using a peak-scoring algorithm (Cook, 2001). Magnitude was transformed to microvolts, and trials in which there was no response were scored as zero. Blink data were excluded for 3 participants due to an excessive number of offscale responses, resulting in n 5 12. Heart rate, skin conductance, and corrugator EMG responses were calculated as change scores relative to the 1-s baseline prior to cue onset and averaged over the 12-s cue period. For respiratory data, a measure of inspiratory volume was obtained by computing the integral of the baseline-deviated inspiratory flow waveform for each 12-s cue period. SPSS (version 17.0, Chicago, IL) univariate repeated measures analyses of variance (ANOVAs) were performed for each measure, evaluating cue type (safe or threat) as a within-subjects factor. Results Exposure Following exposure to the inspiratory resistive load, participants reported moderate fear (mean 5 4.0 out of 9, s.d. 5 2.3) and moderate sensation of suffocation (mean 5 4.5 out of 9, s.d. 5 2.3). Figure 1 illustrates that addition of the respiratory load during the initial exposure trial resulted in reliably reduced inspiratory flow (Figure 1 inlay), which was accompanied by a dramatic increase in skin conductance throughout the 60-s period. Cued Threat and Safety Figure (top left panel) illustrates that startle probes presented in the context of cues threatening respiratory distress elicited potentiated startle blinks, compared to probes presented during safe periods, F(1,11) 5 11.1, po.01, Z2p 5 0.50. Figure 2 (top right panel) illustrates that skin conductance change from the pre-cue baseline significantly increased during threat of respiratory distress, compared to safe periods, F(1,14) 5 6.8, po.05, Z2p 5 0.33 (see Table 1). Consistent with the hypothesis that the threat of respiratory distress is aversive, corrugator muscle (i.e., frown) tension also significantly increased when viewing threat, compared to safety, cues, F(1,14) 5 7.1, po.05, Z2p 5 0.34 (Figure 2, bottom left panel). Relatedly, significant cardiac deceleration was observed when participants viewed threat cues, compared to safe cues (Figure 2, bottom right panel), F(1,14) 5 6.5, po.05, Z2p 5 0.32. Inspiratory volume was significantly greater when
Threat of suffocation
Figure 1. When the inspiratory resistive load was applied, skin conductance increased across the 60-s exposure trial. Inlay shows an example inspiratory flow waveform for one participant during exposure to the load, illustrating that inspiratory flow was reliably impeded across the 60-s period. mS 5 microsiemens.
viewing cues signaling threat of respiratory distress, compared to cues that signaled unobstructed breathing, F(1,14) 5 7.15, po.05, Z2p 5 0.34.
Discussion The present findings show that anticipation of only a brief, partial blockage of breathing prompts activation of the brain’s
395 defense reflex system, reflected here in potentiated startle responses, activation of the sympathetic chain as indexed by increases in skin conductance, and facial displays of distress (i.e., marked corrugator muscle tension). Furthermore, the sustained cardiac deceleration seen during anticipated respiratory distress is consistent with a state of immobility and heightened vigilance (Bradley, 2009), as seen in the ‘‘fear bradycardia’’ found when animals confront a predator (Campbell, Wood, & McBride, 1997). It has previously been shown (Davenport & Vovk, 2009; von Leupoldt, Sommer, Kegat, Eippert, Baumann, et al., 2008) that a respiratory load activates neural structures (e.g., amygdala, insula) within a motivational circuit typically implicated in affective experience. The current study shows that simply anticipating possible breathing difficulty prompts a range of reflex reactions mediated by this same defense circuit (Lang & Davis, 2006). Furthermore, the pattern of defensive engagement for participants under threat of disrupted breathing parallels that found for participants under the threat of receiving a painful electric shock (Bradley et al., 2005, 2008; Grillon & Davis, 1995), which has also been related to defense circuit activation in brain imaging research (Costa, Bradley, Versace, & Lang, 2006; Dalton, Kalin, Grist, & Davdison, 2005). The similarity in defensive profiles when threatened with shock or disrupted breathing indicates that cues associated with threat to human lifeFwhether painful or notFactivate a similar pattern of heightened vigilance and preparation for action (Bradley, 2009). The strong defense reactions found here in healthy participants after exposure to a brief respiratory blockageFdescribed
Figure 2. Top left: Reflexive startle blinks are larger in response to probes presented during cues signaling threat of respiratory distress, compared to cues signaling unobstructed breathing. Top right: Mean change in skin conductance level is heightened in the context of threat, compared to safe, cues. Bottom right: Mean heart rate shows deceleratory changes during threat, compared to safe, periods. Bottom left: Mean change in corrugator EMG (frown) activity is larger during threat, compared to safe, cues. mV 5 microvolts; mS 5 microsiemens; BPM 5 beats per minute. Error bars represent 95% confidence intervals (Loftus & Masson, 1994).
396
P. J. Lang et al.
Table 1. Psychophysiological Change from Pre-Cue Baseline for Threat and Safe
Skin conductance change (mSiemens) Heart rate change (beats per min) Corrugator EMG change (mV) Inspiratory volume (ADU)
Threat
Safe
0.10 (0.03) ! 0.95 (0.42) 0.58 (0.20) 74963 (2680)
0.00 (0.02) 0.13 (0.38) ! 0.06 (0.14) 77739 (4189)
Note: Mean (SE) for each dependent measure as a function of cue type. Values reflect changes in skin conductance, heart rate, and corrugator electromyographic activity relative to a 1-s pre-cue baseline. mV 5 microvolts; ADU 5 analog-to-digital units.
as only moderately fear-provoking, on averageFemphasize the broad evolutionary foundation for respiratory anxiety. Furthermore, a post-hoc analysis revealed that individuals reporting greater fear during exposure to inspiratory resistance subsequently showed greater sympathetic activation (i.e., skin conductance level) when viewing cues signaling threat of impeded breathing (r[13] 5 0.58, po.05). Although the small sample size encourages caution, the data support the view that greater fear of
respiratory symptoms is associated with exaggerated sensitivity to threat cues, suggesting that the current paradigm could have meaningful clinical value. These findings highlight the importance of refining treatments targeting suffocation fear in individuals with respiratory disease as well as anxiety disorders. In this regard, Coventry (2009) cites several pulmonary rehabilitation studies suggesting that incremental breathing exercises can be effective in reducing dyspnearelated anxiety. Similarly, a desensitization treatment using supervised anticipation of breathing under load, as in interoceptive exposure therapy for panic disorder (e.g., Craske, Barlow, & Meadows, 2000), could be beneficial in addressing anxiety associated with a dread of respiratory distress. Fear of suffocation and dyspnea have been signaled as key features of panic and other anxiety diagnoses (Ley, 1989; Roth, 2005). The current data show that defensive activation and negative affect are reliable consequences in anticipation of even transient respiratory resistance. Thus, assessing the degree of defensive activation consequent on a controlled respiratory load could provide a safe method for measuring individual differences in resiliency, and potentially provide a tool for prognostic assessment in chronic fear and anxiety disorders.
REFERENCES Bradley, M. M., Silakowski, T., & Lang, P. J. (2008). Fear of pain and defensive activation. Pain, 137, 156–163. Bradley, M. M., Moulder, B., & Lang, P. J. (2005). When good things go bad: The reflex physiology of defense. Psychological Science, 16, 468–473. Bradley, M. M. (2009). Natural selective attention: Orienting and emotion. Psychophysiology, 46, 1–11. Brenes, G. A. (2003). Anxiety and chronic obstructive pulmonary disease: Prevalence, impact, and treatment. Psychosomatic Medicine, 65, 963–970. Campbell, B. A., Wood, G., & McBride, T. (1997). Origins of orienting and defensive responses: An evolutionary perspective. In P. J. Lang, R. F. Simons, & M. T. Balaban (Eds.) Attention and orienting: Sensory and motivational processes (pp. 41–67). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Cook, E. W. III. (2001). VPM reference manual. Birmingham, Alabama: Author. Costa, V. D., Bradley, M. M., Versace, F., & Lang, P. J. (2006). Fear relevance modulates frontal cortex activity during anticipation of pain. Program No.73.8. 2006 Neuroscience Meeting Planner. Atlanta, GA: Society for Neuroscience, 2006. Online. Coventry, P. A. (2009). Does pulmonary rehabilitation reduce anxiety and depression in chronic obstructive pulmonary disease? Current Opinion in Pulmonary Medicine, 15, 143–149. Craske, M. G., Barlow, D. H., & Meadows, E. A. (2000). Mastery of your anxiety and panic: Therapist guide for anxiety, panic, and agoraphobia (MAP-3). San Antonio, TX: Graywind/Psychological Corporation. Dalton, K.M, Kalin, N. H., Grist, T. M., & Davdison, R. J. (2005). Neural-cardiac coupling in threat-evoked anxiety. Journal of Cognitive Neuroscience, 17, 969–80. Davenport, P. W., & Vovk, A. (2009). Cortical and subcortical central neural pathways in respiratory sensations. Respiratory Physiology & Neurobiology, 167, 72–86.
Fannes, S., van Diest, I., Meulders, A., de Peuter, S., Vansteenwegen, D., & van den Bergh, O. (2008). To inhale or not to inhale: Conditioned avoidance in breathing behavior in an odorF20% CO2 paradigm. Biological Psychology, 78, 87–92. Fridlund, A. J., & Cacioppo, J. T. (1986). Guidelines for human electromyographic research. Psychophysiology, 23, 567–589. Grillon, C., Ameli, R., Woods, S. W., Merikangas, K., & Davis, M. (1991). Fear-potentiated startle in humans: Effects of anticipatory anxiety on the acoustic blink reflex. Psychophysiology, 28, 588–595. Grillon, C., & Davis, M. (1995). Acoustic startle and anticipatory anxiety in humans: Effects of monaural right and left ear stimulation. Biological Psychiatry, 38, 68–70. Klein, D. F. (1993). False suffocation alarms, spontaneous panics, and related conditions. Archives of General Psychiatry, 50, 306–317. Lang, P. J., & Davis, M. (2006). Emotion, motivation, and the brain: Reflex foundations in animal and human research. Progress in Brain Research, 156, 3–29. Ley, R. (1989). Dyspneic-fear and catastrophic cognitions in hyperventilatory panic attacks. Behavior Research and Therapy, 5, 549–54. Loftus, G. R., & Masson, M. E. J. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review, 1, 476–490. Melzig, C. A., Michalowski, J. M., Holtz, H., & Hamm, A. O. (2008). Anticipation of interoceptive threat in highly anxiety sensitive persons. Behaviour Research and Therapy, 46, 1126–1134. Roth, W. T. (2005). Physiological markers for anxiety: Panic disorder and phobias. International Journal of Psychophysiology, 58, 190–198. Timmons, B. H., & Ley, R. (1994). Behavioral and psychological approaches to breathing disorders. New York: Plenum. von Leupoldt, A., Sommer, T., Kegat, S., Eippert, F., Baumann, H. J., Klose, H., et al. (2008). The unpleasantness of perceived dyspnea is processed in the anterior insula and amygdala. American Journal of Respiratory and Critical Care Medicine, 177, 1026–1032. (Received November 19, 2009; Accepted April 28, 2010)
Psychophysiology, 48 (2011), 397–409. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01068.x
The role of somatic threat feature detectors in the attentional bias toward pain: Effects of spatial attention
ROBERT DOWMAN Department of Psychology, Clarkson University, Potsdam, New York, USA
Abstract Our previous work suggests that somatic threat feature detectors indexed by a pain-evoked midlatency negative scalp potential play an important role in the attentional bias toward pain. In these studies the somatic threat feature detectors facilitated the shift in attention to a somatic threat when attention was focused on another stimulus modality but not when it was focused on another spatial location. This experiment used the Posner cuing paradigm to investigate possible explanations for this discrepancy. The results demonstrate that the different somatic threat effects observed in previous modal and spatial cuing studies are not due to attentional set. Rather, this discrepancy may be related to differences in contingent attention capture. This study also verifies earlier findings suggesting that some of the painrelated P2 component generators are involved in stimulus-driven shifts in spatial attention toward task-relevant stimuli. Descriptors: Threat, Pain, Attention, Threat feature detection, P2, P3a
Crombez, Eccleston, and Goubert (2004) performed a task determining whether an auditory tone or a painful electrical target stimulus was presented. Each target was preceded by the cue word ‘‘PAIN’’ or ‘‘TONE.’’ Some of the trials were validly cued (e.g., the cue word ‘‘PAIN’’ was followed by the painful electrical target stimulus), some were invalidly cued (e.g., the cue word ‘‘TONE’’ was followed by the painful electrical target stimulus), and on some trials no cue was given. As expected, reaction times were faster when the target was validly cued than when it was invalidly cued. More importantly, the reaction time validity effect (difference between the invalidly and validly cued conditions) was longer for the tone target than the painful electrical target. Interestingly, the somatic threat effect was seen in the invalidly cued–no cue condition reaction time difference and not in the no cue–validly cued reaction time difference. Spence, Bentley, Phillips, McGlone, and Jones (2002) and Dowman (2007a,b) used the cuing paradigm to investigate the attentional biases toward painful laser and sural nerve electrical target stimuli, respectively. The nonthreatening target was a visual stimulus, and the visual and somatic threat target stimuli were presented at or very near the same location. Hence, these experiments involved shifts of attention between sensory modalities (modal) and not spatial location. Both Spence et al. (2002) and Dowman (2007a,b; Dowman & ben-Avraham, 2008) reported that the reaction time validity effect was longer for the nonthreatening visual target than for the somatic threat targets. These studies did not, however, include a no-cue or neutral-cue condition. Investigators often use the no-cue or neutral-cue condition to gain insight into the attentional components responsible for the cuing effects. For example, Van Damme et al. (2002, 2004) used the reaction time difference between the invalidly cued and
The ability to rapidly detect an unattended threat, interrupt ongoing tasks, and orient attention toward the threat has clear and obvious implications for survival (Corbetta & Schulman, 2002; Norman & Shallice, 1986; Posner & DiGiorlamo, 1998). An important category of threats are those to the body, especially those that cause pain. Indeed, pain can be especially effective at capturing and holding attention (Eccleston & Crombez, 1999), and is well known to disrupt ongoing cognitive processes (e.g., Eccleston, 1994, 1995; Kuhajda, Thorn, Klinger, & Rubin, 2002; Veldhuijzen et al., 2006). Interestingly, hypervigilance toward body sensations coupled with catastrophic interpretations of those sensations appears to play a role in the etiology of chronic pain (Gatchel, Peng, Peters, Fuchs, & Turk, 2007; Pincus & Morley, 2001; Turk & Okifuji, 2002). Hence, understanding how somatic threats capture and hold attention is important not only for understanding a fundamental cognitive process, but it may also have important clinical implications. The ability of somatic threats to capture and hold attention is thought to involve interactions between top-down voluntary and bottom-up, stimulus-driven attentional processes. However, the neural mechanisms underlying these processes are not fully understood (Legrain et al., 2009). Psychophysical studies involving the Posner cuing paradigm (Posner, Snyder, & Davidson, 1980) have revealed some interesting phenomenological characteristics of the attentional bias toward somatic threats that may provide some insight into how they capture and hold attention. Participants in studies by Van Damme, Crombez, and Eccleston (2002) and Van Damme, Address correspondence to: Robert Dowman, Department of Psychology, Clarkson University, Potsdam, NY 13699-5825, USA. Email:
[email protected] 397
398 no-cue conditions to estimate somatic threat effects on the attention disengage process and differences between the no-cue and validly cued conditions to estimate effects on the attention engage process (see Cisler, Bacon, & Williams, 2009). Because somatic threat effects were seen for the invalidly cued–no cue reaction time difference (nonthreat longer than threat) and not the no-cue–validly cued reaction time difference, Van Damme et al. (2002, 2004) concluded that it took longer to disengage attention from the cued somatic threat target and that the somatic threat did not affect attentional capture. There is, however, an alternative explanation for these results. The shorter invalidly cued–no-cue condition reaction time difference could be due to the unattended somatic threat target being more efficient at engaging attention than the unattended nonthreatening target, which would have resulted in a faster shift away from the cued target and toward the somatic threat. This explanation is usually discarded if there is no effect on the nocue–validly cued reaction time difference, which is presumed to index attentional capture. However, the attentional capture explanation would work if the somatic threat’s ability to capture attention was greater only when it was unattended. Our electrophysiological studies suggest that this might be the case. That is, our studies suggest that somatic threats are detected by threat feature detectors early in sensory processing and that these detectors facilitate the activation of other brain areas that execute the shift in attention toward the threat. Our studies also provide evidence that the somatic threat detector response is greater when the threat is unattended (invalidly cued) than when it is attended (validly cued). Evidence supporting the role of threat feature detectors in the attentional bias toward somatic threats comes from the electrophysiological data obtained in the Dowman (2007a,b) modal cuing studies described above. There are three components of the somatosensory evoked potential elicited by painful sural nerve electrical stimuli that appear to be involved in detecting and orienting attention toward unattended somatic threats. That is, the amplitudes of these components are larger when the painful sural nerve target is unattended (invalidly cued) than when it is attended (validly cued; Dowman, 2001, 2004b, 2007a). These components are a bilateral negative potential over the temporal scalp at about 150 ms poststimulus whose amplitude is largest contralateral to the sural nerve evoking stimulus (contralateral temporal negativity or CTN), a frontocentral negativity (FCN) that temporally overlaps with the CTN, and the anterior component of the P3a event-related potential that occurs 320–400 ms poststimulus (Dowman, 2004c). The CTN validity effect (amplitude is larger in the invalidly cued than the validly cued condition) is only seen when the sural nerve target stimulus is given at a strong, threatening level (Dowman, 2007a). The CTN amplitude is the same across the validly and invalidly cued conditions when the sural nerve target is given at a level that produces a weak, nonthreatening paresthesia or tapping sensation (Dowman, 2007b). Hence, the CTN validity effect appears to be specific to somatic threats. The FCN and anterior P3a validity effects, on the other hand, are also seen at nonthreatening levels (Dowman, 2007b). This implies that the FCN and P3a generators are involved in more general orienting processes. As will be seen below, the response properties of these components match those of their underlying neural generators. The putative generators of the CTN, FCN, and P3a components were estimated from dipole source localization
R. Dowman analyses (Dowman, 2001, 2004a,b; Dowman & Darcey, 1994; Dowman & Schell, 1999a), scalp topographic studies (Dowman, 2004c; Dowman & Schell, 1999a), by comparing the response properties of the component and its putative generators (see below), and intracranial recordings in epilepsy patients undergoing presurgical evaluation for treatment of their seizures (Dowman, Darcey, Barkan, Thadani, & Roberts, 2007; Halgren, Marinkovic, & Chauvel, 1998), the latter of which is considered the gold standard for localizing scalp potentials (Halgren et al., 1998). The P3a event-related potential generators have also been estimated from lesions in patients (for review, see Soltani & Knight, 2000). Note that the scalp topographic patterns of the sural nerve-evoked CTN, FCN, and P3a components obtained in this study are essentially the same as in the studies used to estimate their neural generators. The scalp topographic pattern is determined by the number, location, orientation, and relative magnitudes of its underlying generators (Nunez, 1981; Scherg, 1990). Hence, it is reasonable to assume that the CTN, FCN, and P3a component generators estimated in these earlier studies are the same here. The CTN is generated in the perisylvian area, and likely involves a combination of activity in the dorsal posterior insula and the adjacent second somatosensory area (see Dowman, 2007a,b; Dowman & ben-Avraham, 2008; Dowman et al., 2007). The FCN is generated in part by the medial prefrontal cortex, and the anterior component of the P3a event-related potential is generated in part by the lateral prefrontal cortex. The response properties of these components and their putative generators are consistent with roles in detecting and orienting attention toward somatic threats. The dorsal posterior insula is located about 1.5 cm medial to the second somatosensory area and contains cells that respond to stimuli that are associated with deviations from homeostasis, including but not limited to noxious stimuli and air hunger (Craig, 2002, 2003), all of which can be considered threats to the body. The idea of somatic threat feature detectors is also consistent with the feature detectors that have been proposed for other biologically-relevant threats, such as dangerous animals and the facial expressions of conspecifics (O¨hman, 2000, 2005; O¨hman & Mineka, 2001). The medial and lateral prefrontal cortices have been shown to play important roles in attentional control. Cohen and his colleagues have provided strong evidence that the medial prefrontal cortex is involved in monitoring any situation that requires a change in attentional control (Botvinick, Braver, Barch, Carter, & Cohen, 2001). Indeed, the presence of an unattended somatic threat certainly qualifies as such a situation. Furthermore, there are extensive connections between the dorsal posterior insula and medial prefrontal cortex (Craig, 2003) that could mediate this monitoring process. The medial prefrontal cortex, in turn, signals the lateral prefrontal cortex to initiate the shift in attention (Botvinick et al., 2001; Yeung, Botvinick, & Cohen, 2004). To date, all of the evidence supporting the somatic threat detection and orienting hypothesis has come from modal cuing studies. The present study sought to determine whether the somatic threat effects can be demonstrated in a spatial cuing paradigm. This is important, given that an earlier spatial cuing study failed to find any validity effects on the CTN (Dowman, 2004b). This presents the possibility that the somatic threat detector activity is facilitated when the somatic threat is presented outside the focus of modal attention but not spatial attention. However, there is another important difference between the spatial and modal cuing studies that might explain
Threat detectors and attentional bias toward pain this discrepancy. In the spatial cuing study, the sural nerve target stimuli presented to the left and right sides were all painful (i.e., threatening), whereas in the modal cuing study the visual targets were nonthreatening, and only the sural nerve target stimuli were threatening. This was also the case for the other modal cuing studies reviewed above that reported an attentional bias toward somatic threats. The threat value of the uncued target stimulus could impact the ratio of attention allocated to the cued and uncued targets (i.e., the attentional set), which in turn might explain why reaction time validity effects were seen in the modal but not the spatial cuing studies. For example, when the cued and uncued targets are threatening and nonthreatening, respectively, one would expect that the ratio of attention allocated to the cued and uncued sides would be larger (i.e., much more attention allocated to the cued than uncued target) than when both targets are threatening. This follows from the assumption that more attention will be allocated to the uncued target, possibly at the expense of the cued target, if it is threatening than if it is nonthreatening. The ratio of attention allocated to the cued and uncued targets has important implications for the magnitude of the somatic threat detector activity. According to the threat detection and orienting hypothesis, allocating attention toward the threat detector decreases its response, as suggested by the CTN amplitude being smaller in the validly cued than the invalidly cued condition (Dowman, 2007a,b; Dowman & ben-Avraham, 2008). This decrease might be mediated by inhibitory connections between the lateral prefrontal cortex areas involved in biasing the excitability of task-relevant stimulus and response representations or by a template-matching process similar to that reported for transient detector responses in the auditory cortex (see Dowman & ben-Avraham, 2008). According to this model, the more attention allocated to the threat detector in the invalidly cued condition the smaller its response and, consequently, the smaller the difference in threat detector activity between the validly and invalidly cued conditions. This would result in the CTN amplitude showing a small, possibly nonsignificant difference between the validly and invalidly cued conditions, as was the case in the spatial cuing experiment (Dowman, 2004b). The smaller ratio of attention allocated to the cued and uncued sides would also result in smaller differences between the validly and invalidly cued reaction times (Dowman & ben-Avraham, 2008). This is consistent with the finding that the reaction time validity effect was 63 ms in the spatial cuing study (Dowman, 2004b) compared to 100 ms in the modal cuing study (Dowman, 2007a). The goal of the present study was to determine whether the absence of a threat effect in our previous spatial cuing experiment was due to attentional set. Each participant was given two blocks. In one block both the left and right sural nerve electrical targets were weak and nonthreatening (nonpain/nonpain condition). In the other block, all of the targets presented to one side were nonthreatening and all of the targets presented to the other side were threatening (i.e., painful; nonpain/pain condition). The nonpain/pain condition should produce an attentional set that is comparable to that in our modal cuing studies. If attentional set was responsible for the discrepancy between our previous spatial and modal cuing studies, then we should see somatic threat effects on the CTN and reaction times here. Comparing the reaction times for the validly cued nonthreatening sural nerve targets in the nonpain/nonpain and nonpain/pain conditions will
399 determine whether the uncued threatening target stimulus is drawing attentional resources from the cued nonthreatening target. If it does, then the reaction times for the validly cued nonthreatening target will be longer in the nonpain/pain than the nonpain/nonpain condition.
Methods Participants Twenty-two healthy adults participated in the experiment (mean ! SD age 5 19.0 ! 1.0 years; 17 men). Four subjects (3 men) were not included in the analysis because of poor task performance (accuracyo.80). All had normal or corrected-tonormal vision. Each participant was given a detailed explanation of the procedure, and each read and signed an informed consent document prior to participating. The research conformed to the American Psychological Association standards for the ethical treatment of human subjects and was approved by the Clarkson University Institutional Review Board. The subjects were comfortably positioned in a recliner chair located in an electrically shielded, sound-attenuated, and temperature controlled (21 ! 11C) recording chamber. The participants’ lower legs rested on L-shaped braces to help maintain foot and lower limb position. Cuing and Sural Nerve Electrical Target Stimuli The cuing stimuli consisted of the uppercase letters, ‘‘L’’ and ‘‘R,’’ presented on a computer monitor located 1.9 m in front of the participant. The letters were 2.5 cm high (visual angle 5 0.81) and had a duration of 500 ms. Somatosensory electrical target stimuli were presented to the left and right sural nerves at the ankle. The electrical stimulus consisted of a five-pulse train of square-wave pulses (1-ms pulse duration, 250-Hz pulse frequency, 17-ms duration) delivered through two electrodes positioned 2 cm apart on the skin overlying the nerve. The stimulating electrode impedances were less than 5000 O. The stimulator output was computer controlled (see Dowman, 2007a). Two sural nerve electrical stimulus intensities were used. One produced a very weak tapping and/or paresthesia sensation and the other a moderately painful prickling sensation. The current levels were determined at the beginning of the session using a single ascending/descending series of stimulations to estimate the sensory and pain thresholds and the pain tolerance level. The current levels were then adjusted during magnitude rating trials, where the two current levels were presented in pseudorandom order and a 4-s interstimulus interval, and the subjects rated the intensity of each stimulus on a 9-point scale (1 5 sensory threshold, 5 5 pain threshold, and 9 5 maximum pain tolerable). The magnitude rating trials were performed on the left and right sides separately. At least 30 trials for each level and side combination were given during the magnitude rating trials. The mean ( ! SD) stimulus currents for the left and right weak nonpainful levels were 1.04 ! 0.35 and 0.98 ! 0.31 mA, respectively, and were 4.33 ! 1.19 and 5.32 ! 3.07 mA, respectively, for the left and right moderately painful levels. The magnitude ratings for the left and right weak nonpainful levels were 1.54 ! 0.26 and 1.52 ! 0.34, respectively, and were 6.46 ! 0.68 and 6.95 ! 0.41, respectively, for the left and right moderately painful levels.
400 Stimulus current was measured online (see Dowman, 2007a) on every trial during the cuing experiment to ensure that it remained constant across trials (current standard deviations were less than 3% of the mean value) and did not show any increasing or decreasing trend during the course of the experiment. Previous work using this protocol has shown that there is very little or no change in the perceived magnitude or sensory quality of the sural nerve electrical stimuli during the course of experiments of this length (Dowman, 2001, 2004b, 2007a). Recording Parameters Evoked potentials elicited by the sural nerve electrical stimuli were recorded from 29 electrodes arranged on the scalp in a grid centered on a location 2 cm posterior to the vertex position (Cz 0 ) of the International 10–20 Electrode System (Sharbrough et al., 1991). The interelectrode distance along the sagittal and coronal axes was 5 cm, which samples at twice the highest spatial frequency of the mid- and long-latency peaks of the sural nerve somatosensory evoked potential (Dowman, 1994). The scalp electrodes were referenced to the noncephalic sternovertebral electrode (Dowman & Goshko, 1992). Eye movements were recorded from 2 electrodes positioned just lateral to the lateral canthus and over the inferior portion of the orbicularis oculi muscle of the left eye. The recording electrode impedances were less than 5000 O. The somatosensory and eye movement potentials were amplified 50,000 times, filtered between 0.3 and 100 Hz (! 6 dB points), and sampled at 250 Hz. Trials were rejected by computer when the scalp potentials exceeded 100 mV and/or when the eye movement potentials exceeded 40 mV to reduce head and neck muscle EMG and eye movement artifacts. Procedure Each trial began with a cuing stimulus consisting of either the uppercase letter ‘‘L’’ or ‘‘R.’’ The ‘‘L’’ signaled that the upcoming sural nerve target stimulus would be presented to the left side, and the ‘‘R’’ signaled that it would be presented to the right side. The onset of the sural nerve target stimulus followed the cue offset by 1512 ms. The subjects pressed a left response switch with their left index finger if the target was presented to the left side and a right response switch with their right index finger if the target was presented to the right. The response switch assignments were kept the same across subjects to eliminate reaction time differences and response conflicts due to the Simon effect (Lu & Proctor, 1995). The trial duration was 4 s. The target stimuli were validly cued on a randomly determined (without replacement) 75% of the trials, where the left sural nerve electrical target stimulus followed the ‘‘L’’ cue and the right sural nerve target stimulus followed the ‘‘R’’ cue. The remaining 25% of the trials were invalidly cued, where the left target followed the ‘‘R’’ cue and the right target followed the ‘‘L’’ cue. The participants were told that the cue would correctly signal the location of the next target on most but not all of the trials. They were instructed to focus their attention as directed by the cue and to make the response to the target as quickly and accurately as possible. There were two recording blocks given in counterbalanced order across subjects. In the nonpain/nonpain block, both the left and right targets were the weak nonpainful stimuli. In the nonpain/pain block, the nonpainful target was presented to one side and the painful target to the other. The side receiving the painful target was counterbalanced across subjects, and the subjects were told which side would receive the painful targets
R. Dowman and which would receive the nonpainful targets. Each recording block was comprised of 320 artifact-free trials. There were 160 left and 160 right target stimuli, given in random order, with 120 trials for each of the validly cued targets and 40 for each of the invalidly cued targets. Data Analysis The time segments used to measure the midlatency negative components of the sural nerve somatosensory evoked potential (SEP), including a central negativity at 80 ms poststimulus (CN), and the CTN and FCN, were determined by visual inspection of the 29-channel grand average SEP topographic patterns to identify the signature pattern for that component and its minimum location. Then the grand average SEP waveform from the minimum location was used to select segment onset and offset times that encompassed the peak. The onset and offset latencies for the amplitude segments of the components comprising the late positive potential, including the P1, P2, and P3a, were determined by separating the late positive potential into stable periods, where a stable period refers to consecutive time points having the same 29-channel topographic pattern. The stable periods were defined as those time points whose r2 with the peak was " 0.85 and greater than its r2 with the adjacent peak. The r2 statistic ensured that the topographic pattern, and by implication the source configuration, was the same throughout the time segment (Dowman, 1994). The SEP component amplitude was computed by averaging the potential across its time segment. The amplitudes of the CN and CTN components were obtained from their minimum scalp location. The amplitudes of the FCN, P1, P2, and P3a components were obtained from the five sagittal midline electrodes. Our earlier studies have shown that the amplitudes obtained from these electrode locations adequately capture the components of interest (Dowman, 2004b, 2007a,b). The reaction time analysis only used trials with accurate responses and whose reaction times were greater than 150 ms and less than 1400 ms. The mean # SD maximum reaction times values ranged from 721.3 # 304.0 ms to 891.0 # 191.0 ms across the stimulus level and validity conditions. Reaction times and SEP component amplitudes for the nonpainful targets given in the nonpain/nonpain condition were taken from the same side as the nonpainful target given in the nonpain/pain condition to better evaluate the effects of the uncued painful target on attentional set. Cue validity (validly vs. invalidly cued) and stimulus intensity (painful vs. nonpainful given in the nonpain/nonpain and nonpain/pain conditions) effects on the spatial discrimination task reaction time and electrophysiological measures were evaluated by a repeated measures analysis of variance (ANOVA). Greenhouse–Geisser corrections were applied when the numerator degrees of freedom were greater than 1 to correct for violations of the sphericity assumption. In reporting the significance levels the uncorrected degrees of freedom are given along with the epsilon (e) values used to adjust the significance level. Post hoc analyses were performed using Newman–Keuls tests to correct for multiple comparisons. Results Behavioral Measures The spatial discrimination task accuracy was high (mean # SD ranging from 0.99 # .03 to 1.00 # .01) and did not differ across
Threat detectors and attentional bias toward pain
401 validly cued invalidly cued
REACTION TIME (ms)
500
400
300
200
100
0 NN
NT
PAIN
Figure 1. The effects of sural nerve target stimulus intensity and cue validity on the spatial discrimination task reaction times. PAIN: painful sural nerve target; NP: nonpainful sural nerve target given in the nonpain/pain condition; NN: nonpainful sural nerve target given in the nonpain/nonpain condition. nNewman–Keuls test comparing the validly and invalidly cued conditions, po.05.
the stimulus intensity or validity conditions (main and interaction effect Fso1.99, p4.10). The spatial discrimination task reaction times were affected by the sural nerve target intensity (intensity main effect, F[2,34] 5 6.82, po.05, e 5 .62), where reaction times for the painful targets were faster than those for the nonpainful targets given in the nonpain/nonpain and nonpain/ pain conditions (Newman–Keuls test: po.05; Figure 1). There were no differences in the reaction times between the nonpainful target stimuli given in the nonpain/nonpain and nonpain/pain conditions (p4.10). Reaction times were faster in the validly cued than the invalidly cued condition (validity main effect, F[1,17] 5 41.52, po.0001), and this validity effect was the same at all stimulus intensity conditions (Intensity ! Validity interaction effect, F[2,34] 5 2.34, p4.10, e 5 .99). Hence, there was no attentional bias toward the painful target stimuli, nor did the uncued painful sural nerve target draw attention from the validly cued nonpainful target. Somatosensory Evoked Potential The sural nerve SEP grand average topographic patterns and waveforms are shown in Figures 2 and 3, respectively. As in our previous studies (Dowman, 2007a,b), there was a central negativity that peaked at 80 ms poststimulus, a contralateral temporal negativity and a fronto-central negativity that both peaked at about 150 ms, and a late positive potential comprised of a central peak at about 200 ms (P1), a central-parietal peak at about 300 ms (P2), and the P3a event-related potential that peaked at about 360 ms. The onset and offset latencies of the time segments used to measure the component amplitudes are given in Figure 2. Two subjects had a positive potential at the central scalp during the CN epoch instead of the central negativity seen in most subjects and the grand average. Our dipole source localization (Dowman, 1994, 1997, Dowman & Schell, 1999b) and intracranial recording studies (Dowman et al., 2007) provide strong evidence that the CN (which occurs during the SP1 and
SP2 stable periods described in our earlier publications) is generated in the primary somatosensory cortex. Normally, this dipole is oriented such that there is a small positive potential at the contralateral frontal regions and a large negative potential at the central scalp (Figure 2A). Given the individual variability in the convolutions of the cerebral cortex, it is reasonable to expect that in some subjects the dipole will be oriented such that the positive end of the dipole is pointing up, as may be the case in these 2 subjects (see Dowman, 1994, for a more detailed discussion of the individual variability in the sural nerve SEP scalp topographic patterns). Including these subjects in the CN amplitude analysis would have decreased the statistical power, as they would have appeared to be changing in the opposite direction as the rest of the subjects. That is, an increase in the dipole magnitude would have resulted in an increase in the central positivity in these 2 subjects and an increase in the central negativity in the other subjects. Hence these subjects were not included in the CN amplitude analysis. Note, however, that the amplitude of the central positivity in these subjects showed the same validity and intensity effects as described below for the CN. The topographic patterns of the other SEP components in these 2 subjects were comparable to the other subjects, and hence they were included in those analyses. The CN amplitude measured using the time segment that encompassed the CN peak was affected by stimulus intensity (intensity main effect, F[2,30] 5 28.87, po.0001, e 5 .58), where its amplitude was larger at the painful level than the nonpainful level obtained in the nonpain/pain and the nonpain/nonpain conditions (Newman–Keuls tests po.05; Figure 4). There was no difference in the CN amplitude of the nonpainful target presented in the nonpain/pain and nonpain/nonpain conditions (Newman–Keuls test p4.10). CN amplitudes were the same across the validly and invalidly cued conditions (validity main effect, F[1,15] 5 0.01, p4.10, Intensity ! Validity interaction effect, F[2,30] 5 1.78, p4.10, e 5 .89; Figure 4). Inspection of the waveforms shown in Figure 3 suggests that the CN amplitude may be larger in the invalidly than the invalidly cued condition at a short time segment whose onset is near the peak. I reanalyzed the CN amplitude evoked by the painful target using this smaller time segment (81–97 ms) and found that the increase in the invalidly cued condition approached statistical significance, t(15) 5 2.07, p 5 .056 (Figure 4). The CTN segment amplitude likewise exhibited a significant intensity main effect, F(2,34) 5 22.29, po.0001, e 5 .64, where the CTN amplitude evoked by the painful target was larger than those evoked by the nonpainful targets given in the nonpain/pain and nonpain/nonpain conditions (Newman–Keuls tests po.05). There was no difference in the CTN amplitudes evoked by the nonpainful targets given in the nonpain/pain and nonpain/ nonpain conditions (Newman–Keuls p4.10; Figure 4). CTN amplitude was not affected by validity (main and interaction effect Fso1, p4.10). As we reported earlier (Dowman, 2007a), the FCN is characterized by an increasing negativity in the frontocentral regions with increasing stimulus intensity (Electrode ! Intensity interaction, F[2,34] 5 2.97, p 5 .06, e 5 .27; Figure 5). Likewise, the FCN amplitude was affected by validity (Electrode ! Validity interaction, F[8,136] 5 9.90, po.01, e 5 .55), where the FCN was more negative in the invalidly cued than the validly cued condition at the frontal midline. The central-parietal positive peak that overlaps temporally with the FCN (Figure 2A) was larger in the invalidly cued than the validly cued
402
R. Dowman
A
FCN / CTN
CN
P1
P2
LEFT PAIN +
65–113 ms
CTN:121–161 ms FCN:133–161 ms
65–85 ms
CTN:125–161 ms FCN:145–165 ms
173–269 ms
277–333 ms
LEFT NP 165–253 ms
261–325 ms
165–229 ms
237–321 ms
CTN/FCN
RIGHT PAIN +
CTN:121–161 ms FCN:133–165 ms
61–105 ms
RIGHT NP 65–81 ms
CTN:125–161 ms FCN:137–161 ms
B
169–265 ms
273–337 ms
P3a LEFT
RIGHT
PAIN
NP
PAIN
341–381 ms
329–357 ms
329–357 ms
337–381 ms
325–381 ms
329–377 ms
NP
VC + 341–369 ms
IC
325–381 ms
Figure 2. A: Grand average sural nerve SEP topographic patterns for the CN, CTN, P1, and P2 components evoked by the painful sural nerve target (PAIN) and by the nonpainful sural nerve target given in the nonpain/pain (NP) condition for the left and right sides. Similar topographic patterns were obtained for the nonpainful sural nerve target given in the nonpain/nonpain condition. All of the topographies were obtained in the validly cued condition. Similar topographic patterns were obtained in the invalidly cued condition. B: Grand average sural nerve SEP topographic patterns for the P3a event-related potential obtained in the validly cued (VC) and invalidly cued (IC) conditions. The increase in the frontal positivity in the invalidly cued condition is characteristic of the P3a (see Dowman, 2004c). The numbers under each topography are the onsets and offsets of the time segments used to determine the component amplitude. The solid circles in each topography identify the recording electrode locations, where Cz 0 is the third electrode from the bottom along the sagittal midline. The lines are isovoltage contours. The scaling was adjusted in each of the topographies to best illustrate the topographic pattern.
Threat detectors and attentional bias toward pain
403
PAIN
NP
CT
NN
CT
CTN
CT
P3a
CZ’+10
CZ’+10
CZ’+10
FCN
AMPLITUDE (uV)
CZ’ 25 P2 P1 20 CN 15 10 5 CZ’–5 0 –5 –10 –15 –20 –600 –400 –200 0 200 400 TIME (ms)
CZ’
CZ’
20 15 10 5 0
CZ’–5
–5 –10
VALIDLY CUED INVALIDLY CUED
–15 –20 –600 –400 –200 0 200 TIME (ms)
400
20 15 10 5 CZ’–5 0 –5 –10 –15 –20 –600 –400 –200 0 200 TIME (ms)
400
Figure 3. Sural nerve somatosensory evoked potential (SEP) waveforms recorded from the contralateral temporal scalp (CT) and three locations along the sagittal midline: Cz 0 , 10 cm anterior to Cz 0 (Cz 0 110), and 5 cm posterior to Cz 0 (Cz 0 " 5). The left panel shows the SEPs evoked by the painful sural nerve target stimuli (PAIN), the middle SEPs evoked by the nonpainful sural nerve targets given in the nonpain/pain condition (NP), and the right SEPs evoked by the nonpainful sural nerve targets given in the nonpain/nonpain condition (NN). The sural nerve evoking stimulus was given at time 0.
condition at the central and parietal midline sites (Figure 5). The FCN validity effects were the same across the stimulus intensities (Intensity ! Validity and Electrode ! Intensity ! Validity interaction, Fso1, p4.10). The effects of stimulus intensity and validity on the P1, P2, and P3a components are shown in Figure 6. All three showed the same intensity effect (intensity main effect, Fs414.0, po.01), where the amplitude at the painful level was greater than the nonpainful level given in the nonpain/pain and the nonpain/ nonpain conditions (Newman–Keuls tests po.05), and the amplitude for the nonpainful level given in the nonpain/pain condition was larger than that given in the nonpain/nonpain condition (Newman–Keuls test po.05; Figure 6). The validity effect for P1 depended on the stimulus level and the electrode location (Electrode ! Intensity ! Validity interaction, F[8,136] 5 3.13, po.05, e 5 .30; Figure 6). At the painful level, P1 amplitude was smaller in the invalidly cued than the validly cued condition at the central (Cz 0 ) and frontocentral (Cz 0 15) midline electrodes. At the nonpainful stimulus level given in the nonpain/pain condition, P1 amplitude was slightly larger in the invalidly cued than the validly cued condition at the central midline electrode (Cz 0 ). At the nonpainful stimulus level given in the nonpain/nonpain condition, P1 amplitude was slightly smaller in the invalidly cued than the validly cued condition at the fronto-central (Cz 0 15) and frontal (Cz 0 110) midline electrodes.
Both P2 and P3a amplitudes were larger in the invalidly cued than the validly cued condition (validity main effect, Fs428.0, po.0001), and the magnitude of this effect depended on both stimulus intensity and electrode site (Electrode ! Intensity ! Validity interaction, Fs42.8, po.05; Figure 6). For P2 evoked at the painful level, the validity effect was evident at all five midline electrodes but was largest at the central (Cz 0 ), centroparietal (Cz 0 " 5 cm) and parietal (Cz 0 " 10 cm) midline. For P2 evoked at the nonpainful level given in the nonpain/pain and nonpain/ nonpain conditions, the validity effect was largest at the frontocentral (Cz 0 15 cm), central (Cz 0 ), and centroparietal (Cz 0 " 5 cm) midline. For P3a evoked at the painful level, the validity effect was the same across all of the frontal, central, and parietal midline electrodes. The validity effect for the P3a evoked at the nonpainful level given in the nonpain/pain and nonpain/ nonpain conditions was largest at the central (Cz 0 ), centroparietal (Cz 0 " 5 cm), and parietal (Cz 0 " 10 cm) midline electrodes (Figure 6). Neither P2 nor P3a showed a significant Intensity ! Validity interaction (Fso0.1, p4.10), which demonstrates that the increase in P2 and P3a amplitudes in the invalidly cued condition was the same for the painful and nonpainful sural nerve target stimuli. Indeed, post hoc two-way ANOVAs (Intensity ! Validity) run separately for P3a at the frontal (CZ 0 110) and parietal (CZ 0 " 5) midline electrodes (the sites showing the largest validity effects for the painful and nonpainful targets, respectively) and for P2 at Cz 0 revealed significant
404
R. Dowman FCN electrode x intensity 8
np
–12
7
nn
6
–10 –8 –6 –4 –2 0
5
a,b
4 a
3
a,b
2 1 0
NN
NP
PAIN
–1
PAIN 81–97ms
a,b
–2 CZ'+0
CTN
CZ'+5
CZ'
CZ'–5
CZ'–10
electrode x validity
–10
validly cued invalidly cued
validly cued invaldly cued
–8
6
–6 –4 –2 0 NN
NP
PAIN
Figure 4. The effects of stimulus intensity and cue validity on the CN (upper panel) and CTN (lower panel) amplitudes. The CN PAIN 81–97 ms is the CN amplitude measured using the 81–97-ms time segment for the painful sural nerve target stimulus. The other amplitudes were computed using the segment onset and offsets given in Figure 2. PAIN: painful sural nerve target; NP: nonpainful sural nerve target given in the nonpain/pain condition; NN: nonpainful sural nerve target given in the nonpain/nonpain condition. nNewman–Keuls test p 5 .056, comparing the invalidly and validly cued conditions.
validity main effects (Fs46, po.01) and nonsignificant Intensity ! Validity interaction effects (Fso1.2, p4.10). Possible Movement-Related Potential Contamination of the Sural Nerve SEP The mean reaction times for the spatial discrimination task occurred very soon after the offset of the grand average P3a event-related potential: between 13 and 80 ms for the validly cued condition and between 84 and 140 ms in the invalidly cued condition (Table 1). This presents the possibility that the sural nerve SEP was contaminated by movement-related potentials. In cuing paradigms such as the one used here, movement-related potentials consist of a slowly rising negative potential (referred to as the N1, Bereitschaftspotential, or readiness potential) whose onset can begin 1000–1500 ms prior to the movement onset (Coles, 1989; Kutas & Donchin, 1980). The increase in the amplitude of this negative potential begins to accelerate at about 250–300 ms prior to and peaks just after the movement onset (N2 or movement potential; Kutas & Donchin, 1980; Shibasaki, Barrett, Halliday, & Halliday, 1980). The N2 movement-related potential is followed by a positive peak at about 300 ms following
AMPLITUDE (µV)
AMPLITUDE (µV)
pain
–14
AMPLITUDE (µV)
AMPLITUDE (µV)
CN
4
2
0
CZ’+10
CZ’+5
CZ’
CZ’–5
CZ’–10
Figure 5. The effects of electrode location and stimulus intensity (upper panel) and electrode location and cue validity (lower panel) on the FCN amplitude. The x-axis is electrode location along the sagittal midline, where Cz 0 110 denotes 10 cm anterior to Cz 0 and Cz 0 " 10 denotes 10 cm posterior to Cz 0 . pain: painful sural nerve target; np: nonpainful sural nerve target given in the nonpain/pain condition; nn: nonpainful sural nerve target given in the nonpain/nonpain condition. aNewman–Keuls test po.05 between the PAIN and NP conditions; bNewman–Keuls test po.05 between the PAIN and NN conditions; nNewman–Keuls test po.05 between the validly and invalidly cued conditions.
the movement onset (Kutas & Donchin, 1980; Shibasaki et al., 1980). There is also a small positive-going inflection on the N2 peak that occurs near the movement onset (Kutas & Donchin, 1980; Shibasaki et al., 1980). In our study, the N1 movement-related potential would have been evident in the prestimulus electroencephalogram (EEG), which was clearly not the case (Figure 3). The N2 movementrelated potential is largest over the sensorimotor cortex hand area contralateral to the hand making the response (Coles, 1989; Kutas & Donchin, 1980; Shibasaki et al., 1980) and, as a consequence, is isolated by computing the difference in the potentials recorded from the scalp overlying the contralateral and ipsilateral sensorimotor hand areas (contralateral side–
Threat detectors and attentional bias toward pain validly cued invalidly cued
P1
AMPLITUDE (µV)
P2
AMPLITUDE (µV)
20 18 16 14 12 10 8 6 4 2 0 20 18 16 14 12 10 8 6 4 2 0
P3a
AMPLITUDE (µV)
PAIN 22 20 18 16 14 12 10 8 6 4 2 0
CZ’+10 CZ’+5
CZ’
405
CZ’–5 CZ’–10
NP
CZ’+10 CZ’+5
CZ’
NN
CZ’–5 CZ’–10
CZ’+10 CZ’+5
CZ’
CZ’–5 CZ’–10
Figure 6. The effects of electrode location, stimulus intensity, and cue validity on the P1 (top panels), P2 (middle panels), and P3a (bottom panels) component amplitudes. The x-axis is electrode location along the sagittal midline, where Cz 0 110 denotes 10 cm anterior to Cz 0 and Cz 0 ! 10 denotes 10 cm posterior to Cz 0 . PAIN: painful sural nerve target (left panels); NP: nonpainful sural nerve target given in the nonpain/pain condition (middle panels); NN: nonpainful sural nerve target given in the nonpain/nonpain condition (right panels). nNewman–Keuls test po.05 between the validly and invalidly cued conditions.
ipsilateral side; see Coles, 1989). This difference potential is referred to as the lateralized potential. Note that the sural nerve electrical target stimuli are asymmetrical, and, hence, some of the sensory-related components, most notably the CN and CTN, Table 1. Mean P3a Component Offsets and Spatial Discrimination Task Reaction Times (RT; in Milliseconds) Pain
Validly cued Invalidly cued
NP
NN
P3a
RT
P3a
RT
P3a
RT
369.0 359.0
382.0 443.5
363.0 381.0
442.9 519.2
373.0 383.0
431.2 502.4
Note: Pain: moderately painful target stimulus; NP: nonpainful target given in the nonpain/pain condition; NN: nonpainful target given in the nonpain/nonpain condition.
will also show up in the lateralized potential. It is not, therefore, specific to movement-related potentials. In our electrode grid, the sensorimotor hand areas are under the scalp electrodes located 5 and 10 cm lateral to Cz 0 along the coronal midline (see Dowman & Schell, 1999b). The N2 movement-related potential will be evident as a negative peak in the lateralized potential that occurs close to the mean reaction time for that stimulus intensity and validity condition (see Table 1). Lateralized potentials averaged across the left and right sural nerve target stimuli are shown in Figure 7. The initial positive peak in the lateralized potential corresponds to the asymmetry in the CN, where the scalp central negativity is oriented toward the side ipsilateral to the sural nerve evoking stimulus (see Figure 3; Dowman, 1994; Dowman & Schell, 1999b). The following negative peak corresponds to the CTN, where the temporal negativity is considerably larger on the side contralateral to the sural nerve evoking stimulus (Figure 3). There do not, however,
406
R. Dowman validly cued invalidly cued
5 cm lateral CZ’
PAIN
AMPLITUDE (µV)
25 20 NP 15 10 5 NN 0 –5 –10 –15 –20 –600 –400 –200 0 200 400 600 TIME (ms)
AMPLITUDE (µV)
PAIN
10 cm lateral CZ’
25 20 NP 15 10 5 NN 0 –5 –10 –15 –20 –600 –400 –200 0 200 400 600 TIME (ms)
Figure 7. Lateralized potentials obtained from electrode pairs located 5 cm lateral to Cz 0 (left panels) and 10 cm lateral to Cz 0 (right panels) along the coronal midline for the painful sural nerve target (PAIN) and for the nonpainful sural nerve target obtained during the nonpain/pain (NP) and nonpain/ nonpain (NN) conditions. The sural nerve evoking stimulus was given at time 0.
appear to be any peaks in the lateralized potential at latencies corresponding to the spatial discrimination response reaction times, nor do there appear to be any consistent differences between the validly and invalidly cued lateralized potentials at these latencies. Hence, it appears unlikely that movement-related potentials are responsible for the validity effects on the P2 and P3a components reported here. There are two possible reasons for this. First, our amplifier low frequency cutoff filter (0.3 Hz) was higher than that used to record movement-related potentials (e.g., 0.02 and 0.03 Hz in Kutas & Donchin, 1980, and Shibasaki et al., 1980, respectively). Second, there was a substantial amount of trial-to-trial variability in the reaction time latencies (Table 2), which would have significantly reduced the amplitude of movement-related potentials time-locked to the sural nerve evoking stimuli.
Discussion The Role of Somatic Threat Detectors in the Attentional Bias toward Pain The present study confirms our earlier report (Dowman, 2004b) that presenting a painful sural nerve target stimulus outside the focus of spatial attention does not affect the somatic threat detectors indexed by the CTN. According to our somatic threat detection and orienting hypothesis, threat detector activity is monitored by the medial prefrontal cortex (indexed by the FCN), which, in turn, signals the lateral prefrontal cortex to redirect Table 2. Mean ! 1 SD (68.3% Confidence Intervals) Spatial Discrimination Response Reaction Times (in Milliseconds) Pain
NP
NN
" 1 SD ! 1 SD " 1 SD ! 1 SD " 1 SD ! 1 SD Validly cued Invalidly cued
291.3 338.9
472.7 548.1
334.6 407.1
551.2 631.3
309.4 376.7
553.0 628.1
Note: Pain: moderately painful target stimulus; NP: nonpainful target given in the nonpain/pain condition; NN: nonpainful target given in the nonpain/nonpain condition; SD: mean of the subject reaction time standard deviations.
attention toward the threat (indexed by the P3a event-related potential; Dowman, 2007a; Dowman & ben-Avraham, 2008). Because there was no difference in threat detector activity across the validly and invalidly cued conditions, there should not have been any difference in the FCN and P3a amplitudes between the threatening and nonthreatening conditions, which was indeed the case. Rather, the validity effects observed for the FCN and P3a are comparable to those reported for weak nonpainful sural nerve target stimuli in the Dowman (2007b) modal cuing study. This validity effect was probably triggered by the transient detector activity located in the primary somatosensory cortex that is indexed by the CN component (see Dowman, 2007b; Dowman & ben-Avraham, 2008). The absence of threat effects in the reaction time data is also consistent with the idea that the somatic threat detectors indexed by the CTN play a role in the attentional bias toward somatic threats. Legrain and his colleagues have also proposed a detector-type mechanism to explain the electrophysiological signs of orienting elicited by infrequent and unexpected painful laser stimuli observed in their oddball and oddball–deviant studies (Legrain, Bruyer, Guerit, & Plaghki, 2003; Legrain et al., 2009). However, Legrain et al. propose that the orienting response is triggered by the same transient and change detector processes used to detect the unexpected, abrupt onset of a nonthreatening stimulus or change in the stimulus stream (see Escera & Corral, 2007; Friedman, Cycowicz, & Gaeta, 2001; Na¨a¨ta¨nen, 1992; Na¨a¨ta¨nen & Picton, 1987; Na¨a¨ta¨nen, Jacobsen, & Winkler, 2005). The change detectors are unlikely to play a role in the attentional bias toward somatic threats observed in the Spence et al. (2002) and Dowman (2007a,b) modal cuing studies, however, because the painful target stimuli used in the validly and invalidly cued conditions were randomly selected from the same stream of target stimuli. Furthermore, the transient and change detector activations are not specific to threatening stimuli (Dowman, 2007b), and, hence, it is not clear how they could mediate the attentional bias toward somatic threats. Differences in Somatic Threat Effects in the Spatial and Intermodal Cuing Studies The absence of a somatic threat effect in the spatial cuing studies contrasts with those reported in the modal cuing studies, where the somatic threat detector activity indexed by the CTN was
Threat detectors and attentional bias toward pain larger in the invalidly than the validly cued condition and where the reaction time validity effect was smaller for painful sural nerve target stimuli than for weak nonpainful sural nerve and visual target stimuli (Dowman, 2007a,b; Dowman & benAvraham, 2008). A similar attentional bias toward painful stimuli was reported by Spence et al. (2002) and Van Damme et al. (2002, 2004), who also used a cuing paradigm that involved shifts in modal attention. An important difference between the spatial and modal cuing studies was that, in the former, all of the target stimuli were threatening, whereas in the latter, one of the target stimuli was threatening and the other was nonthreatening. As explained in the Introduction, this difference might have resulted in a smaller ratio of attention allocated to the cued and uncued targets (i.e., the attentional set) in the spatial than in the intermodal cuing studies, which, in turn, would have resulted in a smaller, possibly nonsignificant difference in CTN threat detector activity across the validly and invalidly cued conditions. We tested this hypothesis here by presenting the weak nonthreatening sural nerve targets to one side and the painful sural nerve targets to the other. Nonetheless, we were still unable to observe enhanced threat detector activity when the painful sural nerve target was presented outside the focus of spatial attention. We also evaluated the possibility that an uncued threatening sural nerve target draws attention away from the cued side and toward the uncued side by comparing the reaction times for the validly cued nonthreatening sural nerve target stimulus when the uncued side was nonthreatening (nonpain/ nonpain condition) to when it was threatening (nonpain/pain condition). However, the data demonstrate that the uncued somatic threat does not draw more attention from the cued side than a weak nonthreatening stimulus. Hence, the different threat effects observed in the spatial and modal cuing studies cannot be attributed to attentional set. Another difference between our modal and spatial cuing studies is that the former required a change in task set (from the visual color discrimination to the sural nerve intensity discrimination) and the latter did not. However, Spence et al. (2002) demonstrated behavioral signs of the attentional bias toward somatic threat in an experiment that involved shifts in modal attention without a change in spatial attention or task set. Hence, task set is unlikely to be solely responsible for the differences in the threat effects observed in the spatial and modal cuing studies. Given the importance of responding to unattended threats to survival, it is puzzling that there is an attentional bias toward a somatic threat when it is presented outside the focus of modal attention but not spatial attention. This would make sense if attending to a specific stimulus speeds its sensory and motor processing even if attention is directed to the wrong location (i.e., all of the relevant target stimulus features are being attended to except its location). In this case an attentional bias toward threat may not be needed, as appears to be the case when the threat is validly cued. This idea is consistent with evidence from the contingent attention capture literature, which has shown that task-irrelevant distracter stimuli that share stimulus features with the target are more likely to capture attention than those that do not (Pashler, Johnston, & Ruthruff, 2001; Remington, Folk, & McLean, 2001). The contingent capture mechanism is presumably mediated by top-down excitatory biases exerted by the prefrontal cortex on sensory areas representing the target-related stimulus features (Desimone & Duncan, 1995; D’Esposito, 2007; Miller & Cohen, 2001; Muller & Knight, 2006; Shimamura, 2000). Recent studies in the visual system have shown that
407 attending to one stimulus feature (e.g., color) results in a global increase in the sensory activation for that feature irrespective of its spatial location (Yantis, 2008). Hence, the top-down excitatory biases on the stimulus feature representations do not require congruence with spatial location. According to this model, unattended stimuli capture attention via the bidirectional connections between the sensory areas and the prefrontal cortex and by the competition between the prefrontal areas representing the different stimuli (Desimone & Duncan, 1995; Miller & Cohen, 2001). Therefore, an unattended stimulus with features similar to the target will have a larger activation of its sensory representation than an unattended stimulus that does not. This, in turn, will produce a larger activation of its prefrontal cortex representation and greater inhibition of prefrontal cortex areas representing competing stimuli. The uncued and cued targets had more stimulus features in common in our spatial cuing than our modal cuing studies. The contingent attention capture mechanism should, therefore, have resulted in faster shifts in attention to the invalidly cued target in the spatial cuing study, which would be evident as a smaller reaction time validity effect. Indeed, the reaction time validity effects for the nonthreatening and threatening sural nerve target stimuli in the Dowman (2007a,b) modal cuing studies were 132 ms and 100 ms, respectively, whereas they were 68 ms and 61 ms, respectively, in the spatial cuing study described here and 63 ms for the threatening sural nerve target stimuli in the Dowman (2004b) spatial cuing study. Post hoc analyses comparing the validity effects between the modal and spatial cuing studies were significant for both the painful, t(51) 5 2.48, po.05, and nonpainful sural nerve targets, t(36) 5 3.80, po.001. The absence of a threat effect on the electrophysiological or the reaction time data in the spatial cuing studies suggests that the contingent capture mechanism is supplanting the threat detector mechanism and not simply adding to it. This is consistent with the excitatory bias on the sural nerve electrical target’s feature representations being accompanied by an inhibitory bias on the threat detector suggested by our electrophysiological studies (Dowman & ben-Avraham, 2008). P2 and Orienting Spatial Attention Our studies provide converging evidence that at least some of the P2 component generators are involved in stimulus-driven shifts in spatial attention. P2 amplitude was larger when the sural nerve target stimulus was presented outside the focus of spatial attention (Dowman, 2004b; present study), but not when it was presented outside the focus of modal attention (Dowman, 2007a, b). These effects were the same for both the nonthreatening and the threatening sural nerve target stimuli. Hence, the P2 generators do not appear to be involved in the attentional bias toward somatic threats. Intracranial recording studies have reported P2-like activity in a number of cortical areas, including the anterior cingulate cortex, dorsolateral prefrontal cortex, the supplementary somatosensory area, and the inferior parietal cortex (Dowman et al., 2007; Lenz et al., 1998; Ohara, Crone, Weiss, Treede, & Lenz, 2004). Of particular interest is the inferior parietal cortex, given that its response properties are very similar to those of the scalprecorded P2. Like the P2, task-relevant visual stimuli presented outside the focus of spatial attention produce an increase in inferior parietal cortex activity (Constantinidis & Steinmetz, 2005; Desimone & Duncan, 1995; Kincaide, Abrams, Astafiev, Shulman, & Corbetta, 2005) that appears to play an important
408
R. Dowman
role in stimulus-driven shifts in spatial attention (Corbetta, Patel, & Shulman, 2008). Inferior parietal cortex activity depends on task relevance, where its activation is greater for task-relevant than task-irrelevant stimuli (Corbetta & Schulman, 2002; Corbetta et al., 2008; Kincaide et al., 2005), as is also the case for P2 (Dowman, 2004a; Lorenz & Garcia-Larrea, 2003). Together, these data suggest that P2 indexes inferior parietal cortex activity that is involved in stimulus-driven shifts in spatial attention. The P2 generators may be involved in other orienting-related processes as well. For example, the P2 evoked by painful laser stimuli has been shown to increase when the evoking stimulus is infrequent and unexpected, an effect that is thought to be triggered by change detectors activated by a change in the stimulus stream (Legrain, Bruyer, Guerit, & Plaghi, 2005;
Legrain, Guerit, Bruyer, & Plaghki, 2002; Legrain et al., 2003, 2009). Interestingly, this effect has been reported when the laser evoking stimulus is presented both within and outside focus of spatial attention. Conclusions The results of this experiment in conjunction with our earlier studies suggest that somatic threat detector activity is enhanced when the threat is presented outside the focus of modal but not spatial attention. This discrepancy does not appear to be related to differences in attentional set, but rather may be due to differences in contingent attention capture. The results also verify earlier studies suggesting that the P2 component’s inferior parietal cortex generator is involved in stimulus-driven shifts in spatial attention.
REFERENCES Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S., & Cohen, J. D. (2001). Conflict monitoring and cognitive control. Psychological Review, 108, 624–652. Cisler, J. M., Bacon, A. K., & Williams, N. L. (2009). Phenomenological characteristics of attentional biases towards threat: A critical review. Cognitive Therapy and Research, 33, 221–234. Coles, M. G. H. (1989). Modern mind-brain reading: Psychophysiology, physiology, and cognition. Psychophysiology, 26, 251–269. Constantinidis, C., & Steinmetz, M. A. (2005). Posterior parietal cortex automatically encodes the location of salient stimuli. Journal of Neuroscience, 25, 233–238. Corbetta, M., Patel, G., & Shulman, G. L. (2008). The reorienting system of the human brain: From environment to theory of mind. Neuron, 58, 306–324. Corbetta, M., & Schulman, G. L. (2002). Control of goal-directed and stimulus driven attention in the brain. Nature Reviews Neuroscience, 3, 201–215. Craig, A. D. (2002). How do you feel? Interoception: The sense of physiological condition of the body. Nature Reviews Neuroscience, 3, 655–666. Craig, A. D. (2003). Interoception: The sense of the physiological condition of the body. Current Opinion in Neurobiology, 13, 500–505. Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual Reviews of Neuroscience, 18, 193– 222. D’Esposito, M. (2007). From cognitive to neural models of working memory. Philosophical Transactions of the Royal Society, B, 362, 761– 772. Dowman, R. (1994). SEP topographies elicited by innocuous and noxious sural nerve stimulation. I. Identification of stable periods and individual differences. Electroencephalography and Clinical Neurophysiology, 92, 291–302. Dowman, R. (1997). Interstimulus interval has no effect on a mid-latency scalp potential generated by innocuous-related activity in the primary somatosensory cortex. Brain Topography, 10, 145–154. Dowman, R. (2001). Attentional set effects on human innocuous somatosensory and pain pathways. Psychophysiology, 38, 451–464. Dowman, R. (2004a). Distraction produces an increase in pain-evoked anterior cingulate activity. Psychophysiology, 41, 613–624. Dowman, R. (2004b). Electrophysiological indices of orienting attention towards pain. Psychophysiology, 41, 749–761. Dowman, R. (2004c). The pain-evoked P2 peak is not a P3a event-related potential. Brain Topography, 17, 3–12. Dowman, R. (2007a). Neural mechanisms of detecting and orienting attention towards unattended threatening somatosensory targets. I. Modality effects. Psychophysiology, 44, 407–419. Dowman, R. (2007b). Neural mechanisms of detecting and orienting attention towards unattended threatening somatosensory targets. II. Intensity effects. Psychophysiology, 44, 420–430. Dowman, R., & ben-Avraham, D. (2008). An artificial neural network model of orienting attention towards threatening somatosensory stimuli. Psychophysiology, 45, 229–239.
Dowman, R., & Darcey, T. M. (1994). SEP topographies elicited by innocuous and noxious sural nerve stimulation. III. Dipole source analysis. Electroencephalography and Clinical Neurophysiology, 92, 373–391. Dowman, R., Darcey, T. M., Barkan, H., Thadani, V., & Roberts, D. (2007). Human intracranially-recorded cortical responses evoked by painful electrical stimulation of the sural nerve. NeuroImage, 34, 743– 763. Dowman, R., & Goshko, L. (1992). Evaluation of reference sites for scalp potentials elicited by painful and non-painful sural nerve stimulation. Electroencephalography and Clinical Neurophysiology, 84, 477–485. Dowman, R., & Schell, S. (1999a). Evidence that the anterior cingulate and supplementary somatosensory cortices generate the pain-related negative difference potential. Clinical Neurophysiology, 110, 2117– 2126. Dowman, R., & Schell, S. (1999b). Innocuous-related sural nerve-evoked activity and finger-evoked potentials generated in the primary somatosensory and supplementary motor cortices. Clinical Neurophysiology, 110, 2104–2116. Eccleston, C. (1994). Chronic pain and attention: A cognitive approach. British Journal of Clinical Psychology, 33, 535–547. Eccleston, C. (1995). Chronic pain and distraction: An experimental investigation into the role of sustained and shifting attention in the processing of chronic persistent pain. Behavioral Research and Therapy, 33, 391–405. Eccleston, C., & Crombez, G. (1999). Pain demands attention: A cognitive-affective model of the interruptive function of pain. Psychological Bulletin, 125, 356–366. Escera, C., & Corral, M. J. (2007). Role of mismatch negativity and novelty-P3 in involuntary auditory attention. Journal of Psychophysiology, 21, 251–264. Friedman, D., Cycowicz, Y. M., & Gaeta, H. (2001). The novelty P3: An event-related brain potential (ERP) sign of the brain’s evaluation of novelty. Neuroscience and Biobehavioral Reviews, 25, 355–373. Gatchel, R. J., Peng, Y. B., Peters, M. L., Fuchs, P. N., & Turk, D. C. (2007). The biopsychosocial approach to chronic pain: Scientific advances and future directions. Psychological Bulletin, 133, 581–624. Halgren, E., Marinkovic, K., & Chauvel, P. (1998). Generators of the late cognitive potentials in auditory and visual oddball tasks. Electroencephalography and Clinical Neurophysiology, 106, 156–164. Kincaide, J. M., Abrams, R. A., Astafiev, V., Shulman, G. L., & Corbetta, M. (2005). An event-related functional magnetic imaging study of voluntary and stimulus-driven orienting of attention. Journal of Neuroscience, 25, 4593–4604. Kuhajda, M. C., Thorn, B. E., Klinger, M. R., & Rubin, N. J. (2002). The effect of headache pain on attention (encoding) and memory (recognition). Pain, 97, 213–221. Kutas, M., & Donchin, E. (1980). Preparation to respond as manifested by movement-related brain potentials. Brain Research, 202, 95–115. Legrain, V., Bruyer, R., Guerit, J.-M., & Plaghki, L. (2003). Nociceptive processing in the human brain of infrequent task-relevant and task
Threat detectors and attentional bias toward pain irrelevant noxious stimuli. A study with event-related potentials evoked by CO2 laser radiant heat stimuli. Pain, 103, 237–248. Legrain, V., Bruyer, R., Guerit, J.-M., & Plaghi, L. (2005). Involuntary orientation of attention to unattended deviant nociceptive stimuli is modulated by concomitant visual task difficulty. Evidence from laser evoked potentials. Clinical Neurophysiology, 116, 2165–2174. Legrain, V., Guerit, J.-M., Bruyer, R., & Plaghki, L. (2002). Attentional modulation of nociceptive processing into the human brain: Selective attention, probability of stimulus occurrence, and target detection effects on laser evoked potentials. Pain, 99, 21–39. Legrain, V., Van Damme, S., Eccleston, C., Davis, K. D., Seminowicz, D. A., & Crombez, G. (2009). A neurocognitive model of attention to pain: Behavioral and neuroimaging evidence. Pain, 144, 230–232. Lenz, F. A., Rios, M., Zirh, A., Chau, D., Krauss, G., & Lesser, R. P. (1998). Painful stimuli evoke potentials recorded over the human anterior cingulate gyrus. Journal of Neurophysiology, 79, 2231–2234. Lorenz, J., & Garcia-Larrea, L. (2003). Contribution of attentional and cognitive factors to laser evoked brain potentials. Neurophysiologie Clinique/Clinical Neurophysiology, 33, 293–301. Lu, C.-H., & Proctor, R. W. (1995). The influence of irrelevant location information on performance: A review of the Simon and spatial Stroop effects. Psychonomic Bulletin and Review, 2, 174–207. Miller, E. K., & Cohen, J. D. (2001). An integrative theory of prefrontal cortex function. Annual Reviews of Neuroscience, 24, 167–202. Muller, N. G., & Knight, R. T. (2006). The functional neuroanatomy of working memory: Contributions of human brain lesion studies. Neuroscience, 139, 51–58. Na¨a¨ta¨nen, R. (1992). Attention and brain function. Hillsdale, NJ: Erlbaum. Na¨a¨ta¨nen, R., Jacobsen, T., & Winkler, I. (2005). Memory-based or afferent processes in mismatch negativity (MMN): A review of the evidence. Psychophysiology, 42, 25–32. Na¨a¨ta¨nen, R., & Picton, T. (1987). The N1 wave of the human electric and magnetic response to sound: A review and an analysis of the component structure. Psychophysiology, 24, 375–425. Norman, D. A., & Shallice, T. (1986). Attention to action. In R. J. Davison, G. E. Schwartz, & D. Shapiro (Eds.), Consciousness and self-regulation. Advances in research and therapy (pp. 1–18). New York: Plenium Press. Nunez, P. L. (1981). Electric fields of the brain. New York: Oxford University Press. Ohara, S., Crone, N. E., Weiss, N., Treede, R.-D., & Lenz, F. A. (2004). Amplitudes of laser evoked potentials recorded from primary somatosensory, parasylvian and medial frontal cortex are graded with stimulus intensity. Pain, 110, 318–328. O¨hman, A. (2000). Fear and anxiety: Evolutionary, cognitive, and clinical perspectives. In M. Lewis & J. M. Haviland-Jones (Eds.), Handbook of emotions (pp. 573–593). New York: Guilford Press. O¨hman, A. (2005). The role of the amygdala in human fear: Automatic detection of threat. Psychoneuroendocrinology, 30, 953–958. O¨hman, A., & Mineka, S. (2001). Fears, phobias, and preparedness: Toward an evolved module of fear and fear learning. Psychological Review, 108, 483–522.
409 Pashler, H., Johnston, J. C., & Ruthruff, E. (2001). Attention and performance. Annual Review of Psychology, 52, 629–651. Pincus, T., & Morley, S. (2001). Cognitive-processing bias in chronic pain: A review and integration. Psychological Bulletin, 127, 599–617. Posner, M. I., & DiGiorlamo, G. J. (1998). Executive function: Conflict, target detection, and cognitive control. In R. Parasuraman (Ed.), The attentive brain (pp. 401–423). Cambridge MA: MIT Press. Posner, M. I., Snyder, C. R. R., & Davidson, B. J. (1980). Attention and the detection of signals. Journal of Experimental Psychology: General, 109, 160–174. Remington, R. W., Folk, C. L., & McLean, J. P. (2001). Contingent attentional capture or delayed allocation of attention? Perception and Psychophysics, 63, 298–307. Scherg, M. (1990). Fundamentals of dipole source potential analysis. In G. Grandori, M. Hoke, & G. L. Romani (Eds.), Auditory evoked magnetic and electric potentials (pp. 40–69). Basel: Karger. Sharbrough, F., Chatrian, G. E., Lesser, R. P., Luders, H., Nuwer, M., & Picton, T. W. (1991). Guidelines for standard electrode position nomenclature. Journal of Clinical Neurophysiology, 8, 200–202. Shibasaki, H., Barrett, G., Halliday, E., & Halliday, A. M. (1980). Components of the movement-related cortical potential and their scalp topography. Electroencephalography and Clinical Neurophysiology, 49, 213–226. Shimamura, A. P. (2000). The role of the prefrontal cortex in dynamic filtering. Psychobiology, 28, 207–218. Soltani, M., & Knight, R. T. (2000). Neural origins of the P300. Critical Reviews in Neurobiology, 14, 199–224. Spence, C., Bentley, D. E., Phillips, N., McGlone, F. P., & Jones, A. K. P. (2002). Selective attention to pain: A psychophysical investigation. Experimental Brain Research, 145, 395–402. Turk, D. C., & Okifuji, A. (2002). Psychological factors in chronic pain: Evolution and revolution. Journal of Consulting and Clinical Psychology, 70, 678–690. Van Damme, S., Crombez, G., & Eccleston, C. (2002). Retarded disengagement from pain cues: The effects of pain catastrophizing and pain expectancy. Pain, 100, 111–118. Van Damme, S., Crombez, G., Eccleston, C., & Goubert, L. (2004). Impaired disengagement from threatening cues of impending pain in a cross modal cueing paradigm. European Journal of Pain, 8, 227–236. Veldhuijzen, D. S., Kenemans, J. L., van Wijck, A. J. M., Oliver, B., Kalkman, C. J., & Volkerts, E. R. (2006). Processing capacity in chronic pain patients: A visual event-related potentials study. Pain, 121, 60–68. Yantis, S. (2008). The neural basis of selective attention. Current Directions in Psychological Science, 17, 86–90. Yeung, N., Botvinick, M. W., & Cohen, J. D. (2004). The neural basis of error detection: Conflict monitoring and the error-related negativity. Psychological Review, 111, 931–959.
(Received August 27, 2009; Accepted April 13, 2010)
Psychophysiology, 48 (2011), 410–414. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01071.x
BRIEF REPORT
Postauricular and superior auricular reflex modulation during emotional pictures and sounds
STEPHEN D. BENNING Department of Psychology, Vanderbilt University, Nashville, Tennessee
Abstract The postauricular reflex is a relatively new psychophysiological measure of appetitive emotional processing during picture viewing. However, the degree to which other auricular (i.e., superior and anterior auricular) muscles might exhibit reflexive activity congruent with that found in the postauricular muscle has not been investigated, nor has the robustness of postauricular reflex modulation across stimulus modality. In this study, postauricular reflexes were the only reflexes that showed consistent emotional modulation across ears and genders. Additionally, postauricular reflexes were significantly modulated for both emotional pictures and sounds; in both cases, postauricular reflexes were greatest during pleasant stimuli. Descriptors: Postauricular reflex, Auricular, Startle, Emotion, Pictures, Sounds
response to clicks). The facial nerve also appears to be the efferent pathway to the postauricular muscle in generating the postauricular reflex, as postauricular reflexes are abolished on the side ipsilateral to intracranial facial nerve palsies of various etiologies but not in patients with extracranial facial nerve palsies (Bochenek & Bochenek, 1976). Indeed, postauricular muscle activity can be used as an indicator of facial nerve conduction velocities (De Meirsman, Claes, & Geerdens, 1980), and postauricular reflexes are often the first electromyographic (EMG) responses to recover from facial Bell’s palsy (Serra, Tugnoli, Cristofori, Eleopra, & De Grandis, 1986). However, the circuitry of the postauricular reflex between the cochlea and the facial nerve remains unstudied. Benning, Patrick, and Lang (2004) investigated the effects of emotional picture presentation on the postauricular reflex in undergraduate males. The magnitude of the postauricular reflex was generally smaller during pictures than during intertrial intervals (ITIs), indicating that perceptual engagement with a foreground stimulus generally inhibited the reflex (cf. Hackley, Woldorff, & Hillyard, 1987). In opposition to this effect, the magnitude of the postauricular reflex was potentiated during pleasant pictures and inhibited during aversive pictures, yielding a linear pattern of emotional modulation opposite that observed for the startle blink reflex. These results were not attributable to differences in baseline postauricular EMG activity, as baseline postauricular EMG activity did not differ during pleasant, neutral, and aversive pictures. Additionally, these modulatory effects were stronger for high-intensity affective pictures than for lowintensity pictures (cf. Cuthbert, Bradley, & Lang, 1996). These effects have been replicated in undergraduates (Gable & Harmon-Jones, 2009; Sandt, Sloan, & Johnson, 2009) and late pubertal (but not early pubertal) adolescents (Quevedo, Benning, Gunnar, & Dahl, 2009). They have also been extended in a study demonstrating that only happy facial expressions posed by
The postauricular reflex is a vestigial muscle response in humans that acts to pull the ear upward and backward (Berzin & Fortinguerra, 1993; Gray, 1901/1995) and is recorded from electrodes positioned over the postauricular muscle behind the ear (O’Beirne & Patuzzi, 1999). Kiang, Crist, French, and Edwards (1963) first isolated the postauricular reflex as an electrical potential evoked by noise clicks in awake humans. This potential was also shown to be myogenic in origin, suggesting that the moniker ‘‘postauricular reflex’’ is an appropriate one: Its magnitude increased with forward pressure on the back of the scalp to tense the postauricular muscle, decreased substantially with backward pressure on the forehead to relax the muscle, and was abolished when the postauricular muscle was injected with the muscular paralytic curare (Bickford, Jacobson, & Cody, 1964). Further supporting the myogenic origin of the postauricular reflex is the finding that the magnitude of the postauricular reflex recorded over the skin is proportional to the number of postauricular muscle motor units activated, as measured by transdermal recordings (De Grandis & Santoni, 1980). Nevertheless, it is important to note that this reflex is vestigial and does not generate movement of the pinna. Based on studies of individuals with severe sensorineural hearing loss, the postauricular reflex appears to originate from the cochlea, not the vestibular system (Yoshie & Okudairi, 1969; but see Cody, Jacobson, Walker, & Bickford, 1964, for data regarding the possible vestibular origin of the similar inion
This work was completed in partial fulfillment for the doctoral degree of Stephen D. Benning from the University of Minnesota. Funding for this work was provided by the National Institutes of Mental Health Ruth L. Kirschstein National Research Service Award MH070104-01. Address correspondence to: Stephen D. Benning, PMB 407817, 2301 Vanderbilt Place, Nashville, TN 37240. E-mail: s.benning@ vanderbilt.edu 410
Auricular reflexes and emotion women significantly potentiated postauricular reflex magnitudes, congruent with a functional equivalence hypothesis of facial expressivity (Hess, Sabourin, & Kleck, 2007). Additionally, postauricular reflexes are greater during anticipation of gustatory reward than punishment (Hackley, Mun˜oz, Hebert, ValleIncla´n, & Vila, 2009), indicating that postauricular reflexes may be measures of appetitive processing in the absence of a foreground emotional stimulus. Nevertheless, the validity of the postauricular reflex as a measure of appetitive processing during emotional sounds has yet to be studied. This pattern of valence modulation is also counterintuitive, given that the postauricular muscle may have acted defensively to pull the ear back to protect hearing when the muscle’s function was more than vestigial (Berzin & Fortinguerra, 1993; Cassella & Davis, 1986), and decreased tonic postauricular muscle activity is involved in the orienting response (Stekelenburg & van Boxtel, 2001). Thus, postauricular reflex activity may be an artifact of cross-talk from another auricular muscle, the function of which is more obviously appetitive than the postauricular muscle. The two additional extrinsic auricular muscles may be promising targets. Because the superior auricular muscle acts to perk up the ear rather than flatten it against the head, and the anterior auricular muscle also pulls the ear forward (Gray, 1901/1995), it would be sensible to expect that these auricular muscles might also show appetitive reflexive modulation. It is notable that, of the three extrinsic auricular muscles, the superior auricular muscle is largest, followed by the postauricular muscle, with the anterior auricular muscle being smallest. Thus, there may be mean level differences in reflex magnitude based on the sheer area of the muscles. Assessing the reflexive activity of these muscles may also narrow which branch of the facial nerve is responsible for generating the appetitive modulation of the postauricular reflex. Whereas the postauricular muscle is innervated by the postauricular branch of the facial nerve, the superior and anterior auricular muscles are instead innervated by the temporal branch of the facial nerve (Sataloff & Selber, 2003). Thus, if only postauricular reflexes are potentiated during pleasant vs. neutral pictures, then a relatively restricted branch of the facial nerve (namely, its postauricular branch) would be implicated in its emotional modulation. Conversely, if other auricular reflexes are also potentiated during pleasant vs. neutral pictures, then either multiple branches of the facial nerve conduct activity related to appetitive processing, or the neural activity responsible for potentiating the auricular reflexes arises at a point upstream from the postauricular branch of the facial nerve. There have been no reports in the literature about the reflexive activity of the superior auricular or anterior auricular muscles. Thus, the current study explored whether the superior auricular and anterior auricular muscles also show reflexive activity to noise probes, and whether reflexes observed in these muscles would show a pattern of modulation by emotional stimuli similar to that for the postauricular reflex. This study also explored whether these auricular muscles had comparable patterns of modulation during emotional pictures and sounds.
Method Participants Participants in this study were 38 undergraduates (21 women) from an introductory psychology course. Four participants were excluded because they failed to show significant reactivity to the
411 startle probes (n 5 2) or due to equipment failure (n 5 2), leaving a final sample of 34 participants (19 women). Stimuli The startle probe was a bilateral 50 ms, 105 dB white noise probe with nearly instantaneous rise time; startle probes were presented 3000, 4000, or 5000 ms after picture onset. A total of 55 pictures from the International Affective Picture System (IAPS; CSEANIMH, 1999) were used in the experiment.1 Maximally intense exemplars of pleasant and aversive picture contents depicting stimuli that are directly (pleasant: erotic, food; aversive: mutilation, threat) or indirectly (pleasant: adventure, nurturant; aversive: disgust, victim) related to an organism’s survival were included, and all picture contents were gender balanced on dimensions of normatively rated valence (median t(6) between men and women 5 0.35, p 5 .735) and arousal (median t(6) between men and women 5 0.13, p 5 .897). The first three pictures (IAPS numbers 4650, 7080, and 9252) were startled at the beginning of the experiment to habituate abnormally large initial startle magnitudes (Graham, 1979); data from these pictures were not analyzed. Startle probes were not presented during four pictures during the experiment (IAPS numbers 2220, 5460, 7233, and 8485); instead, probes were presented during the ITI after these pictures to examine the magnitude of postauricular reflexes in the absence of a visual foreground and to reduce the predictability of the startle probes. Furthermore, 16 sounds from the IADS were presented throughout the experiment. Because there are many fewer sounds in the IADS than there are pictures in the IAPS, examining content categories of sounds was not possible. Therefore, four sounds of each valence (pleasant, neutral, and aversive) were probed. Additionally, four sounds (IADS numbers 115, 311, 352, and 425) were probed during the ITI. All sounds were genderbalanced on normative ratings of valence (median t(6) 5 0.17, p 5 .874) and arousal (median t(6) 5 0.35, p 5 .735). Pleasant and aversive sounds were chosen to be maximally extreme in valence and high in arousal, and neutral stimuli were chosen to be near the midpoint of the valence scale and low in arousal. A total of eight run orders were used in this study: Four different serial positions of the stimuli were used, with appropriate stimulus substitutions made for women and men in the study. In each run order, no more than two stimuli of the same valence occurred contiguously, and pictures of the same content did not follow each other. Psychophysiology All physiological channels were sampled at 2000 Hz with a NeuroScan NuAmps bioamplifier (Compumedics, Charlotte, NC) at direct current (DC) with a 500 Hz low-pass filter to avoid 1 The following IAPS pictures comprised each content category: adventure: 5623, 8034, 8180, 8210; nurturant: (1811, 2071, 2160, 2340/ 1463, 1722, 2341, 2655); erotic: 4640, 4660, 4680, (4255/4572); food: 7200, 7230, 7260, 7460; buildings: 5731, 7180, 7490, 7491; humans: 2190, 2393, 2870, 2890; landscapes: 5120, 5390, 5740, 9210; objects: 7002, 7004, 7034, (7031/7038); disgust: 9342, 9520, 9560, 9830; mutilation: (3051, 3061, 9253, 9420/9042, 9265, 9440, 9490); threat: 6250, 6260, 9630, (6243/6190); victim: 6570, 9920, (6312, 6540/6530, 6561). Pictures not in parentheses were presented to participants of either gender; pictures within parentheses to the left of the slash were presented only to men, and those within parentheses to the right of the slash were presented only to women. Likewise, the following IADS stimuli comprised each sound valence: pleasant: 215, 815, (205, 226/201, 220); neutral: 358, 704, 708, 726; aversive: 286, 290, (277, 600/261, 501).
412 aliasing of the physiological signals. Recordings of the EMG activity in the postauricular, superior auricular, and anterior auricular muscles were taken from each ear. Postauricular electrodes were placed according to locations detailed in O’Beirne and Patuzzi (1999). Superior auricular and anterior auricular electrode placements were guided by Berzin and Fortinguerra (1993) and Gray (1901/1995). Superior auricular muscle activity was recorded with one electrode on the scalp by the tendon of insertion of the muscle and the other electrode placed immediately posterior to it. Anterior auricular muscle activity was recorded with one electrode placed anterior to the tragus (the cartilaginous protrusion over the ear canal), and the other electrode was placed immediately anterior to the first electrode. Because the postauricular reflex is a microreflex, reflexive auricular muscle activity was assessed using aggregate rectified waveforms. Auricular EMG activity to noise probes was averaged across all pictures of a given content category, yielding average waveforms composed of 4 trials. In each aggregation, auricular reflex magnitudes were assessed as the peak EMG activity occurring 8–30 ms after the onset of the startle probe minus the mean 50 ms pre-probe EMG baseline activity (Benning et al., 2004). Visual inspection confirmed the appropriateness of this window for scoring these reflexes (cf. Sandt et al., 2009). Because of the large inter-individual variation in responses to startle probes, auricular reflex magnitudes were then z-scored within each participant for valence and content analyses. Procedure Participants completed a consent form and a biographical questionnaire to screen for hearing and visual impairments before being escorted into the laboratory. They then completed the study questionnaires in the order noted above as the electrodes were attached. Once hookups were completed, participants were told to follow the directions on the screen, keeping as still as possible at all times. Participants were instructed to watch each picture the entire time it appeared on the screen and to keep their gaze directed toward the fixation cross whenever no picture was on the screen. They were also told that they would hear brief noises through the headphones which they could simply ignore. They were then presented with the three habituation pictures and given a demonstration of the ratings procedure they would use to rate each picture. After the demonstration, participants attended to each stimulus in the sequence determined by the run order. Pictures and sounds were preceded by a 3-s baseline consisting of a blank screen with a fixation point; each was presented for 6 s, followed by a 3-s recovery period, during which the ITI startle probes were presented halfway through. After each trial, participants completed valence and arousal ratings of their current emotional state using the Self-Assessment Manikin (SAM; Bradley & Lang, 1994) via computer keyboard, and the ratings were followed by a blank screen lasting 3 s to allow participants to prepare for the next picture or sound. After participants viewed and rated all pictures and sounds, they received experimental debriefing and compensation. Data Analysis To investigate the relative strength of each auricular response to the noise probes, the mean raw peak magnitude for the anterior, superior, and posterior auricular reflexes of each ear during pictures were entered into a mixed ANOVA, with gender as the between-subjects factor and ear and auricular muscle as the within-subjects factors. Only the auricular muscle main effect
S. D. Benning was significant, F(2,29) 5 7.01, p 5 .002, partial Z2 5 .350 (all other Fso2.5, ps4.1, partial Z2so.15). Helmert contrasts revealed that postauricular reflexes (M 5 13.9 mV, SE 5 3.16) were larger than superior auricular reflexes (M 5 2.74 mV, SE 5 0.41) and anterior auricular reflexes (M 5 1.68 mV, SE 5 0.08), F(1,30) 5 14.3, p 5 .001, partial Z2 5 .322; superior auricular reflexes were also greater than anterior auricular reflexes, F(1,30) 5 6.78, p 5 .014, partial Z2 5 .184. However, as evidenced in Figure 1A, visual inspection of the grand average waveforms for each auricular reflex by ear revealed that only the postauricular and superior auricular waveforms had reflexive activity that could be discerned from that of the background EMG. Thus, only these reflex magnitudes were analyzed further. Postauricular and superior reflex magnitudes were entered into mixed ANOVAs, with gender as the between-subjects factor and ear and valence as the within-subjects factors with planned follow-up polynomial contrasts. A critical a level of .05 was used for all analyses, and within-subjects degrees of freedom were adjusted using the Huyhn-Feldt correction for non-sphericity. Results Postauricular Reflexes As shown in Figure 1B, there was a significant effect of picture valence on postauricular reflex magnitude, F(1.92,61.3) 5 4.67, p 5 .014, partial Z2 5 .127. In particular, postauricular reflexes were greater during pleasant pictures than during aversive pictures, linear F(1,32) 5 9.81, p 5 .004, partial Z2 5 .235 (quadratic F(1,32) 5 1.97, p 5 .170, partial Z2 5 .058). None of the other main effects or interactions were significant, Fso1.1, ps4.35. Compared to those during neutral pictures (M 5 ! 0.03, SE 5 0.06), postauricular reflexes were larger during erotic pictures (M 5 0.39, SE 5 0.14), t(33) 5 2.52, p 5 .017, and during nurturant pictures (M 5 0.28, SE 5 0.13), t(33) 5 2.09, p 5 .044. There were no significant differences between postauricular reflexes during any aversive picture contents and those during neutral pictures, tso1, ps4.3. Likewise, postauricular reflexes during sounds were modulated by sound valence, F(1.99,63.8) 5 4.03, p 5 .023, partial Z2 5 .112. Postauricular reflexes were greater during pleasant sounds than during aversive sounds, linear F(1,32) 5 6.05, p 5 .020, partial Z2 5 .159; postauricular reflex magnitudes during pleasant and aversive sounds also tended to be greater than those during neutral sounds, quadratic F(1,32) 5 3.52, p 5 .070, partial Z2 5 .099. Across both stimulus modalities, all other main effects and interactions were not significant, Fso1.5, ps4.2. Thus, in the following analyses, postauricular reflexes were averaged across ears and genders. Analyses of the consistency of the observed valence modulation pattern for pictures (cf. Benning et al., 2004) revealed that 82% of participants (95% confidence interval 66%–92%) had numerically larger postauricular reflexes during pleasant than during aversive pictures. However, only 62% of participants (95% confidence interval 45%–76%) had greater postauricular reflexes during pleasant than during neutral pictures. During emotional sounds, 65% of participants (95% confidence interval 48%–79%) showed numerically greater postauricular reflexes during pleasant sounds than either neutral or aversive sounds. An ANOVA in which mean postauricular reflexes during stimuli and during ITIs were compared showed that postauricular reflexes during ITIs (M 5 0.14, SE 5 0.09) were greater than those during stimuli (M 5 ! 0.13, SE 5 0.04), F(1,33) 5 5.23,
Auricular reflexes and emotion
A
413 Anterior Auricular − Left
Anterior Auricular − Right 2
2 Pleasant pictures Neutral pictures Aversive pictures Pleasant sounds Neutral sounds Aversive sounds
1.5 1 0.5
1.5 1 0.5 0
0 Superior Auricular - Left
M 2 i c 1.5 r o 1 v o l 0.5 t s 0
Superior Auricular - Right 2 1.5 1 0.5 0
Postauricular - Left
Postauricular - Right
10
10
5
5
0
–20
0
20
40
0
60
–20
0
20
40
60
ms relative to startle onset
B
.4 .2 .0 –.2 –.4 –.6 –.8
Pictures Sounds
Pleasant
.0 –.2 –.4 –.6
Neutral
RIGHT
.2 z score
z score
.4
LEFT
Aversive
–.8
Pictures Sounds
Pleasant
Neutral
Aversive
Figure 1. (A) Grand average auricular reflex magnitude by picture and sound valence, separated by ear of recording. (B) Mean within-subject z scored postauricular reflex magnitude by picture and sound valence, separated by ear of recording. Error bars represent the standard error of the mean.
p 5 .029, partial Z2 5 .137. Postauricular reflexes during or after pictures (M 5 0.20, SE 5 0.08) were also greater than those during or after sounds (M 5 ! 0.19, SE 5 0.09), F(1,33) 5 7.40, p 5 .010, partial Z2 5 .183. However, there was no interaction between probe time (during stimulus vs. ITI) and stimulus modality (picture vs. sound), F(1,33) 5 0.08, p 5 .785, partial Z2 5 .002. Superior Auricular Reflexes For superior auricular reflexes during pictures, the strongest effect was the three-way Valence " Ear " Gender interaction, F(2.00,62.0) 5 6.68, p 5 .002, partial Z2 5 .177. For men, the left ear’s superior auricular reflexes were greater during pleasant (M 5 0.16, SE 5 0.09) than during aversive pictures (M 5 ! 0.19, SE 5 0.09), linear Valence F(1,13) 5 7.48, p 5 .017, partial Z2 5 .365, with those during neutral pictures
falling in between (M 5 ! 0.10, SE 5 0.11), quadratic Valence F(1,13) 5 0.29, p 5 .601, partial Z2 5 .022. However, in the right ear, the opposite pattern was found, with superior auricular reflexes greater during aversive pictures (M 5 0.35, SE 5 0.13) than during pleasant pictures (M 5 ! 0.16, SE 5 0.12), linear Valence F(1,13) 5 5.14, p 5 .041, partial Z2 5 .283, with those during neutral pictures (M 5 ! 0.14, SE 5 0.12) falling in between, quadratic Valence F(1,13) 5 2.75, p 5 .121, partial Z2 5 .174. For women, left superior auricular reflexes were greater during pleasant (M 5 0.21, SE 5 .10) and aversive pictures (M 5 0.10, SE 5 0.11) than during neutral pictures (M 5 ! 0.22, SE 5 0.09), quadratic Valence F(1,18) 5 8.44, p 5 .009, partial Z2 5 .319. However, there was no effect of Valence in the right ear, F(2.00,36.0) 5 1.63, p 5 .211, partial Z2 5 .083 (pleasant M 5 0.10, SE 5 0.09; neutral M 5 0.13, SE 5 0.08; aversive M 5 ! 0.08, SE 5 0.08). Thus, these effects
414
S. D. Benning
substantially qualified the significant Valence ! Ear effect, F(2.00,62.0) 5 3.91, p 5 .031, partial Z2 5 .106. No other effects were significant for superior auricular reflexes during pictures, Fso2.2, ps4.12. There were no significant effects for superior auricular reflexes during sounds, Fso2.5, ps4.13. Discussion Because superior auricular muscle reflexes were at best inconsistently modulated by emotional stimuli and because anterior auricular muscle reflexes seemed absent, it appears that it is the postauricular muscle, innervated by the postauricular branch of the facial nerve, whose reflex is responsible for the observed patterns of emotional modulation. Furthermore, the postauricular reflex appears to be a measure of appetitive processing during both pictures and sounds, indicating that this reflex may be used to study emotional processing across stimulus modalities. Additional attentional work should be performed to confirm that postauricular reflexes are smaller during sounds than during pictures, which may be an intriguing method of studying unimodal vs. cross-modal attention inhibition. It is possible that electrodes measuring the superior and anterior auricular muscles were not
optimally placed to record a signal from them. However, this seems unlikely as even suboptimally placed postauricular electrodes yield interpretable signals (O’Beirne & Patuzzi, 1999), Furthermore, the superior auricular muscle is relatively large compared to the postauricular muscle, so its reflexive activity should be comparatively easy to record. Nevertheless, it should be reiterated that the neural pathways underlying each of the auricular reflexes remain unclear. Thus, the interpretation of potentiation of the postauricular reflex remains ambiguous. Bickford et al. (1964) reported that ‘‘preliminary studies on unanesthetized monkeys, cats, rabbits, and dogs have failed to reveal a similar response to that recorded in the human’’ (p. 213), implying that finding animal models of this reflex may be difficult. There is also reason to doubt that the pinna reflex is an appropriate analog of the postauricular reflex, as pinna reflex magnitudes were greater in mice during presentation of a CS1 than at baseline in a single-cue conditioning experiment (Cassella & Davis, 1986). Dissection of human cadavers or anterograde tracing may be needed to confirm the neural circuitry of the postauricular reflex (as has been done in examinations of the anatomy of the postauricular muscle; Guerra, Metzinger, Metzinger, Xie, Xie, et al., 2004).
REFERENCES Benning, S. D., Patrick, C. J., & Lang, A. R. (2004). Emotional modulation of the post-auricular reflex. Psychophysiology, 41, 426–432. Berzin, F., & Fortinguerra, C. R. (1993). EMG study of the anterior, superior, and posterior auricular muscles in man. Anatomischer Anzeiger, 175, 195–197. Bickford, R. G., Jacobson, J. L., & Cody, T. R. (1964). Nature of average evoked potentials to sound and other stimuli in man. Annals of the New York Academy of Sciences, 112, 204–223. Bochenek, W., & Bochenek, Z. (1976). Postauricular (12 msec latency) responses to acoustic stimuli in patients with peripheral, facial nerve palsy. Acta Otolaryngologica, 81, 264–269. Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: The SelfAssessment Manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25, 49–59. Cassella, J. V., & Davis, M. (1986). Habituation, prepulse inhibition, fear conditioning, and drug modulation of the acoustically elicited pinna reflex in rats. Behavioral Neuroscience, 100, 39–44. Center for the Study of Emotion and Attention [CSEA-NIMH] (1999). The international affective picture system: Digitized photographs. Gainesville, FL: The Center for Research in Psychophysiology, University of Florida. Cody, D. T. R., Jacobson, J. L., Walker, J. C., & Bickford, R. G. (1964). Averaged evoked myogenic and cortical potentials to sound in man. Annals of Ootology, Rhinology, and Laryngology, 73, 763–777. Cuthbert, B. N., Bradley, M. M., & Lang, P. J. (1996). Probing picture perception: Activation and emotion. Psychophysiology, 33, 103–111. De Grandis, D., & Santoni, P. (1980). The post-auricular response: A single motor unit study. Electroencephalography and Clinical Neurophysiology, 50, 437–440. De Meirsman, J., Claes, G., & Geerdens, L. (1980). Normal latency value of the facial nerve with detection in the posterior auricular muscle and normal amplitude value of the evoked action potential. Electromyography and Clinical Neurophysiology, 20, 481–485. Gable, P. A., & Harmon-Jones, E. (2009). Postauricular reflex responses to pictures varying in valence and arousal. Psychophysiology, 46, 487– 490. Graham, F. K. (1979). Distinguishing among orienting, defense, and startle reflexes. In H. D. Kimmel, E. H. Van Olst, & J. F. Orlebeke (Eds.), The orienting reflex in humans (pp. 137–167). Hillsdale, NJ: Erlbaum. Gray, H. (1901/1995). Anatomy: Descriptive and surgical (15th ed). New York: Barnes and Noble Books, Inc.
Guerra, A. B., Metzinger, S. E., Metzinger, R. C., Xie, C., Xie, Y., Rigby, P. L., & Naugle, T. (2004). Variability of the postauricular muscle complex: Analysis of 40 hemicadaver dissections. Archives of Facial and Plastic Surgery, 6, 342–347. Hackley, S. A., Mun˜oz, M. A., Hebert, K., Valle-Incla´n, F., & Vila, J. (2009). Reciprocal modulation of eye-blink and pinna-flexion components of startle during reward anticipation. Psychophysiology, 46, 1154–1159. Hackley, S. A., Woldorff, M., & Hillyard, S. A. (1987). Combined use of microreflexes and event-related brain potentials as measures of auditory selective attention. Psychophysiology, 24, 632–647. Hess, U., Sabourin, G., & Kleck, R. E. (2007). Postauricular and eyeblink startle responses to facial expressions. Psychophysiology, 44, 431–435. Kiang, N. Y.-S., Crist, A. H., French, M. A., & Edwards, A. G. (1963). Postauricular electric response to acoustic stimuli in humans. Quarterly Progress Report, Research Laboratory of Electronics, 44, 218– 225. Cambridge, MA: Massachusetts Institute of Technology. O’Beirne, G. A., & Patuzzi, R. B. (1999). Basic properties of the soundevoked post-auricular muscle response (PAMR). Hearing Research, 138, 115–132. Quevedo, K. M., Benning, S. D., Gunnar, M. R., & Dahl, R. E. (2009). The onset of puberty: Effects on the psychophysiology of defensive and appetitive motivation. Development and Psychopathology, 21, 27–45. Sandt, A. R., Sloan, D. M., & Johnson, K. J. (2009). Measuring appetitive processing with the postauricular reflex. Psychophysiology, 46, 491–497. Sataloff, R. T., & Selber, J. C. (2003). Phylogeny and embryology of the facial nerve and related structures: Part IIFembryology. Ear, Nose, and Throat Journal, 82, 764–766, 769–772, 774, 779. Serra, G., Tugnoli, V., Cristofori, M. C., Eleopra, R., & De Grandis, D. (1986). The electromyographic examination of the posterior auricular muscle. Electromyography and Clinical Neurophysiology, 26, 661–665. Stekelenburg, J. J., & van Boxtel, A. (2001). Inhibition of pericranial muscle activity, respiration, and heart rate enhances auditory sensitivity. Psychophysiology, 38, 629–641. Yoshie, N., & Okudairi, T. (1969). Myogenic evoked potential responses to clicks in man. Acta Oto-Laryngologica Supplementum, 252, 89– 103. (Received December 7, 2009; Accepted April 22, 2010)
Psychophysiology, 48 (2011), 415–419. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01070.x
Pitch and loudness information encoded in auditory imagery as revealed by event-related potentials
JIANHUI WU,a ZULIN YU,b XIAOQIN MAI,c JINGHAN WEI,a and YUEJIA LUOd a
Institute of Psychology, Chinese Academy of Sciences, Beijing, China State Key Laboratory of Brain and Cognitive Science, Institute of Biophysics, Chinese Academy of Sciences, Beijing, China c Center for Human Growth and Development, University of Michigan, Ann Arbor, Michigan d State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China b
Abstract Two experiments using the ERP method and a task that involved comparing an imagined-S1 (the first stimulus) with a perceived-S2 (the second stimulus) were conducted to investigate whether imagined auditory representations encode pitch and loudness information. It was found that the amplitude of the imagery-related late positive complex (LPC) decreased with pitch but increased with loudness of the imagined sound, which was consistent with amplitude modulations of the auditory perception-related N1 component, thereby providing the first neural evidence that auditory imagery encodes perceptual attributes of auditory experiences. Descriptors: Auditory imagery, Pitch, Loudness, Event-related potentials, Late positive complex
sound at a lower loudness threshold when the pitch of the imagined sound is the same as the following target sound, suggesting auditory imagery also encodes pitch. These behavioral results may be confounded by the subject’s knowledge of the task; however, for example, subjects may use their prior knowledge to simulate what would happen in a real-world situation (Pylyshyn, 2002). Evidence from studies of brain activity, which cannot be simulated by a subject’s tacit knowledge, however, is still scarce. The aim of the present study was to provide neural evidence that auditory imagery encodes pitch and loudness information by using the event-related potential (ERP) method. Auditory stimuli typically elicit an N1 component in the ERP with a peak latency of about 100 ms and a scalp distribution over frontocentral areas. The amplitude of the N1 is related to perception and increases with sound intensity but decreases with sound frequency (for a review, see Na¨a¨ta¨nen & Picton, 1987). Previous ERP studies have shown that a late positive complex (LPC) is related to the generation of both visual and auditory imagery (Farah, Peronnet, Weisberg, & Monheit, 1989; Meyer, Elmer, Baumann, & Jancke, 2007; Wu, Mai, Chan, Zheng, & Luo, 2006). In the present study, we investigated whether the amplitude modulation of this imagery-related LPC will follow the same pattern as that of the auditory perception-related N1. If imagery is similar to perception, as the depictive theory suggests, the amplitude of the imagery-related LPC should also increase with loudness (the subjective dimension of intensity), but decrease with pitch (the subjective dimension of frequency) of the imagined sound. We developed a paradigm called ‘‘imagined-S1/perceived-S2 comparison’’ (Wu, Mai, Yu, Qin, & Luo, 2010). Before starting the EEG (electroencephalography) recordings, participants learned to associate each of three visual cues with one of three pure tones varying in pitch (Experiment 1) or loudness (Exper-
The most influential theory of mental imagery is Kosslyn’s depictive (picture-like) representation of imagery processing, emphasizing the close similarities between imagery and perception (Kosslyn, Thompson, & Ganis, 2006). Neuroimaging and neuropsychological data indicate that the modality-specific visual and auditory cortex is involved in visual and auditory imagery, respectively (Aleman, Formisano, Koppenhagen, Hagoort, de Haan, & Kahn, 2005; Kraemer, Macrae, Green, & Kelley, 2005; Slotnick, Thompson, & Kosslyn, 2005; for reviews, see Kosslyn, Ganis, & Thompson, 2001; and Kosslyn & Thompson, 2003). In the visual modality, neuroimaging data further indicate that, during visual imagery, spatial information is mapped in the primary visual cortex (Klein, Dubois, Mangin, Kherif, Flandin, et al., 2004; Kosslyn, Thompson, Kim, & Alpert, 1995), providing strong evidence for the depictive theory. In the auditory modality, behavioral studies also indicate that auditory imagery represents perceptual attributes of sound such as loudness and pitch (Farah & Smith, 1983; Intons-Peterson, 1980; for reviews, see Intons-Peterson, 1992; and Hubbard, 2010). In Intons-Peterson’s (1980) study, the time required to mentally adjust the loudness of one imagined sound to that of another imagined sound increased with the distance between the loudness of the imagined sounds, suggesting auditory imagery encodes loudness. Farah and Smith (1983) found that one could detect the target This work was supported by the NSFC (30930031, 30900442), and the Ministry of Education, China (PCSIRT, IRT0710), and the Project for Young Scientists Fund, IP, CAS (O9CX042004), and the GSCAS (2006). We thank Professor Sun Junxian for the use of the sound-level meter and Justin Ryder for his editorial assistance. Address correspondence to: Yue-jia Luo, State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, 19 Xin Jie Kou Wai Street, Beijing, 100875, China. E-mail: luoyj@bnu. edu.cn 415
416 iment 2). During EEG recording, participants were presented with one of these three visual cues and asked to imagine hearing the corresponding tone (imagined-S1). After a short delay, one of the three tones was actually presented (perceived-S2), and participants were required to make a same/different comparison between the real tone and the previously imagined one. The effective use of imagery was ensured by imagery training prior to EEG recordings and evaluated by means of imagery vividness and subvocalization questionnaires after the recording. The content of the questionnaires has been reported in our previously published paper that focused on both behavioral and ERP results associated with the perceived-S2 stimulus (Wu et al., 2010). In the present paper, we focus on the ERP elicited by the imagined-S1 stimulus in the same experiment in addition to interpreting the results of the perceived-S2 period from the perspective of mental imagery.
Methods Participants As reported previously (Wu et al., 2010), data were obtained from 22 right-handed participants (mean age 21.6 ! 0.9 years, 10 males) in Experiment 1 and 23 right-handed participants (mean age 21.3 ! 1.3 years, 11 males) in Experiment 2. Two participants were excluded in Experiment 1 and one participant in Experiment 2 due to excessive movement artifacts. All reported normal hearing and normal or corrected-to-normal vision. None of the participants reported a history of neurological or psychiatric disorders. Subjects were paid for their participation and gave their informed consent. Stimuli Three shapes (circle, square, and triangle) were chosen as visual cues to induce auditory imagery. Cues were presented on a computer screen placed 75 cm away from the participants’ eyes and subtended at an angle of approximately 21 both horizontally and vertically. In Experiment 1, three pure tone bursts of different pitch (low-pitch: 400 Hz; medium-pitch: 1000 Hz; and highpitch: 2500 Hz) at a constant loudness (75 dB SPL [sound pressure level]) were chosen as the imagined sounds. In Experiment 2, tones were at a constant pitch (400 Hz) while their loudness varied (soft: 50 dB SPL; medium: 75 dB SPL; and loud: 85 dB SPL). Tone bursts (250 ms duration, including the 25 ms rise/fall times) were broadcast from a loudspeaker (Fostex FE107E, Tokyo, Japan) placed beside the computer screen. SPLs were measured with a condenser microphone (Bru¨el & Kjaer 4135, Nærum, Denmark) and a sound-level meter (Bru¨el & Kjaer 2610), accurate to ! 1 dB over 0.1–10 kHz. The association between visual cues and tones was counterbalanced across participants. Procedure The procedure in both Experiment 1 and Experiment 2 was as follows. Participants were seated in a relaxed position on a comfortable chair in a dimly lit, sound-attenuated, and electrically isolated room. There were three training sessions before the EEG recordings. The first session was a familiarization session in which the visual cues and corresponding pure tones were presented simultaneously at least 50 times for each pair, until participants reported that they had learned these new associations.
J. Wu et al. The second session was an imagery training session in which only the visual cues were presented and the participants were encouraged to vividly imagine hearing the corresponding sounds; the real sound was then presented, and the participants were required to adjust their previously imagined sound to the real sound. The third session was an imagery-comparison training session in which one of the visual cues was presented and the participants were asked to imagine hearing the corresponding sound; after a short delay (600 ms) one of the three sounds was then presented and participants were asked to make a same/different comparison between the real sound and the previously imagined one by pressing a button as accurately and quickly as possible (see Figure 1). Following the button press, ‘‘Correct’’ or ‘‘Incorrect’’ feedback was presented to encourage both response accuracy and speed. After these three training sessions, the participants completed ten blocks of the EEG recording experiment with short breaks between blocks. Each block started with another short familiarization and imagery training session, and then the imagery-comparison task was performed (see Figure 1). Unlike the training session, however, no response feedback was provided during the experiment. Sequential effects of trial-to-trial transitions were counterbalanced within each block. The number of trials requiring ‘‘same’’ and ‘‘different’’ responses was equal. Each sound was both imagined and heard 240 times, leading to a total of 720 trials completed by each participant. After the ten experimental blocks, the participants completed a brief questionnaire by rating the vividness of their auditory imagery on a 7-point scale (1 5 no imagery at all, 7 5 very vivid imagery), and reporting whether they had experienced subvocalization (i.e., silent movements of the lips, tongue, or larynx) while imagining hearing sounds. EEG Recording and Analysis During each experimental block, EEG data were continuously recorded from 64 cap-mounted Ag/AgCl electrodes arranged according to the 10–20 international placement system (Compumedics Neuroscan, Charlotte, NC) with an on-line reference to the left mastoid and off-line algebraic re-reference to the average of the left and right mastoids. The EEG data were amplified with a bandpass filter of 0.05–100 Hz and digitized at 500 Hz. The vertical and horizontal electrooculogram (VEOG and HEOG) were recorded from two pairs of electrodes: one pair placed 1 cm above and below the left eye, and another pair placed 1 cm lateral from the outer canthi of both eyes. Interelectrode impedances were maintained below 5 kO. The EEG data were processed offline using the Neuroscan 4.3 software. Ocular artifacts were removed using a regression procedure implemented in the Neuroscan software (Semlitsch, Anderer, Schuster, & Presslich, 1986). Data were digitally filtered at 30 Hz lowpass and were epoched into periods of 1000 ms (including 200 ms of pre-stimulus time as a baseline) time-locked to the onset of the visual cue (S1-period) and the real sound during the familiarization session before each block of imagery-comparison task. Epochs containing artifacts exceeding ! 70 mV were rejected from the analysis. The data were then averaged for each condition. The present analysis focused on the LPC elicited by the imagined-S1. The LPC does not have a clear peak and was therefore measured as the mean amplitude in the time window of 330–500 ms at the following 18 sites: Fz, FCz, Cz, CPz, Pz, POz, F3, FC3, C3, CP3, P3, PO3, F4, FC4, C4, CP4, P4, and PO4.
Perceptual attributes of auditory imagery
417
Figure 1. Schematic description of the experimental paradigm. After the fixation, one of the visual cues was presented for 1500 ms and the subjects were required to imagine the corresponding sound; after a 600-ms delay, one of the real sounds was presented for 250 ms (or less if the RTwas less than 250 ms) and the subjects’ task was to compare the real sound with the preceding imagined sound by pressing a button. The screen then remained blank until a response. The next fixation appeared 800 ms after the response.
These amplitudes were subjected to repeated measures analysis of variance (ANOVA) with factors of pitch (imagining low and high-pitch sound) or loudness (imagining soft and loud sound) ! anterior-posterior scalp location (F, FC, C, CP, P, and PO) ! laterality (left, midline, and right). The data from mediumpitch/loudness sounds were not analyzed due to the interference of both high-pitch/loud and low-pitch/soft sounds on the imagery of medium sound. Behaviorally incorrect trials were also not analyzed. For the direct comparison between the imagery-related LPC and the auditory perception-related N1 within the same experiments, we also analyzed the effect of pitch and loudness of the perceived sound on the N1 peak amplitude at these same 18 sites when the sound was actually presented during the familiarization session before each block of imagery-comparison task.
The Greenhouse-Geisser correction was used to compensate for sphericity violations.
Results Experiment 1 For the amplitude of the LPC elicited by the imagined-S1, the main effect of pitch was not significant, but there was a significant main effect of anterior-posterior electrodes, F(5,105) 5 12.70, po.01, e 5 .27, and there was a marginally significant pitch ! anterior-posterior electrodes interaction, F(5,105) 5 3.85, po.1, e 5 .24. Further analysis revealed that the imagined low-pitch sound elicited a more positive LPC than did the imagined high-
Figure 2. Grand averaged ERPs illustrating pitch effect (Experiment 1, left panel) and loudness effect (Experiment 2, right panel) of imagined sound (S1) and real sound presented in the familiarization session before each block of the main imagery-comparison task. The gray areas highlighted the time windows of LPC (330–500 ms) used for statistical analysis.
418 pitch sound at fronto-central areas (see Figure 2), F: F(1,21) 5 3.22, po.1, FC: F(1,21) 5 6.80, po.05 and C: F(1,21) 5 7.52, po.05, but not at more posterior areas. Imagining both low-pitch and high-pitch sounds have maximum amplitudes over centro-parietal areas. For the amplitude of the N1 elicited by the perceived sound, the low-pitch sound elicited a significantly greater N1 than did the high-pitch sound (see Figure 2), F(1,21) 5 12.27, po.01, and there was a significant main effect of anterior-posterior electrodes, F(5,105) 5 69.09, po.001, e 5 .27. Further analysis revealed that hearing both low-pitch and high-pitch sounds have maximum amplitudes over fronto-central areas. Experiment 2 For the effect of loudness of the imagined sound on the amplitude of LPC, ANOVA revealed a significant main effect, F(1,22) 5 6.24, po.05. There was also a significant main effect of anterior-posterior electrodes, F(5,110) 5 13.01, po.001, e 5 .28, and a marginally significant loudness ! anterior-posterior electrodes interaction effect, F(5,110) 5 3.90, po.1, e 5 .30. Further analysis revealed that imagining the loud sound elicited a more positive LPC than did imagining the soft sound at centroparietal areas (see Figure 2), C: F(1,22) 5 4.96, po.05, CP: F(1,22) 5 9.98, po.01, P: F(1,22) 5 10.53, po.01, and PO: F(1,22) 5 11.21, po.01, but not at more anterior areas. Imagining both soft and loud sounds have maximum amplitudes over centro-parietal areas. For the amplitude of the N1 elicited by the perceived sound, the loud sound elicited a significantly greater N1 than did the soft sound (see Figure 2), F(1,22) 5 15.04, po.01, and there was a significant main effect of anterior-posterior electrodes, F(5,110) 5 35.72, po.001, e 5 .26. Further analysis revealed that hearing both soft and loud sounds have maximum amplitudes over fronto-central electrodes.
Discussion The present study used the ERP method to investigate whether perceptual attributes of auditory experiences are represented in auditory imagery. The main findings of the present study can be summarized as follows: imagining a low-pitch sound elicited a higher amplitude LPC than did imagining a high-pitch sound, and imagining a loud sound elicited a higher amplitude LPC than did imagining a soft sound. This response pattern parallels that of the effects of sound frequency and intensity on the auditory perception-related N1, which has been well documented in the literature (for a review, see Na¨a¨ta¨nen & Picton, 1987) and also replicated in the present study. The similar pattern of amplitude modulation between the imagery-related LPC and the auditory perception-related N1 suggests that auditory imagery may encode pitch and loudness information in a similar manner as auditory perception, thereby implying a functional similarity between auditory imagery and auditory perception. Behavioral studies also have suggested that auditory imagery reflects perceptual attributes of sound such as pitch and loudness (Farah & Smith, 1983; Intons-Peterson, 1980). The results of the present study provide to our knowledge the first neural evidence that auditory imagery may represent perceptual attributes of auditory experiences and provide evidence supporting the depictive theory of mental imagery in the auditory modality.
J. Wu et al. Previous studies provide insight into the possible cortical mechanisms underlying the amplitude modulation of the auditory perception-related N1 and imagery-related LPC. Studies using various neuroimaging techniques have shown both tonotopic and amplitopic organization of the human auditory cortex (e.g., Bilecen, Seifritz, Scheffler, Henning, & Schulte, 2002; Romani, Williamson, & Kaufman, 1982). Particularly worth mentioning among these is the magnetic equivalent of the electric N1, the N1m. The source of N1m, which is located in the auditory cortex (for a review, see Jacobson, 1994), varies with the frequency and intensity of the experienced sound: lower-frequency or higherintensity sounds activate more superficial cortical regions (Pantev, Bertrand, Eulitz, Verkindt, Hampson, et al., 1995; Pantev, Hoke, Lehnertz, & Lu¨tkenho¨ner, 1989; Pantev, Roberts, Elbert, Ross, & Wienbruch, 1996). Therefore, the larger amplitude of the electrical N1 elicited by lower-frequency or higher-intensity sounds might be attributed to the shorter distance between the brain source and the recording electrodes (Jacobson, Lombardi, Gibbens, Ahmad, & Newman, 1992; Wunderlich & Cone-Wesson, 2001). The same logic might apply to the imagery-related LPC in the present study: the amplitude modulation of the scalp recorded imagery-related LPC might be attributed to the tonotopicity and amplitopicity in the auditory cortex during auditory imagery. Previous brain imaging studies have revealed that the auditory cortex, especially the primary auditory cortex, is activated by auditory imagery (Aleman et al., 2005; Kraemer et al., 2005). To confirm this assumption of tonotopicity and amplitopicity of the auditory cortex when imagining auditory stimuli, further studies with high spatial resolution neuroimaging will be needed. The ERPs elicited by the perceived-S2, in particular the effects of the discrepancy between the imagined and perceived sounds on the N2 component and the behavioral performance of the same/ different comparison, have been reported in our previous paper (Wu et al., 2010). The different comparison pairings between presented sounds (S2) and imagined sounds (S1) led to three levels of discrepancy (no, small, and large discrepancy) defined by the degree of physical separation. The results can be summarized as follows: an N2 component with latency of approximately 220 ms was reliably elicited when the heard S2 was different from the imagined S1, suggesting that the N2 ERP, typically elicited in an S1–S2 matching paradigm and considered to reflect mismatch processing, can still be elicited when the S1 was imagined instead of perceived. Furthermore, the amplitude of the N2 increased with the degree of discrepancy, and accuracies were lower and reaction times were longer for the small discrepancy than large discrepancy conditions. These behavioral results replicate the fourth task by Intons-Peterson (1980) in which the discrimination times increased with decreasing distances between loudness of two imagined sounds. The effect of the degree of discrepancy on both the N2 amplitude and the behavioral performance provide further evidence that auditory imagery includes perceptual information of auditory experiences (i.e., pitch in Experiment 1 and loudness in Experiment 2), supporting the assumption of functional similarity between auditory imagery and perception. In summary, auditory imagery encodes pitch and loudness information as evidenced by the similar pattern of amplitude modulation between the imagery-related LPC and the auditory perception-related N1. The previously reported effects of discrepancy between the real sound and the previously imagined sound on both behavioral performance and N2 component amplitude (Wu et al., 2010) provide further evidence supporting this assumption.
Perceptual attributes of auditory imagery
419 REFERENCES
Aleman, A., Formisano, E., Koppenhagen, H., Hagoort, P., de Haan, E. H. F., & Kahn, R. S. (2005). The functional neuroanatomy of metrical stress evaluation of perceived and imagined spoken words. Cerebral Cortex, 15, 221–228. Bilecen, D., Seifritz, E., Scheffler, K., Henning, J., & Schulte, A. C. (2002). Amplitopicity of the human auditory cortex: An fMRI study. NeuroImage, 17, 710–718. Farah, M. J., Peronnet, F., Weisberg, L. L., & Monheit, M. (1989). Brain activity underlying mental imagery: Event-related potentials during mental image generation. Journal of Cognitive Neuroscience, 1, 302–316. Farah, M. J., & Smith, A. F. (1983). Perceptual interference and facilitation with auditory imagery. Perception and Psychophysics, 33, 475–478. Hubbard, T. L. (2010). Auditory imagery: Empirical findings. Psychological Bulletin, 136, 302–329. Intons-Peterson, M. J. (1980). The role of loudness in auditory imagery. Memory and Cognition, 8, 385–393. Intons-Peterson, M. J. (1992). Components of auditory imagery. In D. Reisberg (Ed.), Auditory imagery (pp. 45–71). Hillsdale, NJ: Erlbaum. Jacobson, G. P. (1994). Magnetoencephalographic studies of auditory system function. Journal of Clinical Neurophysiology, 11, 343–364. Jacobson, G. P., Lombardi, D. M., Gibbens, N. D., Ahmad, B. K., & Newman, C. W. (1992). The effects of stimulus frequency and recording site on the amplitude and latency of multichannel cortical auditory evoked potential (CAEP) component N1. Ear and Hearing, 13, 300–306. Klein, I., Dubois, J., Mangin, J. F., Kherif, F., Flandin, G., Poline, J. B., et al. (2004). Retinotopic organization of visual mental images as revealed by functional magnetic resonance imaging. Cognitive Brain Research, 22, 26–31. Kosslyn, S. M., Ganis, G., & Thompson, W. L. (2001). Neural foundations of imagery. Nature Reviews Neuroscience, 2, 635–642. Kosslyn, S. M., & Thompson, W. L. (2003). When is early visual cortex activated during visual mental imagery? Psychological Bulletin, 129, 723–746. Kosslyn, S. M., Thompson, W. L., & Ganis, G. (2006). The case for mental imagery. New York: Oxford University Press. Kosslyn, S. M., Thompson, W. L., Kim, I. J., & Alpert, N. M. (1995). Topographical representations of mental images in primary visual cortex. Nature, 378, 496–498. Kraemer, D. J., Macrae, C. N., Green, A. E., & Kelley, W. M. (2005). Musical imagery: Sound of silence activates auditory cortex. Nature, 434, 158.
Meyer, M., Elmer, S., Baumann, S., & Jancke, L. (2007). Short-term plasticity in the auditory system: Differential neural responses to perception and imagery of speech and music. Restorative Neurology and Neuroscience, 25, 411–431. Na¨a¨ta¨nen, R., & Picton, T. (1987). The N1 wave of the human electric and magnetic response to sound: A review and an analysis of the component structure. Psychophysiology, 24, 375–425. Pantev, C., Bertrand, O., Eulitz, C., Verkindt, C., Hampson, S., Schuierer, G., & Elbert, T. (1995). Specific tonotopic organizations of different areas of the human auditory cortex revealed by simultaneous magnetic and electric recordings. Electroencephalography and Clinical Neurophysiology, 94, 26–40. Pantev, C., Hoke, M., Lehnertz, K., & Lu¨tkenho¨ner, B. (1989). Neuromagnetic evidence of an amplitopic organization of the human auditory cortex. Electroencephalography and Clinical Neurophysiology, 72, 225–231. Pantev, C., Roberts, L. E., Elbert, T., Ross, B., & Wienbruch, C. (1996). Tonotopic organization of the sources of human auditory steady-state responses. Hearing Research, 101, 62–74. Pylyshyn, Z. W. (2002). Mental imagery: In search of a theory. Behavioral and Brain Sciences, 25, 157–182. Romani, G. L., Williamson, S. J., & Kaufman, L. (1982). Tonotopic organization of the human auditory cortex. Science, 216, 1339–1340. Semlitsch, H. V., Anderer, P., Schuster, P., & Presslich, O. (1986). A solution for reliable and valid reduction of ocular artifacts, applied to the P300 ERP. Psychophysiology, 23, 695–703. Slotnick, S. D., Thompson, W. L., & Kosslyn, S. M. (2005). Visual mental imagery induces retinotopically organized activation of early visual areas. Cerebral Cortex, 15, 1570–1583. Wu, J., Mai, X., Chan, C. C., Zheng, Y., & Luo, Y. (2006). Event-related potentials during mental imagery of animal sounds. Psychophysiology, 43, 592–597. Wu, J., Mai, X., Yu, Z., Qin, S., & Luo, Y. (2010). Effects of discrepancy between imagined and perceived sounds on the N2 component of the event-related potential. Psychophysiology, 47, 289–298. Wunderlich, J. L., & Cone-Wesson, B. K. (2001). Effects of stimulus frequency and complexity on the mismatch negativity and other components of the cortical auditory-evoked potential. Journal of the Acoustical Society of America, 109, 1526–1537.
(Received November 6, 2009; Accepted April 22, 2010)
Psychophysiology, 48 (2011), 420–429. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01078.x
Association between Cold Face Test-induced vagal inhibition and cortisol response to acute stress
ROBERTO LA MARCA,a PATRICIA WALDVOGEL,a HANNA THO¨RN,a ME´LANIE TRIPOD,a PETRA H. WIRTZ,a JENS C. PRUESSNER,b and ULRIKE EHLERTa a
Department of Clinical Psychology and Psychotherapy, University of Zurich, Zurich, Switzerland Departments of Psychiatry, Neurology, and Neurosurgery, McGill University & Douglas Hospital Research Centre, Montre´al, Que´bec, Canada
b
Abstract Low vagal function is related to several disorders. One possible underlying mechanism linking the vagus nerve and disorders is the HPA axis. Thirty-three healthy male subjects participated in a stress task, while heart rate (HR), respiratory sinus arrhythmia (RSA), salivary cortisol, and mood were assessed. Vagal function was determined using baseline, stress-induced inhibition, and Cold Face Test (CFT)-induced stimulation. The stress task induced a significant increase in cortisol and HR, a decrease in RSA, and a worsening of mood. A linear regression model with the time from CFT onset until maximum bradycardia as the independent variable explained 17.9% of the total variance in cortisol in response to the stressor (mood: 36.5%). The results indicate that a faster CFT response is associated with reduced cortisol increase and enhanced mood after acute stress. Our data support an inverse relationship between vagal function and the HPA axis. Descriptors: Vagal function, Acute psychosocial stress, Cortisol reactivity
2006). From a structural point of view, the central autonomic network (CAN; Benarroch, 1997) provides a framework for an association between the two systems. The CAN contains complex direct and indirect interconnections between several structures, such as the medullary vagal nuclei and the paraventricular nucleus of the hypothalamus, but also the prefrontal cortex (PFC) and the amygdala (Benarroch, 1997). The CAN modulates psychophysiological resources in emotion (Thayer & Friedman, 2002), with the PFC and the VN providing negative feedback on sympathoexcitatory and HPA axis responses (Benarroch, 1997; Palkovits, 1999; Porges, 2001; Thayer & Friedman, 2002; Thayer & Lane, 2000; Thayer & Sternberg, 2006). Since HRV indexes the inhibitory control of the PFC on the amygdala (Thayer & Sternberg, 2006), and inhibition plays an important role for the ability to rapidly adapt to demands (Thayer & Friedman, 2002), HRV has been proposed as an index for the ability to adapt to changing demands from the environment (Thayer, 2007). While a structural association between the VN and the HPA axis, with an inverse relation between the function (activity and reactivity) of the two systems, can be assumed, non-invasive psychophysiological studies examining HRV and cortisol during different conditions show inconsistent results. Baseline levels of HRV and cortisol were shown to be unrelated to each other (Gunnar, Porter, Wolf, Rigatsu, & Larson, 1995; Johnsen, Hansen, Sollers, Murison, & Thayer, 2002). In contrast, baseline levels of HRV were reported to be related to cortisol stress responses, but results are contradictory, with evidence for both negative (Johnsen et al., 2002) and positive (Gunnar et al., 1995) associations. Similarly, studies investigating stress-induced vagal
The vagus nerve (VN) is the major constituent of the parasympathetic nervous system (PSNS), and innervates several vital organs. Functionally, the VN promotes rest and digest, while the sympathetic nervous system (SNS) promotes fight or flight. Vagal activity can be indexed by variation in inter-beat intervals (heart rate variability, HRV; Task Force, 1996). Reduced activity and reactivity levels of HRV have frequently been reported to be associated with somatic and mental disorders and mortality (Thayer & Brosschot, 2005; Thayer & Lane, 2007), while invasive electrical stimulation of the VN is used for the treatment of certain disorders (Bernstein, Barkan, & Hess, 2006; Hord, Evans, Mueed, Adamolekun, & Naritoku, 2003; Merrill, Jonsson, Minthon, Ejnell, C-son Silander, et al., 2006; Rush, George, Sackeim, Marangell, Husain, et al., 2000). However, the underlying mechanisms linking lower activity of the VN and somatic and mental disorders are not fully understood. The hypothalamic-pituitary-adrenal (HPA) axis, with dysregulations of its own also being related to several somatic and mental disorders (e.g., Christensen & Kessing, 2001; Tsigos & Chrousos, 1994), may play a mediating role. In fact, the VN is assumed to possess a regulating effect on the HPA axis (Thayer & Sternberg, We gratefully acknowledge the help of Susanne Fischer in conducting the experiments. This study was supported by a grant to the first author from the Forschungs- und Nachwuchsfo¨rderungskommission der Universita¨t Zu¨rich. Address correspondence to: Ulrike Ehlert, Department of Clinical Psychology and Psychotherapy, University of Zurich, Binzmuehlestrasse 14/Box 26, CH-8050 Zurich, Switzerland. E-mail: u.ehlert@ psychologie.uzh.ch 420
Vagal function and cortisol stress response inhibition report inconsistent findings: HRV and cortisol in response to stress (social, psychological, physical, workday challenges) show negative (Doussard-Roosevelt, Montgomery, & Proges, 2003; Thayer, Hall, Sollers, & Fischer, 2006) as well as no significant associations (Altemus, Redwine, Leong, Frye, Porges, & Carter, 2001; Cacioppo, Malarkey, Kiegolt-Glaser, Uchino, Sgoutas-Emch, et al., 1995; Gunnar et al., 1995; Heilman, Bal, Bazhenova, Sorokin, Perlman, Hanley, & Porges, 2008). Notably, hitherto all studies investigating the VN and the HPA stress reactivity examine vagal function comprising baseline activity and/or stress-induced vagal inhibition. To the best of our knowledge, associations between acute vagal stimulation and HPA stress reactivity have not yet been investigated. The main purpose of the present study was to examine the relation between vagal function (baseline, inhibition, stimulation) and cortisol measures (baseline, stimulation). To stimulate the VN, the cold face test (CFT; Khurana & Wu, 2006) was conducted (i.e., CFT-induced vagal stimulation). To induce a stress response of the VN (i.e., stress-induced vagal inhibition) and the HPA axis, the Montreal Imaging Stress Task (MIST, Dedovic, Renwick, Khalili-Mahani, Engert, Lupien, & Pruessner, 2005) was applied. To our knowledge, this is the first study to examine the effects of the MIST on the PSNS, representing the second purpose of the present study.
Methods and Materials Study Participants Subjects were recruited by advertisement at the universities of Zurich. Inclusion criteria included male sex and an age range of 18 to 40 years and right-handedness. Exclusion criteria included depression, self-reported acute and chronic somatic or psychiatric disorders, medication in the last 2 months, the consumption of psychoactive substances, and excessive consumption of alcohol (42 alcohol beverages/day) or tobacco (45 cigarettes/day). Inclusion was restricted to male subjects, primarily due to gender differences in possible pain sensation during the CFT (Girdler, Maixner, Naftel, Stewart, Moretz, & Light, 2005), but also due to the effect of oral contraceptives and menstrual cycle phase on cortisol and HRV (Kirschbaum, Kudielka, Gaab, Schommer, & Hellhammer, 1999; Sato, Myake, Akatsu, & Kumashiro, 1995). Participants received monetary compensation for their participation. The study was conducted in accordance with the Declaration of Helsinki and was approved by the cantonal ethics committee. Subjects provided written informed consent prior to participation. Procedure After arriving at the laboratory, subjects were fitted with the cardiorespiratory ambulatory device, and were seated in a comfortable chair. All tests were conducted in the same room at a constant temperature (211C), while subjects were sitting in front of a table with a computer and several available magazines. After a rest period of 30 min, subjects were asked to fill out state mood questionnaires. This was followed by the MIST. Subjects were exposed in random order to the stress (MIST-S) and control condition (MIST-C), on two separate occasions 2 weeks apart. Mood questionnaires were again handed out after the termination of the task. To include cortisol stress recovery, the examination lasted for an additional 60 min. At the end of the control examination, the CFT was conducted, while at the end of the stress examination, subjects were debriefed and informed about
421 the cover story (examination of interaction between cognitive skills and physiological markers). Subjects who first participated in the stress condition were reassured that, on the next occasion, a cognitive but not a stress task would take place. This procedure was chosen to ensure that, at the beginning of each examination, all subjects, independent of the condition order, would expect a cognitive task to be conducted. With this procedure, we intended to avoid an anticipatory stress response in subjects who participated in the control condition in the second examination. At the end of the second examination, participants gave a second written informed consent allowing the further use of their data. Interventions Stress Task To induce a multidimensional stress response, a slightly modified version of the MIST (Dedovic et al., 2005) was used, since the task was originally developed for fMRI environments. The MIST is a standardized computerized stress task combining challenging arithmetic problems with social-evaluative threat and can be carried out with (stress condition; MIST-S) or without time pressure and social evaluation (control condition; MIST-C). During the MIST-S, the program adapts the difficulty and time provided to solve the problems impeding a good performance (45% to 50% of correct answers). During the MIST-C, the difficulty of the arithmetic problems is randomly chosen, and neither time pressure nor social evaluation is applied. In both conditions, three blocks of 4 min each were run, with feedback of 2–3 min provided by the examiner in between blocks. The first feedback of the MIST-S consisted of informing the participants that their performance was poor, with the examiner inquiring whether the subjects were experiencing any application problems (e.g., problems with the keyboard). A fictitious study leader then informed the examiner by telephone to repeat the test. In the second feedback of the MIST-S, the study leader entered the examination room and interrogated the participants about individual problems (e.g., about school performance). Participants were then informed about the high costs due to a possible exclusion if they did not achieve a better performance. The last block was then started, while the study leader remained in the examination room during the first 3.5 min. Before leaving the room, he instructed the examiner to continue with the normal procedure. At the very end of the stress examination, participants were debriefed. During the first and second feedback of the MIST-C, participants were provided with a neutral feedback. They were asked to perform a second and third block, respectively, in order to evaluate the time course of interactions between cognitive and physiological characteristics. Cold Face Test To extend the examination of vagal function, in addition to vagal baseline and stress-induced inhibitory measurement, the CFT was conducted in order to provoke vagal stimulation, since it mimics the diving reflex by inducing a trigeminal-vagal-mediated bradycardia (Arnold, 1999; Khurana, 2007; Khurana, Watabiki, Hebel, Toro, & Nelson, 1980; Khurana & Wu, 2006). Bradycardia was induced by using a full-face mask (Dr. Winkler GmbH, Ainring-Mitterfelden, Germany) covering wide parts of the face, with openings for the eyes avoiding an occulocardiac reflex, and for the nose and mouth allowing normal breathing. An additional cold pack (Nexcare, 3M Health Care, St. Paul, MN) was affixed to the full-face mask to augment the cold mass. While the room temperature was kept constant (211C), the temperature of
422
R. La Marca et al.
the cold stimulus was 11C. The sitting subjects were instructed in advance not to move or talk and to continue breathing normally during the CFT, which lasted for 2 min. Measures Cortisol Examinations started in the afternoon between 1:30 and 4:15 p.m. to control for circadian fluctuations (Smyth, Ockenfels, Gorin, Catley, Porter, et al., 1997). To further control for circaseptan alterations (Bodis, Boncz, & Kriszbacher, in press; Lee, Lee, Lee, Corne´lissen Otsuka, & Halberg, 2003; Maschke, Harder, Corne´lissen, Hecht, Otsuka, & Halberg, 2003), both examinations were conducted with 2 weeks in between. Salivary samples were repeatedly collected with salivettes (Sarstedt, Sevelen, Switzerland) before, during, and after the interventions by collecting unstimulated whole saliva: immediately before the MIST-S and MIST-C instruction, during the third block, and up to 60 min after the completion of the MIST (Figure 1A). Subjects placed a salivette under the tongue and kept the head slightly inclined for 2 min. Saliva samples were stored at ! 201C before the biochemical analysis took place. After thawing, saliva was centrifuged at 3000 rpm for 5 min before free cortisol was analyzed using an immunoassay with time-resolved fluorescence detection (Dressendorfer, Kirschbaum, Rohde, Stahl, & Strasburger, 1992). Electrophysiological Measures The LifeShirt system 200 (Vivometrics, Ventura, CA) was used to measure cardiac and pulmonary activity. This ambulatory cardiopulmonary measurement device consists of a garment with two integrated inductive plethysmography (IP) bands surrounding the midthorax and midabdomen and a connected palm, which additionally saves electrocardiographic data measured by three electrodes. The device was recently evaluated and shown to possess a good accuracy of detection and timing of beat-to-beat values (Heilman & Porges, 2007). After volume calibration of the IP bands by the subjects breathing repeatedly into a fixed volume bag (800 cc), data recording started. The record was then examined for artifacts and edited manually to correct for ectopic beats and arrhythmias by using linear interpolation. The corrected inter-beat-interval (IBI) allowed the calculation of HR. The association of IBI in relation to respiration allowed the measurement of respiratory sinus arrhythmia (RSA), indexing cardiovagal activity. HR and RSA were determined for 1-min intervals and averaged for 5-min intervals (e.g., for time " condition interaction effects). HR and RSA were determined using the VivoLogic 3.1.2 software package (Vivometrics). RSA was determined using the time-domain peak-valley method (Grossman, von Beek, & Wientjes, 1990). The effect of the CFT was measured as alteration in HR and RSA over 4 min, from 1 min before onset until 1 min after completion of the CFT. The HR over the 1-min period preceding the CFT was set as baseline for two further measures with regard to the CFT. Maximum response of HR was determined as the relative difference between the baseline and peak response due to the CFT (CFTmax; [% change with respect to baseline HR]), and the latency of response was defined as the time interval from the first instance of three successively slowing beats below baseline until peak bradycardia (CFTlatency; [sec]) (Khurana & Wu, 2006). Additionally, since respiration alterations might affect RSA independent of cardiovagal involvement, tidal volume (Vt) and respiration rate (fb) were extracted to be considered for statistical analysis.
Figure 1. (A) Salivary cortisol concentration (nmol/l), (B) HR (beats per minute), and (C) RSATR (ms/ml) during the stress (solid line) and control condition (dashed line). Values represent mean # standard error of the mean values.
Psychological Measures Depressive mood was measured with a German version (Allgemeine Depressionsskala-Langform, ADS-L) (Hautzinger & Bailer, 1992) of the Center for Epidemiologic Studies Depression Scale (CES-D) (Radloff, 1977) in order to exclude subjects with a possible depressive disorder (cut-off score 423). State mood was determined repeatedly, immediately before and after the MIST-S
Vagal function and cortisol stress response
423
and MIST-C. To assess different aspects of mood, subjects filled out the Multidimensional Mood State Questionnaire (German original version: Multidimensionaler Befindlichkeitsfragebogen, MDBF) (Steyer, Schwenkmezger, Notz, & Eid, 1997) consisting of three dimensions: ‘good-bad mood,’ ‘calmness-nervousness,’ and ‘wakefulness-tiredness,’ with higher values indicating good mood, calmness, and wakefulness. Furthermore, a visual analogue scale (VAS) was handed out before and after the stress and control condition asking subjects to rate how ‘stressed’ they felt at that moment. Further VAS asking for mood ratings (feeling stressed, exhausted, queasy, relaxed, good humored) and assessing pain were distributed 2 min before onset and after completion of the CFT to assess any changes in subjective sensation. The distributed questionnaires have been broadly used and have shown satisfactory internal consistency and validity.
data points for cortisol (AUCI(MIST-S): N 5 28). The mean age of participants was M 5 24.06 (SD 5 4.56, range 19–34) and the mean BMI was M 5 23.02 (SD 5 2.93, range 18.83–29.05). Depression scores were in a normal range of values (M 5 7.84, SD 5 4.14, range 2–21). Randomization resulted in two groups, with 16 subjects undergoing the control condition first and 17 subjects undergoing the stress condition first. The two groups did not differ with respect to demographic (BMI, age), physiological, and mood changes during the stress condition (all p 5 n.s.). Referring to the control condition, the corresponding variables did not differ, with the exception of the wakefulness-tiredness dimension of the MDBF, whereas subjects participating in the control condition in the second session became less tired compared to subjects who started with the control condition (t31 5 " 2.93, p 5 .006). The randomization was therefore successful.
Data Analyses Analyses were performed using SPSS (17.0) software packages (SPSS, Chicago, IL). Homogeneity of variance was assessed using the Levene test. In addition to raw data, the trapezoid formula for total response (area under the curve with respect to the ground, AUCG) and total change of response in consideration of individual baseline (area under the curve with respect to increase, AUCI) were computed (La Marca, Nedeljkovic, Yuan, Maercker, & Ehlert, 2010; Pruessner, Kirschbaum, Meinlschmid, & Hellhammer, 2003; Scholz, La Marca, Nater, Aberle, Ehlert, et al., 2009). When referring to treatment order or condition (with regard to AUCG and AUCI), paired Student’s t-tests were used for comparisons between groups, while repeated-measures analyses of variance (ANOVA) were computed after Greenhouse–Geisser corrections to reveal possible interaction effects. As post hoc tests, we repeated the previously described calculations separately in each condition group. For assessment of associations between vagal function and biopsychological stress responses, we used a two-step procedure. We first calculated correlations to test for significant associations between vagal function (baseline, stress-induced inhibition, CFT-induced stimulation) and cortisol measures (baseline, stress-induced stimulation). Second, to avoid multiple testing, we only calculated linear regressions with significant vagal markers from step 1 as independent variables. Cortisol, cardiopulmonary, and psychological stress responses were used as dependent variables in separate regression models. Explained variance in regression models is reflected by R2. In terms of RSA, Vt and fb were controlled for in all statistical analyses due to significant alterations of Vt and fb during the MIST-S and the CFT. Therefore, the transfer function of RSA (RSATR; Grossman, Wilhelm, & Spoerle, 2004; Grossman, Spoerle, & Wilhelm, 2006; La Marca et al., 2010) was applied, while further controlling for the AUCI of fb during the MIST-S and/or CFT. In condition ! time interactions, the effect size was determined by partial eta-square (partial eta2) reflecting small (.01), medium (.06), or large (.14) effect sizes (Green, Salkind, & Akey, 2000). All analyses were two-tailed, with the level of significance set at po.05.
Responses to the Stress Task All 33 subjects completed all of the three blocks of the MIST-S and MIST-C.
Results Sample Characteristics Thirty-four healthy subjects agreed to participate in the study. One subject was excluded during the examination period because of acute illness on the day of the stress examination. Low amounts of saliva in some samples further reduced the available number of
Cortisol Response Cortisol levels significantly increased during the stress condition as compared to the control condition (interaction condition by time: F(2.27/56.69) 5 20.94; po.001; partial eta2 5 .46; Figure 1A). Differences in AUC values further supported this finding (main effect of condition: Cortisol-AUCG: t25 5 " 4.98, po.001; Cortisol-AUCI: t25 5 " 5.58, po.001). Post hoc testing separately in each condition revealed a significant increase over time during the MIST-S (main effect of time: F(2.02/54.51) 5 16.59; po.001), and a significant decrease during the MIST-C, which is typical for the circadian course of cortisol (main effect of time: F(1.83/45.65) 5 24.55; po.001). At the peak response of cortisol 20 min after task completion (Figure 1A; min. 40), the MIST-S compared to the MIST-C resulted in a 2.64 times higher value. Electrophysiological Responses HR significantly increased during the stress condition as compared to the control condition (interaction condition by time: F(2.13/68.00) 5 39.34, po.001, partial eta2 5 .55; Figure 1B). Furthermore, this finding was supported by differences in AUC values (main effect of condition: HR-AUCG: t32 5 " 3.53, p 5 .001; HRAUCI: t32 5 " 5.33, po.001). Notably, significant alterations were found in both conditions (main effect of time: MIST-S: F(2.10/ 67.06) 5 42.10, po.001; MIST-C: F(3.15/100.69) 5 4.46, p 5 .005). In contrast to HR, RSATR levels significantly decreased during the MIST-S as compared to the MIST-C (interaction of condition by time: F(3.14/97.21) 5 10.41, po.001; partial eta2 5 .25; Figure 1C). AUC measures, however, did not differ between conditions (main effect of condition: RSATR-AUCG: F(1.00/31.00) 5 1.53, p 5 .23; RSATR-AUCI: F(1.00/31.00) 5 2.08, p 5 .16). A significant decrease of RSATR was found during the stress condition (main effect of time: F(3.15/97.62) 5 10.71, po.001), while no significant alteration was found during the control condition (main effect of time: F(3.23/100.17) 5 1.64, p 5 .18). Mood Responses ANOVAs revealed a significant difference in terms of mood (MDBF) between MIST-S and MIST-C (interaction condition by time: F(1.00/32.00) 5 25.51, po.001, partial eta2 5 .44; Table 1), with a significant worsening in mood during the stress (main effect of time: F(1.00/32.00) 5 27.80, po.001) but not the control condition (main effect of time: F(1.00/32.00) 5 .52, p 5 .48), as revealed by post hoc testing.
424
R. La Marca et al.
Table 1. Paired Student’s t-test Between MIST-S and MIST-C Induced Changes in the Three Dimensions ‘Mood,’ ‘Calmness,’ and ‘Wakefulness’ of the MDBF, and the Visual Analog Scale ‘Stressed’
Table 2. Correlations of Measures of Vagal and Cortisol (Re-)Activity During Different Conditions, Controlling for fb Alterations, Where Appropriate Cortisol measures
Mood Calmness Wakefulness Stressed
DiffMIST-C (SD)
DiffMIST-S (SD)
.21 (1.69) .42 (2.17) .84 (3.32) ! .59 (1.89)
! 3.88nnn (4.22) ! 4.70nnn (3.35) ! .72n (3.19) 2.93nnn (3.15)
Note: Values are means " standard deviations. DiffMIST-S 5 values immediately after MIST-S minus baseline values; DiffMIST-C 5 values immediately after MIST-C minus baseline values. n po.05; nnnpo.001.
Similarly, ANOVAs revealed a significant difference in calmness (MDBF) between the two conditions (interaction condition by time: F(1.00/32.00) 5 52.30, po.001, partial eta2 5 .62), due to a significant decrease in calmness during the stress (main effect of time: F(1.00/32.00) 5 64.90, po.001) but not the control condition (main effect of time: F(1.00/32.00) 5 1.27, p 5 .27). With regard to wakefulness (MDBF), the two conditions differed significantly (interaction condition by time: F(1/31) 5 4.41, p 5 .044, partial eta2 5 .12), although post hoc testing was unable to reveal either the MIST-S (main effect of time: F(1.00/32.00) 5 1.71, p 5 .20) or the MIST-C (main effect of time: F(1.00/31.00) 5 2.06, p 5 .16) to alter rated wakefulness. Finally, the item feeling stressed (VAS) revealed a significant interaction effect (interaction condition by time: F(1/32) 5 25.97, po.001, partial eta2 5 .45), with a significant increase during the MIST-S (main effect of time: F(1/32) 5 28.53, po.001), but not during the MIST-C (main effect of time: F(1/32) 5 3.25, p 5 .08).
Vagal function Baseline Inhibition Stimulation
Vagal parameters
Baseline
AUCI(MIST-S)
RSATR: baseline HR: baseline RSATR: AUCI(MIST-S) RSATR: AUCI(CFT) CFTmax CFTlatency
.164 .094 ! .050 ! .171 .090 ! .024
.017 .034 ! .217 .015 ! .032 .424n
Note: Values are correlational coefficients r. n po.05.
or AUCI of cortisol during the MIST-S (Table 2). Similarly, bivariate correlations with CFTmax showed no relation to cortisol. In contrast, bivariate correlations with CFTlatency revealed a significant positive association with AUCI(MIST-S) of cortisol, but not with baseline. Subjects with longer latency until peak bradycardia showed a higher stress response of cortisol (Figure 2A). Based on the analysis of correlation, we conducted a linear regression analysis
Responses to the Cold Face Test The CFT induced a significant decrease in HR (main effect of time: F(2.17/69.48) 5 39.94, po.001, partial eta2 5 .56) and, accordingly, a significant increase in RSATR (main effect of time: F(2.40/ 74.53) 5 10.73, po.001, partial eta2 5 .26), indicating a successful cardiovagal stimulation. The bradycardia during the CFT peaked after M(CFTlatency) 5 29.85 sec (SD 5 19.95 sec) and resulted in a maximum decrease of M(CFTmax) 5 24.80% (SD 5 7.64%). The cardiopulmonary changes during the CFT could not be attributed to subjective mood or pain changes, which did not alter significantly (main effect of time: VASstressed: t32 5 .48, p 5 .63; VASexhausted: t32 5 .91, p 5 .37; VASqueasy: t32 5 1.47, p 5 .15; VASrelaxed: t32 5 .73, p 5 .47; VASgood humoured: t32 5 ! .05, p 5 .96; VASpain: t32 5 ! .09, p 5 .93). Associations Between Vagal and HPA Function Vagal baseline. Bivariate correlations with baseline of HR and RSATR revealed no significant relation to cortisol measured under baseline and stress conditions (Table 2). Stress-induced vagal inhibition. Similar to vagal baseline, partial correlations controlling for fb changes during the MIST-S revealed no significant association between AUCI(MIST-S) of RSATR and baseline or AUCI(MIST-S) of cortisol (Table 2). CFT-induced vagal stimulation. Partial correlations controlling for the AUCI of fb during the CFT showed no significant association between the AUCI of RSATR during the CFTand the baseline
Figure 2. Stress reaction of (A) salivary cortisol and (B) mood in subjects with short (white circles) and long (black circles) latencies until peak bradycardia during the CFT (CFTlatency) determined after median split. Values represent mean " standard error of the mean values.
Vagal function and cortisol stress response
425
Table 3. Hierarchical Regression Analyses for Associations Between CFTlatency and Stress Responses Variables entered
Standardized coefficient b
AUCI(MIST-S) of cortisol CFTlatency AUCI(MIST-S) of HR CFTlatency AUCI(MIST-S) of RSATR AUCI(MIST-S) of fb CFTlatency DiffMIST-S in mood CFTlatency DiffMIST-S in calmness CFTlatency DiffMIST-S in wakefulness CFTlatency DiffMIST-S in feeling stressed CFTlatency
R2 change
t
p
.42
2.38
.025
.18
.13
.75
.46
.02
! .54 ! .01
! 3.49 ! .08
.002 .94
.29 .00
! .60
! 4.22
o.001
.36
! .20
! 1.11
.28
.04
! .20
! 1.12
.27
.04
.12
.65
.52
.01
Note: We controlled for fb alterations where appropriate. DiffMIST-S 5 values immediately after MIST-S minus baseline values; AUCI(MIST-S) 5 area under the curve with respect to increase during the stress condition; CFTlatency 5 time from CFT onset until maximum bradycardia.
with CFTlatency as independent variable and AUCI(MIST-S) of cortisol as dependent variable. CFTlatency explained 17.9% of total variance in AUCI(MIST-S) of cortisol (Table 3). Associations Between CFTlatency and Cardiopulmonary and Psychological Stress Response After the first step identifying CFTlatency to be correlated with HPA function, in a second step, we were interested in identifying stress responses associated with CFTlatency. Linear regression analyses were conducted entering CFTlatency as independent variable, and AUCI of HR, AUCI of RSATR, or alterations in psychological measures in response to the MIST-S as dependent variables. AUCI(MIST-S) of fb was controlled for when examining AUCI(MIST-S) of RSATR. CFTlatency was not a significant predictor of the AUCI of HR and RSATR in response to the MIST-S (Table 3). In contrast, CFTlatency as independent variables explained 36.5% of total variance in the stress response of mood. Therefore, longer latency until peak bradycardia was associated with worse mood after the MIST-S (Figure 2B). Discussion The main objective of the present study was to test the association between the function of the VN and the HPA axis, with a special focus on the reactivity of the two systems. In this context, longer latency to peak bradycardia during the CFT was associated with higher cortisol and more negative mood response to the stress task, while smaller maximal bradycardia was associated with more tiredness in response to the stress task. Furthermore, for the first time, the MIST-S was demonstrated to be an effective stress test for inducing a strong vagal inhibition, therefore increasing its validity to induce a multidimensional stress response. Our findings replicate those of prior investigations showing an increase in cortisol concentration in response to the MIST-S, indicating an effective stress-induced stimulation of the HPA axis (Dedovic et al., 2005; Pruessner, Dedovic, Khalili-Mahani, Engert, Pruessner, et al., 2008; Soliman, O’Driscoll, Pruessner, Holahan, Boileau, Gagnon, & Dagher, 2008). Furthermore, HR
increased continuously throughout the stress condition, suggesting a successive increase in stress load from block to block, and recovered immediately after termination of the MIST-S. This is the first study to examine the effects of the MIST-S on the PSNS. RSATR decreased during the stress task, with a peak depression during the final block, indicating an effective stress-induced inhibition of the VN. Additionally, we found a significant increase in rated stress and a significant decrease in mood, wakefulness, and calmness ratings after the MIST-S, corresponding to prior studies (Pruessner, Champagne, Meaney, & Dagher, 2004; Soliman et al., 2008). The primary aim of the current study was to investigate the association between the function of the VN and the HPA axis. Therefore, in addition to baseline and stress-induced inhibition or stimulation parameters, the CFT was conducted to further collect data on induced stimulation of the VN. An inhibiting role of the VN on the HPA axis can be assumed (Thayer & Sternberg, 2006). Nevertheless, evidence examining the relation of HPA axis and vagal function is contradictory, as measured by cortisol and HRV associations. In the present study, we found a reduced cortisol response elicited by the MIST-S to be associated with a faster achievement of peak bradycardia (i.e., short CFTlatency), indicating an inverse relationship between the function of the VN and the HPA axis. This inverse association is supported by Johnsen et al. (2002), who reported that a group of healthy men with higher baseline HRV (median split) showed lower levels of cortisol in response to stressful cognitive tasks compared to a group with lower baseline HRV. Similarly, Doussard-Roosevelt et al. (2003) found an inverse relationship between stress responses of RSA and cortisol to negative affect tasks in kindergarten children. Thus, a higher decrease in RSA was associated with a higher increase in cortisol in response to the emotional tasks. Further support for an inverse relation between the two systems is reported by Thayer et al. (2006). In a large sample of healthy men, they found a weak inverse relationship between HRV assessed over repeated hours (o24 h) and overnight urinary cortisol after a working day, variables that therefore possibly represent everyday-life-stress measurement (e.g., work-related stress). Hence, subjects with higher HRV revealed lower urinary cortisol levels. In the present study, we found no significant correlation between the baseline or the AUCI(MIST-S) of RSATR and the AUCI(MIST-S) of cortisol. This is in line with most reports (Altemus et al., 2001; Cacioppo et al., 1995; Doussard-Roosevelt et al., 2003; Gunnar et al., 1995; Heilman et al., 2008), which also found no relation between HRV and cortisol in response to different stressors (social, psychological, physical stress tasks). One study (Gunnar et al., 1995), even found a significant positive association between baseline RSA and stress response of cortisol, with newborns with higher baseline levels of RSA showing higher cortisol levels in response to the stress task. The authors argued, referring to Porges (1991), that higher vagal tone should show greater physiological responses to stressors. In addition, we found no association between baseline levels of RSATR and cortisol, which is also in line with other studies (Gunnar et al., 1995; Johnsen et al., 2002). Taken together, the rarely reported association between HRV and cortisol is mostly limited to stress-induced alterations, especially of the HPA axis. While we were unable to support this association, we found a strong association between CFTlatency as an indicator of the CFT-induced stimulation of the VN and cortisol stress response. There are different possible explanations
426 for this inconsistency. First, several studies examining associations between both HPA axis and vagal function under stress reveal no significant alterations of either cortisol or HRV in response to the applied stress task (e.g., Altemus et al., 2001; Heilman et al., 2008). Second, HRV possesses huge interindividual variations (e.g., Grossman & Taylor, 2007), and is therefore less adequately applicable for interindividual comparisons, but is better for intraindividual comparisons. Therefore, the CFT-induced vagal stimulation response might be more adequate for interindividual comparisons. Third, due to its high adaptation speed to changing demands, vagal contribution in general is of highest importance at the beginning of a demand, since it is the first and fastest way to adapt to a stimulus (Porges, 2001). This is especially true for the CFT, which imitates the diving reflex by directly provoking a lowering in heart activity through trigeminal-vagal stimulation. Therefore, the latency until peak bradycardia during the CFT might be a better and more sensitive index of vagal participation or influence than the more raw estimation of vagal function estimated by baseline or changes of averaged levels of RSATR. Further indirect support for an inverse relationship between the function of the VN and the HPA axis, but without examining cortisol and cardiopulmonary data concomitantly, is provided by O’Keane, Dinan, Scott, and Corcoran (2005), who examined responses to the corticotropin-releasing-hormone (CRH) challenge test in patients with chronic depression before and after three months of treatment with vagus nerve stimulation (VNS), an invasive treatment shown to have good effects in treatmentresistant depression (Rush et al., 2000). They reported significantly increased responses of adrenocorticotropic hormone (ACTH) and cortisol before treatment compared to intraindividual posttreatment responses, and in comparison to a control group. The negative association might point to bidirectional connections between vagal nuclei in the medulla oblongata and the hypothalamus (Benarroch, 1997; Palkovits, 1999). Furthermore, evidence from imaging studies supports interconnections between CAN structures, the hypothalamus, and the vagus nerve. For example, the salivary cortisol in response to stress has been shown to be positively associated with hypothalamic activity (A˚hs, Furmark, Michelga˚rd, La˚ngstro¨m, Appel, et al., 2006), and negatively correlated with medial PFC (A˚hs et al., 2006; Kern, Oakes, Stone, McAuliff, Kirschbaum, & Davidson, 2008), suggesting a possible phasic loss of inhibitory control of the medial PFC over subcortical regions during stress. Similarly, under stressful conditions, associations between High Frequency (HF) HRV (Task Force, 1996) as an index of cardiovagal activity and the activity of several structures of the CAN were demonstrated (Gianaros, Van Der Veen, & Jennings, 2004; Lane, McRae, Reiman, Chen, Ahern, & Thayer, 2009; Matthews, Paulus, Simmons, Nelesen, & Dimsdale, 2004). Analyses of the role of vagal function during acute stress revealed a protective role not only with regard to cortisol but also to mood. Subjects with a longer latency to peak bradycardia during the CFT showed a more negative affect in response to the stressor. This is in line with Ingjaldsson, Laberg, and Thayer (2003), who found a negative correlation of HRV with negative mood but a positive correlation of HRV with positive mood in a population of alcoholics and healthy controls (Ingjaldsson et al., 2003). Furthermore, we found that subjects with smaller relative bradycardia (i.e., lower CFTmax values) were more tired after the stress task compared to subjects with stronger bradycardia (i.e., higher CFTmax values). This is in line with the reported positive
R. La Marca et al. association between HRV and efficient attentional regulation (Johnsen, Thayer, Laberg, Wormnes, Raadal, et al., 2003), which might offer an explanation for the present findings. Additional support for this interpretation is provided by the findings of an inverse course of effort and HRV in burnout patients and healthy controls during a repeated Stroop color word task over one day (Zanstra, Schellenkens, Schaap, & Kooistra, 2006). Controls showed a decrease in HRV and effort over the day, while at the same time, burnout patients revealed an increase in HRV, effort, and tiredness. We found no significant association between the CFT parameters and the cardiopulmonary stress response. One possible explanation could relate to the high interindividual differences in RSATR mentioned previously (Grossman & Taylor, 2007). Furthermore, the CFT parameters indicate a short-term dynamic vagal characteristic. This characteristic might have more influence on the HPA axis response, while vagal baseline, as a less dynamic characteristic of the vagus nerve, was more predictive of the AUCI(MIST-S) of RSATR (data not shown). Referring to HR, the AUCI(MIST-S) of RSATR was negatively associated with the AUCI(MIST-S) of HR (data not shown). Therefore, this finding indicates that the HR response might be more dependent on an acute, concomitant vagal response. With regard to the CFT, it is proposed as a method to test vagal function (Khurana & Wu, 2006). Although, unfortunately, no study has examined the neural associations of latency and maximum bradycardia, the CFT was found to elicit several activity alterations in different CAN structures (Brown, Sanya, & Hilz, 2003; Harper, Macey, Henderson, Woo, Macey, et al., 2003). Interestingly, low vagal response during the CFT was associated with depressed mood (Hughes & Stoney, 2000), indicating an impaired vagal regulation in depression, which is often reported to show increased cortisol concentrations (Wong, Kling, Munson, Listwak, Licinio, et al., 2000) and decreased vagal tone (van der Kooy, van Hout, van Marwijk, de Haan, Stehouwer, & Beekman, 2006). Similar results were found for hostility (Ruiz, Uchino, & Smith, 2006), while an inverse association between anger control and cortisol reactivity was also determined (Gouin, Kiecolt-Glaser, Malarkay, & Glaser, 2008). Furthermore, a missed vagal responsiveness to mental challenge was found in posttraumatic stress disorder (Sahar, Shalev, & Porges, 2001), a disorder which shows an attenuated cortisol response to acute stress (MacMillan, Georgiades, Duku, Steiner, Niec, et al., 2009). Since a faster response during the CFT is associated with a reduced cortisol response to stress, paralleled by better mood, it seems appealing to interpret a fast vagal response as healthprotective. Explanations for a beneficial interpretation of the results is provided by the function of the diving reflex per se, which elicits bradycardia as a protective strategy to conserve oxygen (Rozloznik, Paton, & Dutschmann, 2009), the cardioprotective role of vagal activity (Fallen, 2005), the association between lower vagal activity and morbidity (Thayer & Brosschot, 2005; Thayer & Lane, 2007), and the therapeutic effects of VNS in therapy-resistant patients (Milby, Halpern, & Baltuch, 2008). Furthermore, the protective interpretation of vagal functionality during stress is in line with the assumption of vagal activity as being a type of resource when demands of the environment request emotional regulation (Thayer & Lane, 2009). Evidence for the interpretation of the VN as a resource was provided among others by Ruiz-Padial, Sollers, Vila, and Thayer (2003), who demonstrated a more differentiated emotional startle
Vagal function and cortisol stress response
427
response to emotional pictures in subjects with high HRV compared to the group with low HRV. Furthermore, HRV was found to be positively associated with positive mood and negatively associated with negative mood (Ingjaldsson et al., 2003). Although our hypotheses were mostly supported, the current study holds a number of limitations. First, we examined a small sample size, including only healthy and medication-free males. Therefore, results are restricted to a group of healthy, well-educated, young men and cannot be generalized to women or the general population. Moreover, the CFT in this study was used to determine vagal function (i.e., acute vagal stimulation), suggested by us to be indicative of the flexibility of the whole CAN. It should be mentioned on a critical note that CFT responses are initially mediated by the trigeminal nerve, and, therefore, responses could also be influenced by the latter responsiveness. Despite these limitations, however, our study does have a number of methodological strong points. First, we examined several vagal characteristics by examining not only vagal baseline, but also reactivity to the CFT (acute vagal stimulation) and the MIST-S (acute vagal inhibition). Second, by favoring the MISTS over other strong stressors known to induce a multidimensional stress response (e.g., Trier Social Stress Test, TSST; Kirschbaum, Pirke, & Hellhammer, 1993), several potentially disruptive factors such as postural changes or walking (Chan, Lin, Chao, & Lin, 2007; Nater, La Marca, Florin, Moses, Langhans, Koller, & Ehlert, 2006), and vocalization (Bernardi, Wdowczyk-Szulc, Valenti, Castoldi, Passino, et al., 2000; Sloan, Korten, & Myers, 1991) were controlled for. Therefore, the MIST-S seems particularly useful to investigate the relationship between different
response variables that are not equally susceptible to the potentially disruptive factors. Notably, additional controlling for credibility of the stress task (‘‘How much do you believe that your bad performance was due to a bug in the program?’’; 10-point Likert scale, distributed at the end of the stress examination) did not significantly change any of the reported associations. In summary, our results underline the validity of the MIST-S to provoke a multidimensional stress response including vagal inhibition. The inverse relationship between the vagal function measured during the CFT and stress responses of cortisol and mood support the assumption that the HPA axis plays an important role linking lower vagal function and morbidity. Furthermore, the CFT might be applied as an index to examine biopsychological stress reactivity of individuals. Moreover, we speculate that vagal baseline might reflect the tonic state of the interconnected CAN, while the CFT response might reflect the dynamic capacity of the organism to respond to rapidly changing environmental demands. Therefore, the CFT might be an interesting addition in the measurement of vagal function. However, the mechanisms underlying latency of bradycardia during the CFT are not entirely understood, and, therefore, further research is needed. The association between CFT-induced vagal stimulation and other HPA axis alterations, e.g., in response to other stressors or during awakening or the whole day, should be examined in future studies. Moreover, the reported association should be examined in subjects with (sub-)clinical somatic and mental disorders in prospective studies in order to gain further insight into the role of the HPA axis linking lower vagal function with morbidity.
REFERENCES A˚hs, F., Furmark, T., Michelga˚rd, A˚., La˚ngstro¨m, B., Appel, L., Wolf, O. T., et al. (2006). Hypothalamic blood flow correlates positively with stress-induced cortisol levels in subjects with social anxiety disorder. Psychosomatic Medicine, 68, 859–862. Altemus, M., Redwine, L. S., Leong, Y. M., Frye, C. A., Porges, S. W., & Carter, C. S. (2001). Response to laboratory psychosocial stress in postpartum women. Psychosomatic Medicine, 63, 814–821. Arnold, R. W. (1999). The human heart rate response profiles to five vagal maneuvers. Yale Journal of Biology and Medicine, 72, 237–244. Benarroch, E. E. (1997). Central autonomic network: Functional organization and clinical correlations. Armonk, NY: Futura Publishing Company, Inc. Bernardi, L., Wdowczyk-Szulc, J., Valenti, C., Castoldi, S., Passino, C., Spadacini, G., & Sleight, P. (2000). Effects of controlled breathing, mental activity and mental stress with or without verbalization on heart rate variability. Journal of the American College of Cardiology, 35, 1462–1469. Bernstein, A. L., Barkan, H., & Hess, T. (2006). Vagus nerve stimulation therapy for pharmacoresistant epilepsy: Effect on health care utilization. Epilepsy & Behavior, 10, 134–137. Bodis, J., Boncz, I., & Kriszbacher, I. (in press). Permanent stress may be the trigger of an acute myocardial infarction on the first work-day of the week. International Journal of Cardiology. Brown, C. M., Sanya, E. O., & Hilz, M. J. (2003). Effect of cold face stimulation on cerebral blood flow in humans. Brain Research Bulletin, 61, 81–86. Cacioppo, J. T., Malarkey, W. B., Kiegolt-Glaser, J. K., Uchino, B. N., Sgoutas-Emch, S. A., Sheridan, J. F., et al. (1995). Heterogeneity in neuroendocrine and immune responses to brief psychological stressors as a function of autonomic cardiac activation. Psychosomatic Medicine, 57, 154–164. Chan, H. L., Lin, M. A., Chao, P. K., & Lin, C. H. (2007). Correlates of the shift in heart rate variability with postures and walking by timefrequency analysis. Computer Methods and Programs in Biomedicine, 86, 124–130.
Christensen, M. V., & Kessing, L. V. (2001). The hypothalamo-pituitaryadrenal axis in major affective disorder: A review. Nordic Journal of Psychiatry, 55, 359–363. Dedovic, K., Renwick, R., Khalili-Mahani, N., Engert, V., Lupien, S. J., & Pruessner, J. C. (2005). The Montreal Imaging Stress Task: Using functional imaging to investigate the effects of perceiving and processing psychosocial stress in the human brain. Journal of Psychiatry & Neuroscience, 30, 319–325. Doussard-Roosevelt, J. A., Montgomery, L. A., & Proges, S. W. (2003). Short-term stability of physiological measures in kindergarten children: Respiratory sinus arrhythmia, heart period, and cortisol. Developmental Psychobiology, 43, 230–242. Dressendorfer, R. A., Kirschbaum, C., Rohde, W., Stahl, F., & Strasburger, C. J. (1992). Synthesis of a cortisol-biotin conjugate and evaluation as a tracer in an immunoassay for salivary cortisol measurement. The Journal of Steroid Biochemistry and Molecular Biology, 43, 683–692. Fallen, E. L. (2005). Vagal afferent stimulation as a cardioprotective strategy? Introducing the concept. Annals of Noninvasive Electrocardiology, 10, 441–446. Gianaros, P. J., Van Der Veen, F. M., & Jennings, J. R. (2004). Regional cerebral blood flow correlates with heart period and high-frequency heart period variability during working-memory tasks: Implications for the cortical and subcortical regulation or cardiac autonomic activity. Psychophysiology, 41, 521–530. Girdler, S. S., Maixner, W., Naftel, H. A., Stewart, P. W., Moretz, R. L., & Light, K. C. (2005). Cigarette smoking, stress-induced analgesia and pain perception in men and women. Pain, 114, 372–385. Gouin, J. F., Kiecolt-Glaser, J. K., Malarkay, W. B., & Glaser, R. (2008). The influence of anger expression on wound healing. Brain, Behavior, and Immunity, 22, 699–708. Green, S. B., Salkind, N. J., & Akey, T. M. (2000). Using SPSS for Windows (2nd ed). Englewood Cliffs, NJ: Prentice Hall. Grossman, P., von Beek, J., & Wientjes, C. (1990). A comparison of three quantification methods for estimation of respiratory sinus arrhythmia. Psychophysiology, 27, 702–714.
428 Grossman, P., Wilhelm, F. H., & Spoerle, M. (2004). Respiratory sinus arrhythmia, cardiac vagal control, and daily activity. American Journal of Physiology, Heart & Circulatory Physiology, 287, H728–H734. Grossman, P., Spoerle, M., & Wilhelm, F. H. (2006). Reliability of respiratory tidal volume estimation by means of ambulatory inductive plethysmography. Biomedical Sciences Instrumentation, 42, 193–198. Grossman, P., & Taylor, E. W. (2007). Toward understanding respiratory sinus arrhythmia: Relations to cardiac vagal tone, evolution and biobehavioral functions. Biological Psychology, 74, 263–285. Gunnar, M. R., Porter, F. L., Wolf, C. M., Rigatsu, J., & Larson, M. C. (1995). Neonatal stress reactivity: Prediction to later emotional temperament. Child Development, 66, 1–13. Harper, R. M., Macey, P. M., Henderson, L. A., Woo, M. A., Macey, K. E., Frysinger, R. C., et al. (2003). fMRI responses to cold pressor challenges in control and obstructive sleep apnea subjects. Journal of Applied Physiology, 94, 1583–1595. Hautzinger, M., & Bailer, M. (1992). Allgemeine Depressions Skala. Manual. Konstanz und Mainz: Beltz. Heilman, K. J., Bal, E., Bazhenova, O. V., Sorokin, Y., Perlman, S. B., Hanley, M. C., & Porges, S. W. (2008). Physiological responses to social and physical challenges in children: Quantifying mechanisms supporting social engagement and mobilization behaviors. Developmental Psychobiology, 50, 171–182. Heilman, K. J., & Porges, S. W. (2007). Accuracy of the LifeShirts (Vivometrics) in the detection of cardiac rhythms. Biological Psychology, 75, 300–305. Hord, E. D., Evans, M. S., Mueed, S., Adamolekun, B., & Naritoku, D. K. (2003). The effect of vagus nerve stimulation on migraines. The Journal of Pain, 4, 530–534. Hughes, J. W., & Stoney, C. M. (2000). Depressed mood is related to high-frequency heart rate variability during stressors. Psychosomatic Medicine, 62, 796–803. Ingjaldsson, J. T., Laberg, J. C., & Thayer, J. F. (2003). Reduced heart rate variability in chronic alcohol abuse: Relationship with negative mood, chronic thought suppression, and compulsive drinking. Biological Psychiatry, 54, 1427–1436. Johnsen, B. H., Hansen, A. L., Sollers III, J. J., Murison, R., & Thayer, J. F. (2002). Heart rate variability is inversely related to cortisol reactivity during cognitive stress. Psychosomatic Medicine, 64, 148. Johnsen, B. H., Thayer, J. F., Laberg, J. C., Wormnes, B., Raadal, M., Skaret, E., et al. (2003). Attentional and physiological characteristics of patients with dental anxiety. Journal of Anxiety Disorders, 17, 75–87. Kern, S., Oakes, T. R., Stone, Ch. K., McAuliff, E. M., Kirschbaum, C., & Davidson, R. J. (2008). Glucose metabolic changes in the prefrontal cortex are associated with HPA axis response to a psychosocial stressor. Psychoneuroendocrinology, 33, 517–529. Khurana, R. K. (2007). Cold face test: Adrenergic phase. Clinical Autonomic Research, 17, 211–216. Khurana, R. K., Watabiki, S., Hebel, J. R., Toro, R., & Nelson, E. (1980). Cold face test in the assessment of trigeminal-brainstem-vagal function in humans. Annals of Neurology, 7, 144–149. Khurana, R. K., & Wu, R. (2006). The cold face test: A non-baroreflex mediated test of cardiac vagal function. Clinical Autonomic Research, 16, 202–207. Kirschbaum, C., Kudielka, B. M., Gaab, J., Schommer, N. C., & Hellhammer, D. H. (1999). Impact of gender, menstrual cycle phase, and oral contraceptives on the activity of the hypothalamus-pituitary-adrenal axis. Psychosomatic Medicine, 61, 154–162. Kirschbaum, C., Pirke, K. M., & Hellhammer, D. H. (1993). The ‘Trier Social Stress Test’FA tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology, 28, 76–81. La Marca, R., Nedeljkovic, M., Yuan, L., Maercker, A., & Ehlert, U. (2010). Effects of auricular electrical stimulation on vagal activity in healthy men: Evidence from a three-armed randomized trial. Clinical Science, 118, 537–546. Lane, R. D., McRae, K., Reiman, E. M., Chen, K., Ahern, G. L., & Thayer, J. F. (2009). Neural correlates of heart rate variability during emotion. NeuroImage, 44, 213–222. Lee, M. S., Lee, J. S., Lee, J. Y., Corne´lissen, G., Otsuka, K., & Halberg, F. (2003). About 7-day (circaseptan) and circadian changes in cold pressor test (CPT). Biomedicine & Pharmacotherapy, 57, 39s–44s. MacMillan, H. L., Georgiades, K., Duku, E. K., Steiner, M., Niec, A., Tanaka, M., et al. (2009). Cortisol response to stress in female youths exposed to childhood maltreatment: Results of the youth mood project. Biological Psychiatry, 66, 62–68.
R. La Marca et al. Maschke, C., Harder, J., Corne´lissen, G., Hecht, K., Otsuka, K., & Halberg, F. (2003). Chronoecoepidemiology of ‘‘strain’’: Infradian chronomics of urinary cortisol and catecholamines during nightly exposure to noise. Biomedicine & Pharmacotherapy, 57, 126s–135s. Matthews, S. C., Paulus, M. P., Simmons, A. N., Nelesen, R. A., & Dimsdale, J. E. (2004). Functional subdivisions within anterior cingulate cortex and their relationship to autonomic nervous system function. NeuroImage, 22, 1151–1156. Merrill, C. A., Jonsson, M. A., Minthon, L., Ejnell, H., C-son Silander, H., Blennow, K., et al. (2006). Vagus nerve stimulation in patients with Alzheimer’s disease: Additional follow-up results of a pilot study through 1 year. The Journal of Clinical Psychiatry, 67, 1171–1178. Milby, A. H., Halpern, C. H., & Baltuch, G. H. (2008). Vagus nerve stimulation for epilepsy and depression. Neurotherapeutics, 5, 75–85. Nater, U. M., La Marca, R., Florin, L., Moses, A., Langhans, W., Koller, M. M., & Ehlert, U. (2006). Stress-induced changes in human salivary alpha-amylase activityFAssociations with adrenergic activity. Psychoneuroendocrinology, 31, 49–58. O’Keane, V., Dinan, T. G., Scott, L., & Corcoran, C. (2005). Changes in hypothalamic-pituitary-adrenal axis measures after vagus nerve stimulation therapy in chronic depression. Biological Psychiatry, 58, 963–968. Palkovits, M. (1999). Interconnections between the neuroendocrine hypothalamus and the central autonomic system. Frontiers in Neuroendocrinology, 20, 270–295. Porges, S. W. (1991). Vagal tone: An autonomic mediator of affect. In J. Garber & K. A. Dodge (Eds.), The development of emotion regulation and dysregulation (pp. 111–128). New York: Cambridge University Press. Porges, S. W. (2001). The polyvagal theory: Phylogenetic substrates of a social nervous system. International Journal of Psychophysiology, 42, 123–146. Pruessner, J. C., Champagne, F., Meaney, M. J., & Dagher, A. (2004). Dopamine release in response to a psychological stress in humans and its relationship to early life maternal care: A positron emission tomography study using [11C]Raclopride. The Journal of Neuroscience, 24, 2825–2831. Pruessner, J. C., Dedovic, K., Khalili-Mahani, N., Engert, V., Pruessner, M., Buss, C., et al. (2008). Deactivation of the limbic system during acute psychosocial stress: Evidence from positron emission tomography and functional magnetic resonance imaging studies. Biological Psychiatry, 63, 234–240. Pruessner, J. C., Kirschbaum, C., Meinlschmid, G., & Hellhammer, D. H. (2003). Two formulas for computation of the area under the curve represent measures of total hormone concentration versus time-dependent change. Psychoneuroendocrinology, 28, 916–931. Radloff, L. S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 3, 385–401. Rozloznik, M., Paton, J. F., & Dutschmann, R. (2009). Repetitive paired stimulation of nasotrigeminal and peripheral chemoreceptor afferents cause progressive potentiation of the diving bradycardia. American Journal of PhysiologyFRegulatory, Integrative and Comparative Physiology, 296, R80–R87. Ruiz, J. M., Uchino, B. N., & Smith, T. W. (2006). Hostility and sex differences in the magnitude, duration, and determinants of heart rate response to forehead cold pressor: Parasympathetic aspects of risk. International Journal of Psychophysiology, 60, 274–283. Ruiz-Padial, E., Sollers III, J. J., Vila, J., & Thayer, J. F. (2003). The rhythm of the heart in the blink of an eye: Emotion-modulated startle magnitude covaries with heart rate variability. Psychophysiology, 40, 306–313. Rush, A. J., George, M. S., Sackeim, H. A., Marangell, L. B., Husain, M. M., Giller, C., et al. (2000). Vagus nerve stimulation (VNS) for treatment-resistant depression: A multicenter study. Biological Psychiatry, 47, 276–286. Sahar, T., Shalev, A. Y., & Porges, S. W. (2001). Vagal modulation of responses to mental challenge in posttraumatic stress disorder. Biological Psychiatry, 49, 637–643. Sato, N., Myake, S., Akatsu, J., & Kumashiro, M. (1995). Power spectral analysis of heart rate variability in healthy young women during the normal menstrual cycle. Psychosomatic Medicine, 57, 331–335. Scholz, U., La Marca, R., Nater, U. M., Aberle, I., Ehlert, U., Hornung, R., et al. (2009). Go no-go performance under psychosocial stress: Beneficial effects of implementation intentions. Neurobiology of Learning and Memory, 91, 89–92.
Vagal function and cortisol stress response Sloan, R. P., Korten, J. B., & Myers, M. M. (1991). Components of heart rate reactivity during mental arithmetic with and without speaking. Physiology and Behaviour, 50, 1039–1045. Smyth, J. M., Ockenfels, M. C., Gorin, A. A., Catley, D., Porter, L. S., Kirschbaum, C., et al. (1997). Individual differences in the diurnal cycle of cortisol. Psychoneuroendocrinology, 22, 89–105. Soliman, A., O’Driscoll, G. A., Pruessner, J., Holahan, A. V., Boileau, I., Gagnon, D., & Dagher, A. (2008). Stress-induced dopamine release in humans at risk of psychosis: A [11C]Raclopride PET study. Neuropsychopharmacology, 33, 2033–2041. Steyer, R., Schwenkmezger, P., Notz, P., & Eid, M. (1997). Der Mehrdimensionale Befindlichkeitsfragebogen (MDBF). Handanweisung. Go¨ttingen: Hogrefe. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology (1996). Heart rate variabilityFStandards and measurement, physiological interpretation and clinical use. Circulation, 93, 1043–1065. Thayer, J. F. (2007). What the heart says to the brain (and vice versa) and why we should listen. Psychological Topics, 16, 241–250. Thayer, J. F., & Brosschot, J. F. (2005). Psychosomatics and psychopathology: Looking up and down from the brain. Psychoneuroendocrinology, 30, 1050–1058. Thayer, J. F., & Friedman, B. H. (2002). Stop that! Inhibition, sensitization, and their neurovisceral concomitants. Scandinavian Journal of Psychology, 43, 123–130. Thayer, J. F., Hall, M., Sollers III, J. J., & Fischer, J. E. (2006). Alcohol use, urinary cortisol, and heart rate variability in apparently healthy men: Evidence for impaired inhibitory control of the HPA axis in heavy drinkers. International Journal of Psychophysiology, 56, 244–250. Thayer, J. F., & Lane, R. D. (2000). A model of neurovisceral integration in emotion regulation and dysregulation. Journal of Affective Disorders, 61, 201–216.
429 Thayer, J. F., & Lane, R. D. (2007). The role of vagal function in the risk for cardiovascular disease and mortality. Biological Psychology, 74, 224–242. Thayer, J. F., & Lane, R. D. (2009). Claude Bernard and the heart-brain connection: Further elaboration of a model of neurovisceral integration. Neuroscience and Biobehavioral Reviews, 33, 81–88. Thayer, J. F., & Sternberg, E. (2006). Beyond heart rate variability: Vagal regulation of allostatic systems. Annals of the New York Academy of Sciences, 1088, 361–372. Tsigos, C., & Chrousos, G. P. (1994). Physiology of the hypothalamuspituitary-adrenal axis in health and dysregulation in psychiatric and autoimmune disorders. Endocrinology and Metabolism Clinics of North America, 23, 451–466. van der Kooy, K. G., van Hout, H. P., van Marwijk, H. W., de Haan, M., Stehouwer, C. D., & Beekman, A. T. (2006). Differences in heart rate variability between depressed and non-depressed elderly. International Journal of Geriatric Psychiatry, 21, 147–150. Wong, M. L., Kling, M. A., Munson, P. J., Listwak, S., Licinio, J., Prolo, P., et al. (2000). Pronounced and sustained central hypernoradrenergic function in major depression with melancholic features: Relation to hypercortisolism and corticotropin-releasing hormone. Procedures of the National Academy of Sciences of the United States of America, 97, 325–30. Zanstra, Y. J., Schellenkens, J. M. H., Schaap, C., & Kooistra, L. (2006). Vagal and sympathetic activity in burnouts during a mentally demanding workday. Psychosomatic Medicine, 68, 583–590.
(Received August 6, 2009; Accepted May 10, 2010)
Psychophysiology, 48 (2011), 430–436. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01072.x
Spatial layout of letters in nonwords affects visual short-term memory load: Evidence from human electrophysiology
DAVID PRIME,a ROBERTO DELL’ACQUA,b MARTIN ARGUIN,a FRE´DE´RIC GOSSELIN,a and PIERRE JOLICŒURa a
De´partement de Psychologie, Universite´ de Montre´al, Montre´al, Que´bec, Canada Department of Developmental Psychology and Center for Cognitive Science, University of Padova, Padova, Italy
b
Abstract The sustained posterior contralateral negativity (SPCN) was used to investigate the effect of spatial layout on the maintenance of letters in VSTM. SPCN amplitude was measured for words, nonwords, and scrambled nonwords. We reexamined the effects of spatial layout of letters on SPCN amplitude in a design that equated the mean frequency of use of each position. Scrambled letters that did not form words elicited a larger SPCN than either words or nonwords, indicating lower VSTM load for nonwords presented in a typical horizontal array than the load observed for the same letters presented in spatially scrambled locations. In contrast, prior research has shown that the spatial extent of arrays of simple stimuli did not influence the amplitude of the SPCN. Thus, the present results indicate the existence of encoding and VSTM maintenance mechanisms specific to letter and word processing. Descriptors: Sustained posterior contralateral negativity (SPCN), Visual short-term memory (VSTM), Lexical status, Event-related potentials (ERPs), Reading
whereas psycholinguists have been concerned with the spatial organization of letters at early stages of processing, much less is known about the issue of whether the spatial organization of letters composing a string also influences how letter strings are retained in visual short-term memory (VSTM). VSTM is a fundamental memory system that enables us to retain visual features and objects for a short period of time (in the order of several seconds). This memory system appears to be important for efficient perceptual and cognitive processing in tasks that depend on visual input (Jolicœur & Dell’Acqua, 1998; Jolicœur, Sessa, Dell’Acqua, & Robitaille, 2006a, b; Prime & Jolicœur, in press). The ability to transfer letter information to VSTM in an efficient and error-free manner may be particularly important for reading (Bosse, Tainturier, & Valdois, 2007; Goulandris & Snowling, 1991; Valdois, Bosse, Ans, Carbonnel, Zorman, et al., 2003). For these reasons, we investigated interactions between VSTM and the processing of letters in the context of words and nonwords, and the role of the spatial layout of letters for memory. Predovan, Prime, Arguin, Gosselin, Dell’Acqua, and Jolicœur (2009) investigated the effect of lexical status on the maintenance of letter strings in visual short-term memory. They hypothesized that the lexical status of letter strings that formed words would reduce the load in VSTM in comparison to random letter strings that did not form words, because of the availability of pre-existing long-term memory (LTM) representations for words. In order to test this hypothesis, the sustained posterior
Encoding visually displayed letter strings is enabled by complex perceptual processes that are not entirely compatible with the mechanisms underlying visual processing of nonletter stimuli. For example, letter perception violates the acuity gradient principle, according to which the visual stimuli presented at fixation are processed more efficiently than stimuli presented in the periphery. In striking contrast, studies on letter identification have revealed that the first letter of a string is the most accurately perceived, even when the first letter is far from fixation (e.g., Hammond & Green, 1982; Tydgat & Grainger, 2009). This and related findings (see Whitney, 2001) have suggested that processing of letter strings relies on specialized brain and functional subroutines that are particularly influenced by the spatial organization of letters within a string (Blais, Fiset, Arguin, Jolicœur, Bub, & Gosselin, 2009; Tydgat & Grainger, 2009). Interestingly, This research was made possible by a grant from the Natural Sciences and Engineering Research Council of Canada awarded to PJ, by a grant from the Canadian Institutes of Health Research awarded to PJ, by equipment funded by the Canada Fund for Innovation awarded to PJ, by the Canada Research Chairs program, by infrastructure support from the Fonds Quebe´cois pour la Recherche en Sante´ du Que´bec, by a team research grant from the Fonds Que´becois pour la Recherche sur la Nature et la Technologie awarded to MA, FG, and PJ, and by research support from Universite´ de Montre´al. Address correspondence to: David J. Prime, Department of Psychology, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada, V5A 1S6. E-mail:
[email protected] 430
Lexical status and VSTM contralateral negativity (SPCN), an electrophysiological marker of storage in VSTM, was used to measure VSTM load for word and nonword letter strings. A growing body of evidence (e.g., Jolicœur, Brisson, & Robitaille, 2008; Klaver, Talsma, Wijers, Heinze, & Mulder, 1999; McCollough, Machizawa, & Vogel, 2007; Perron, Lefebvre, Robitaille, Brisson, Gosselin, et al., 2009; Robitaille, Grimault, & Jolicœur, 2009; Vogel & Machizawa, 2004) supports the hypothesis that the SPCN reflects neural activity specifically related to the maintenance of information stored in VSTM. The SPCN is observed following the encoding of laterally presented visual stimuli. In order to avoid confounding effects from low-level stimulus differences, the target stimuli in one visual field are presented with an equivalent set of distractor stimuli in the other visual field. The SPCN is observed as a relatively more negative event-related potentials (ERP) at posterior electrodes contralateral to the encoded visual field (e.g., the voltage is more negative at electrode PO8 than at PO7 for stimuli encoded from the left visual field). In tasks that require the maintenance of visual information for short retention intervals, an SPCN effect is observed during the retention interval beginning at about 300 ms from the onset of the memory array. Importantly, the amplitude of the SPCN increases as the number of target items increased (Dell’Acqua, Sessa, Toffanin, Luria, & Jolicœur, 2010; Luria, Sessa, Gotler, Jolicœur, & Dell’Acqua, 2010; McCollough et al., 2007; Perron et al., 2009; Robitaille et al., 2009; Vogel & Machizawa, 2004), reaching a maximum when the number of stimuli to be encoded equals or exceeds the estimated capacity of VSTM (Vogel & Machizawa, 2004). Predovan et al. (2009) thus predicted that the SPCN elicited by horizontal letter strings that formed words would be attenuated relative to the SPCN elicited by equal length horizontal strings that did not form words. The results were clear-cut in demonstrating that the lexical status of a letter string modulated activity in VSTM during the retention interval of a VSTM task. When the target string formed a word (word condition), the amplitude of the SPCN was reduced relative to when the target string did not form a word (nonword condition). The reduced SPCN for words relative to that for nonwords suggested that reliance on a low-level visual memory system was reduced when the task could be performed on the basis of rapidly activated higher-level representations. Furthermore, this effect of lexical status on the amplitude of the SPCN suggested that the SPCN could become a powerful tool to study basic mechanisms of attention, perception, and memory, supporting reading and individual differences in reading ability (Valdois et al., 2003). For exploratory purposes, Predovan et al. (2009) also included a condition in which the target string did not form a word, and each letter was randomly displaced either upwards or downwards from the horizontal meridian (scrambled condition). Surprisingly, the amplitude of the SPCN was affected by the spatial configuration of the target letters. The amplitude of the SPCN was substantially larger in the scrambled condition than in either of the other two conditions. In contrast, McCollough et al. (2007) found that the SPCN for colored squares was unaffected by the size of the region in which the squares were presented, suggesting that the spatial layout of simple stimuli does not influence the SPCN. Thus, the results from the scrambled letter condition suggest that there may be important differences in the manner in which VSTM encodes letter and nonletter stimuli, even when the letters do not form a word. Here we examine a possible alternative explanation for the results of the scrambled letter condition from Predovan et al.
431 (2009) that arises because the spatial positions of the target letters differed between conditions. In the word and nonword conditions, the horizontal letter strings were always on the horizontal midline. In the scrambled condition, each target letter could appear in either the upper or lower visual field as well as on the horizontal midline. Perron et al. (2009) found that the SPCN elicited by equivalent stimuli was larger for stimuli presented in the lower visual field than for stimuli presented in the upper visual field. However, to our knowledge, a comprehensive mapping of SPCN amplitude across the visual field has not been performed. It is possible that letters encoded from lower visual field positions may have produced a larger SPCN than letters encoded from upper and midline visual field positions, and these effects may be non-linear, leading to a net increase in SPCN amplitude for the scrambled condition. Because of the relevance of Predovan et al.’s (2009) results for our understanding of the role of the spatial layout of letters in words and nonwords in VSTM and for our understanding of basic mechanisms of letter encoding and reading, a re-examination of this issue was particularly important. Here we attempted to replicate their results using an experimental design that eliminated the potentially confounding effect of letter position on SPCN amplitude. We varied the location of the target letters in all three conditions and equated the mean frequency of use of each position across conditions. In the word and nonword conditions, the target strings were presented equally often in one of three positions: on the horizontal meridian or displaced downwards by 0.951 or 1.901. In the scrambled condition, each letter was displaced downwards from the horizontal meridian by 01, 0.951, or 1.901. Furthermore, letters were assigned to spatial positions such that, averaged across trials, the target letters appeared equally often in each spatial position. If, as suggested by Predovan et al.’s results, the spatial arrangement of target letters does affect VSTM load, the scrambled condition should produce a larger SPCN than either of the other two conditions. However, if the larger SPCN observed in the scrambled condition by Predovan et al. was due to the absolute spatial positions of the target letters, then there should be no difference in the amplitude of the SPCN between the nonword and the scrambled conditions.
Method Participants Forty-four native French speakers (ages 19–35 years, mean 5 23.1) participated in this experiment. All participants were naı¨ ve volunteers and were paid $20 Cdn. All participants reported normal or corrected-to-normal vision and no history of neurological problems. Seven participants were excluded from data analysis because an excessive number of trials were rejected, mainly due to a tendency to shift their gaze towards the attended hemifield but also due to electroencephalogram (EEG) artifacts. On average, the 37 remaining subjects moved their eyes less than .151 of visual angle in the direction of the target (based on averaged horizontal electro-oculogram (EOG) results; see Hillyard & Galambos, 1970; Lins, Picton, Berg, & Scherg, 1993). Task and Design The experimental task required participants to encode and retain the identity of three visually presented letters. The target letters were equally likely to be presented in the left or right visual field.
432
D. Prime et al.
A retention period of one second ensured that the task could not easily be carried out on the basis of a very low-level form of memory such as visible persistence (Coltheart, 1980). After the retention interval, the participants performed a two-alternative forced choice identification test on the identity of one of the letters. There were three within-participant conditions based on the lexical status and spatial layout of the target letters. In the word condition, the three target letters were grouped into a horizontal string that formed a French word. In the nonword condition, the target letters were grouped into a horizontal string that did not form a valid word. In the scrambled letters condition, the letters were not aligned horizontally and did not form a word. The target letters in the nonword and scrambled conditions had a pronounceable sequence of consonants and vowels that closely matched the proportions of strings with particular sequence structure in the word condition (i.e., CVC, VCC, CCV, etc.). In addition, bigram and trigram frequency were matched between the nonword and scrambled conditions. Stimuli The visual field cue was a centrally presented 0.21 grey arrow pointing towards the left or right. Target displays consisted of three letters presented on each side of the vertical meridian (6 letters total). The letter stimuli were ! 0.71 tall capital letters (A–Z) presented in grey on a black background. In the word and nonword conditions, the letter strings were centered 2.41 to the left and right of fixation. The horizontal center-to-center distance between each letter was 1.31. The letter strings in these two conditions were presented equally often in one of three positions: on the horizontal meridian or displaced downwards by 0.951 or 1.901, which is illustrated in Figure 1. In the scrambled condition, the same spatial locations of letters used in the other conditions were used but, on any given trial, there was one letter in each row and column of the 3 " 3 matrix of possible positions, yielding 6 possible layouts that were used equally often across trials. The test displays had the same spatial arrangements as the target displays, except that one letter in each visual field was probed by two test letters centered 0.351 above and below the center of the former location of the probed letter (Figure 1). The horizontal position of the test letters was always different in the left and right
visual fields. The other letters were replaced by grey oval outlines. One of the two test letters matched the identity of the letter at the probed position in the target display. In the word condition, both test letters formed valid words in conjunction with the other letters in the target display. For example, in English, if the target letters had been SUN, the identity of the first letter could be tested with the letters S and F, each of which forms a word. Furthermore, the distractor letter string in the opposite hemifield was always of the same type as the string in the attended hemifield (i.e., word, nonword, or scrambled letters). Experimental Procedure The trial sequence is illustrated in Figure 1. Participants viewed the experimental stimuli displayed on a computer monitor from a distance of 57 cm and a chin rest was used to stabilize the head. Each trial began with the presentation of a central fixation cross that remained on screen throughout the trial. Participants were instructed to maintain their gaze at fixation during a trial, and to blink only between trials. An arrow cue indicating the visual field of the to-be-remembered letters replaced the fixation cross for 200 ms. After a further 800 ms delay, the target display was presented for 150 ms followed by a blank screen with the fixation cross. One second after the onset of the target display, the test display was presented. Participants were required to indicate which of the two test letters in the to-be-remembered visual field matched the letter in the target display. To select the letter, participants had to respond by pushing predetermined keys on the keyboard. Half of the subjects responded with the right hand (key ‘j’ to select the top letter and ‘n’ to select the bottom letter) and the other half responded with the left hand (keys ‘g’ and ‘b’). Accuracy feedback was provided immediately after the response, in the form of a plus sign for correct responses or a minus sign for errors. Participants initiated the next trial by pressing the space bar. The experimental session consisted of 18 practice trials and 720 experimental trials (240 trials per condition) divided into two blocks of 360 trials. Participants were required to rest between blocks and could rest at any time during the experiment. The experimental session lasted approximately one and a half hours.
Figure 1. Sequence of events in each trial for the word, nonword, and scrambled conditions.
Lexical status and VSTM Electrophysiological Recording and Analysis The EEG was recorded with active Ag/AgCl electrodes (BioSemi Active Two, Amsterdam, The Netherlands) mounted on an elastic cap at standard 10–10 scalps sites (Fp1, Fpz, Fp2, AF7, AF3, AFz, AF4, AF8, F7, F5, F3, F1, Fz, F2, F4, F6, F8, FT7, FC5, FC3, FC1, FCz, FC2, FC4, FC6, FT8, T7, C5, C3, C1, Cz, C2, C4, C6, T8, TP7, CP5, CP3, CP1, CPz, CP2, CP4, CP6, TP8, P9, P7, P5, P3, P1, Pz, P2, P4, P6, P8, P10, PO7, PO3, POz, PO4, PO8, O1, OZ, O2, Iz, M1, and M2). Eye position was monitored by both the horizontal and vertical EOG. The vertical EOG was recorded as the voltage between Fp1 and an electrode placed below the left eye. The horizontal EOG was recorded was recorded as the voltage between electrodes placed lateral to the external canthi. Amplified EEG and EOG channels were low-pass filtered at 67 Hz and digitized at 256 Hz. After acquisition the EEG channels were referenced to the average of the left and right mastoids and high-pass filtered at 0.01 Hz (half power cut-off). Trials containing blinks, eye movements and EEG artefacts were removed prior to ERP averaging by applying automated artefact detection routines (Luck, 2005). Blinks and eye movements were detected by a function that detects rapid steps in the voltage of the EOG channels. Artefacts in EEG channels were identified by functions that detect flat sections of EEG and sudden rapid changes in voltage. Participants who had less than 60% of trials remaining in one or more conditions after artifact rejection were excluded from further analysis. Separate ERP averages were calculated from EEG epochs time-locked to the presentation of the target display for each experimental condition and visual field of the target items. The resulting ERP averages were low-pass filtered at 8 Hz and baseline corrected relative to mean voltage of the 200-ms pre-stimulus interval. In order to isolate the SPCN activity, ERP waveforms from electrodes ipsilateral to the target items were subtracted from those from contralateral electrodes, and the resulting difference waves for each visual field were averaged. For purposes of statistical analysis, pooled SPCN waves were formed in order to improve the signalto-noise ratio of the data. The pooled SPCN waves were derived from electrode pairs PO7/PO8, O1/O2, and P7/P8. These electrode locations were chosen because they correspond to the location of the maxima of the SPCN topography. The SPCN amplitude for each condition was quantified as the mean voltage of the pooled SPCN wave in the 400 ms to 1000 ms latency range. Results Mean percent correct and mean SPCN amplitude measures for each condition were submitted to separate repeated measures analyses of variance (ANOVAs). The Geisser-Greenhouse correction procedure was used for repeated measures involving more than one degree of freedom. Mean accuracy rates for the word, nonword, and scrambled conditions were respectively 94%, 92%, and 88%. The ANOVA of accuracy rates revealed that the lexical status of the target letters has a significant effect on performance, F(2,72) 5 41.1, po.001, e 5 0.94, Z2p ¼ 0:53. Subsequent paired comparisons revealed that response accuracy was higher in the word condition than either the nonword condition, F(1,36) 5 11.8, po.002, Z2p ¼ 0:25, or the scrambled condition, F(1,36) 5 64.7, po.001, Z2p ¼ 0:64. Furthermore, response accuracy was higher in the nonword condition than in scrambled condition, F(1,36) 5 35.9, po.001, Z2p ¼ 0:50.
433 Figure 2 shows grand average waveforms for stimuli encoded from the left and right visual fields at representative lateral occipital electrode sites. The word condition is shown in Panel A, the nonword condition in Panel B, and the scrambled condition in Panel C. The SPCN can be seen as a relative negative shift in the ERP wave at electrode sites contralateral to the visual field of the target. As expected, the SPCN was observed in all three conditions for stimuli encoded from both visual fields. Figure 3, Panel A, shows the scalp distributions of the SPCN for each type of letter string. These scalp distributions are based on the SPCN subtraction waves at each lateralized electrode pair (zero voltage assumed at midline electrodes), and they exhibit the typical SPCN topography with a voltage peak in the vicinity of electrodes PO7/PO8 (e.g., Brisson & Jolicœur, 2007; Jolicœur et al., 2008; McCollough et al., 2007; Perron et al., 2009; Predovan et al., 2009). The distributions were quite similar in general topography, suggesting that the same component was elicited in the three string type conditions, but with different amplitudes. Figure 3, Panel B, shows the SPCN subtraction waves (contralateralFipsilateral) pooled over electrode pairs O1/O2, PO7/ PO8, and P7/P8, for each condition. The lexical status of the target letters has a significant effect on the amplitude of the SPCN, F(2,72) 5 14.2, po.001, e 5 0.98, Z2p ¼ 0:28. The amplitude of the pooled SPCN wave was highest for the scrambled condition, intermediate for the nonword condition, and smallest for the word condition. Replicating the principal finding of Predovan et al. (2009), the SPCN observed in the word condition was significantly smaller than that observed in the nonword condition, F(1,36) 5 7.0, po.02, Z2p ¼ 0:16. Of more relevance to the present study, paired comparisons also revealed that the scrambled condition elicited a significantly larger SPCN than either the word condition, F(1,36) 5 24.8, po.001, Z2p ¼ 0:41, or the nonword condition, F(1,36) 5 8.3, po.01, Z2p ¼ 0:19. Thus, the spatial configuration of the target letters affected SPCN amplitude even when the letters were presented in the same spatial positions (on average) in all conditions.
Discussion The present results, combined with those of Predovan et al. (2009), show that the degree of involvement of VSTM for visually presented letters depends both on the lexical and spatial context in which the letters are encoded. Letters that form a word lead to the smallest SPCN amplitude, which we interpret as a reflection of a smaller load in VSTM. A skilled reader, as most of our subjects undoubtedly were, can encode words very quickly, and there is good evidence that letters in the context of a word can be perceived more accurately than letters in a nonword (cf. word superiority effect; Reicher, 1969). We hypothesize that subjects were able to rely on abstract representations of words to reduce the need for a literal visual representation of letters, as would be required in the nonword and scrambled conditions. Abstract representations cannot be excluded for the latter conditions, but they would presumably be limited to shape-independent representations of letter identity. In contrast, a higher-order representation must be available for words, and this is likely to have helped. Interestingly, there is some evidence that even poor readers can show a significant word superiority effect in the perception of letters (Grainger, Bouttevin, Truc, Bastien, & Ziegler, 2003).
434
D. Prime et al.
A
B
C
Figure 2. Grand averaged waveforms for representative posterior electrodes. Separate waveforms are shown for stimuli encoded from the left and right visual fields. (A) Word condition. (B) Nonword condition. (C) Scrambled condition.
Most interestingly, however, was the fact that we again found a significantly lower VSTM load for nonwords presented in a typical horizontal array (nonword condition) compared with the load observed for the same letters presented in a spatially scrambled set of locations (scrambled condition). Because we varied the location of the target letters in all three conditions and equated the mean frequency of use of each position across con-
ditions, this difference in SPCN amplitude cannot be due to the effects of visual field location on SPCN amplitude found by Perron et al. (2009). This result, anticipated by Predovan et al. (2009), and confirmed here, suggests that, even for nonwords, there is a significant encoding advantage for letters presented in a word-like structure, over letters presented in spatially scrambled positions. One possibility is that there are specialized encoding
A
B
Figure 3. (A) Scalp distributions of the SPCN for each condition (mean activity from 400–1000 ms post memory stimulus onset). (B) Contralateral minus ipsilateral waveforms showing the SPCN for the pooled response at three posterior electrode sites for each condition.
Lexical status and VSTM
435
routines for word-like letter strings, and that these routines can be deployed on horizontal letter strings and confer an encoding advantage even for nonword letter strings. This would be consistent with a set of sub-lexical letter combinations (Grainger et al., 2003), which may be accessed more efficiently when letters are presented in a horizontal array than when presented in a scrambled array. Results from Peressotti and Grainger (1995) and Humphreys, Evett, and Quinlan (1990) suggest that readers encode multi-letter units (even when they occur in nonwords) that can prime subsequent processing of letter strings (see also Mayall & Humphreys, 1996; Mayall, Humphreys, & Olson, 1997). The present results suggest that such higher-order units can support VSTM by reducing the need to encode visual representations at the individual letter level. However, the efficient encoding of such multi-letter units may not be possible when the letters are presented in a spatially scrambled display. Alternatively, the encoding advantage observed for the word and nonword conditions over the scrambled condition may have arisen because the horizontal layout of the letters in the word and nonword conditions provided structure and reduced ambiguity. The disordered layout of the scrambled condition may have increased the attentional and working memory load required to encode the target letters, and this may have given rise to an increase in SPCN amplitude. According to this account, a reduction in SPCN amplitude relative to the scrambled condition should be observed for any orderly arrangement of letters (e.g., vertical). Further research will be required to test these possible accounts. The present results, for letters, contrast with those of McCollough et al. (2007), who found that the spatial extent of arrays of simple colored stimuli did not influence the amplitude
of the SPCN. The present results, therefore, suggest that the spatial layout of letters may be more critical than the spatial layout of simple colored stimuli. One possible explanation for this difference is that letter detectors tuned for horizontally aligned sets of letters may allow for special encoding algorithms that reduce the information load in VSTM for letter strings by facilitating a grouping or chunking of the letters (Grainger, 2008; Grainger & Jacobs, 1996). The present results, combined with those of McCollough et al. (2007), indicate the existence of encoding and VSTM maintenance mechanisms specific to letter and word processing. Thus, the SPCN provides a useful index of VSTM processes involved in letter processing and reading. One potential application of the present methodology is in investigating the processing deficits underlying dyslexia. Some researchers have proposed that dyslexia arises from inefficient processing at either the grapheme-to-phoneme conversion stage or the phonological short-term memory buffer (or a mixture of both) as primary determinants of impaired nonword reading (e.g., Goswami, Thomson, Richardson, Stainthorp, Hughes, et al., 2000; Ramus, 2003). Others, however, have emphasized problems in the segmentation of letter strings due to sluggish control of visuo-spatial attention mechanisms (e.g., Hari & Renwall, 2001; see also Facoetti, Trussardi, Ruffino, Lorusso, Catteneo, et al., 2010), or problems in the rapid transfer of ordered representations in VSTM (Valdois et al., 2003). The present differences in VSTM load for words and nonwords revealed by substantial modulations of the amplitude of the SPCN suggests that the SPCN may be a powerful tool to study the mechanisms that mediate efficient reading and deficits in these mechanisms that contribute to dyslexia.
REFERENCES Blais, C., Fiset, D., Arguin, M., Jolicœur, P., Bub, D., & Gosselin, F. (2009). Reading between eye saccades. PLoS ONE, 4, e6648. Bosse, M.-L., Tainturier, M. J., & Valdois, S. (2007). Developmental dyslexia: The visual attention span deficit hypothesis. Cognition, 104, 198–230. Brisson, B., & Jolicœur, P. (2007). A psychological refractory period in access to visual short-term memory and the deployment of visualspatial attention: Multitasking processing deficits revealed by eventrelated potentials. Psychophysiology, 44, 323–333. Coltheart, M. (1980). Iconic memory and visible persistence. Perception & Psychophysics, 27, 183–228. Dell’Acqua, R., Sessa, P., Toffanin, P., Luria, R., & Jolicœur, P. (2010). Orienting attention to objects in visual short-term memory. Neuropsychologia, 48, 419–428. Facoetti, A., Trussardi, A. N., Ruffino, M., Lorusso, M. L., Cattaneo, C., Galli, R., Molteni, M., & Zorzi, M. (2010). Multisensory spatial attention deficits are predictive of phonological decoding skills in developmental dyslexia. Journal of Cognitive Neuroscience, 22, 1011–1025. Goswami, U., Thomson, J., Richardson, U., Stainthorp, R., Hughes, D., Rosen, S., & Scott, S. K. (2000). Amplitude envelope onsets and developmental dyslexia: A new hypothesis. Proceedings of the National Academy of Sciences, USA, 99, 10911–10916. Goulandris, N. K., & Snowling, M. (1991). Visual memory deficits: A plausible cause of developmental dyslexia? Evidence from a single case study. Cognitive Neuropsychology, 8, 127–154. Grainger, J. (2008). Cracking the orthographic code: An introduction. Language and Cognitive Processes, 23, 1–35. Grainger, J., Bouttevin, S., Truc, C., Bastien, M., & Ziegler, J. (2003). Word superiority, pseudoword superiority, and learning to read: A comparison of dyslexic and normal readers. Brain and Language, 87, 432–440. Grainger, J., & Jacobs, A. M. (1996). Orthographic processing in visual word recognition: A multiple read-out model. Psychological Review, 103, 518–565.
Hammond, E. J., & Green, D. W. (1982). Detecting targets in letter and non-letter arrays. Canadian Journal of Psychology, 36, 67–82. Hari, R., & Renwall, H. (2001). Impaired processing of rapid stimulus sequences in dyslexia. Trends in Cognitive Science, 5, 525–532. Hillyard, S. A., & Galambos, R. (1970). Eye movement artifact in the CNV. Electroencephalography and Clinical Neurophysiology, 28, 173– 182. Humphreys, G. W., Evett, L. J., & Quinlan, P. T. (1990). Orthographic processing in visual word identification. Cognitive Psychology, 22, 517–560. Jolicœur, P., Brisson, B., & Robitaille, N. (2008). Dissociation of the N2pc and sustained posterior contralateral negativity in a choice response task. Brain Research, 1215, 160–172. Jolicœur, P., & Dell’Acqua, R. (1998). The demonstration of short-term consolidation. Cognitive Psychology, 36, 138–202. Jolicœur, P., Sessa, P., Dell’Acqua, R., & Robitaille, N. (2006a). On the control of visual spatial attention: Evidence from human electrophysiology. Psychological Research, 70, 414–424. Jolicœur, P., Sessa, P., Dell’Acqua, R., & Robitaille, N. (2006b). Attentional control and capture in the attentional blink paradigm: Evidence from human electrophysiology. European Journal of Cognitive Psychology, 18, 560–578. Klaver, P., Talsma, D., Wijers, A. A., Heinze, H.-J., & Mulder, G. (1999). An event-related brain potential correlate of visual short-term memory. NeuroReport, 10, 2001–2005. Lins, O. G., Picton, T. W., Berg, P., & Scherg, M. (1993). Ocular artifacts in EEG and event-related potentials I: Scalp topography. Brain Topography, 6, 51–63. Luck, S. J. (2005). An introduction to the event-related potential technique. Cambridge, MA: MIT Press. Luria, R., Sessa, P., Gotler, A., Jolicœur, P., & Dell’Acqua, R. (2010). Visual short-term memory capacity for simple and complex objects. Journal of Cognitive Neuroscience, 22, 496–512.
436 Mayall, K., & Humphreys, G. W. (1996). Case mixing and the tasksensitive disruption of lexical processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 278–294. Mayall, K., Humphreys, G. W., & Olson, A. (1997). Disruption to word or letter processing? The origins of the case-mixing effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23, 1275–1286. McCollough, A. W., Machizawa, M. G., & Vogel, E. K. (2007). Electrophysiological measures of maintaining representations in visual working memory. Cortex, 43, 77–94. Peressotti, F., & Grainger, J. (1995). Letter-position coding in random consonant arrays. Perception & Psychophysics, 57, 875–890. Perron, R., Lefebvre, C., Robitaille, N., Brisson, B., Gosselin, F., Arguin, M., & Jolicœur, P. (2009). Attentional and anatomical considerations for the representation of simple stimuli in visual short-term memory: Evidence from human electrophysiology. Psychological Research, 73, 222–232. Predovan, D., Prime, D. J., Arguin, M., Gosselin, F., Dell’Acqua, R., & Jolicœur, P. (2009). On the representation of words and nonwords in visual short-term memory: Evidence from human electrophysiology. Psychophysiology, 46, 191–199. Prime, D. J., & Jolicœur, P. (in press). Mental rotation requires visual short-term memory: Evidence from human electric cortical activity. Journal of Cognitive Neuroscience. Ramus, F. (2003). Developmental dyslexia: Specific phonological deficit of general sensorimotor disfunction? Current Opinion in Neurobiology, 13, 212–218.
D. Prime et al. Reicher, G. M. (1969). Perceptual recognition as a function of meaningfulness of stimulus material. Journal of Experimental Psychology, 81, 275–280. Robitaille, N., Grimault, S., & Jolicœur, P. (2009). Bilateral parietal and contralateral responses during the maintenance of unilaterally-encoded objects in visual short-term memory: Evidence from magnetoencephalography. Psychophysiology, 46, 1090–1099. Tydgat, I., & Grainger, J. (2009). Serial position effects in the identification of letters, digits, and symbols. Journal of Experimental Psychology: Human Perception and Performance, 35, 480–498. Valdois, S., Bosse, M.-L., Ans, B., Carbonnel, S., Zorman, M., David, D., & Pellat, J. (2003). Phonological and visual processing deficits can dissociate in developmental dyslexia: Evidence from two case studies. Reading and Writing, 16, 541–572. Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature, 428, 748–751. Whitney, C. (2001). How the brain encodes the order of letters in a printed word: The SERIOL model and selective literature review. Psychonomic Bulletin & Review, 8, 221–243.
(Received February 2, 2010; Accepted April 25, 2010)
