Neural Correlates of Thinking (On Thinking)

On Thinking Editors Ernst Pöppel, Germany Albrecht von Müller, Germany Eduard Kraft • Balázs Gulyás • Ernst Pöpp...

Author: Eduard Kraft | Balázs Gulyás | Ernst Pöppel

218 downloads 1450 Views 4MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

On Thinking

Editors Ernst Pöppel, Germany Albrecht von Müller, Germany

Eduard Kraft

•

Balázs Gulyás

•

Ernst Pöppel

Editors

Neural Correlates of Thinking

ABC

Dr. Eduard Kraft Parmenides Foundation Parmenides Center for the Study of Thinking Kardinal Faulhaber Str. 14a 80333 Munich, Germany and Ludwig-Maximilian University Department of Physical Medicine and Rehabilitation Neuropsychology Unit Marchioninistr.15 81337 Munich, Germany [email protected]

Prof. Dr. Ernst Pöppel Parmenides Foundation Parmenides Center for the Study of Thinking Kardinal Faulhaber Str. 14a 80333 Munich, Germany and Human Science Center Ludwig Maximilian University Goethestr.31 80336 Munich [email protected]

Dr. Balázs Gulyás Psychiatry Section Department of Clinical Neuroscience Karolinska Institute S-171 76 Stockholm Sweden [email protected]

ISBN: 978-3-540-68042-0

e-ISBN: 978-3-540-68044-4

Library of Congress Control Number: 2008927065 c Springer-Verlag Berlin Heidelberg 2009

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, speciﬁcally the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microﬁlm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a speciﬁc statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover design: Boekhorst Design BV, The Netherlands Printed on acid-free paper 9 8 7 6 5 4 3 2 1 springer.com

Series Preface

What is Thinking?—Trying to Deﬁne an Equally Fascinating and Elusive Phenomenon Human thinking is probably the most complex phenomenon that evolution has come up with until now. There exists a broad spectrum of deﬁnitions, from subsuming almost all processes of cognition to limiting it to language-based, sometimes even only to formalizable reasoning processes. We work with a “medium-sized” deﬁnition according to which thinking encompasses all operations by which cognitive agents link mental content in order to gain new insights or perspectives. Mental content is, thus, a prerequisite for and the substrate on which thinking operations are executed. The largely unconscious acts of perceptual object stabilization, categorization and emotional evaluation—and retrieving all the aforementioned from memory inscriptions—are the processes by which mental content is generated, and are, therefore, seen as prerequisites for thinking operations. In terms of a differentia speciﬁca, the notion of “thinking” is seen as narrower than the notion of “cognition” and as wider than the notion of “reasoning.” Thinking is, thus, seen as a subset of cognition processes; and reasoning processes are seen as a subset of thinking. Besides reasoning, the notion of thinking includes also nonexplicit, intuitive and associative processes of linking mental content. According to this deﬁnition, thinking is not dependent on language, for example, also many animals and certainly all mammals show early forms of thinking. The emergence of more complex syntactical structures, however, led to a self-accelerating expansion—or not to say “explosion”—of thinking skills. Syntax boosts the possibility to deal with complex relations and enables the understanding of conceptual hierarchies as well as of self-referential structures. The latter may be directly related to the development of an autobiographic self. The purpose of thinking can be deﬁned in a twofold way: from a biological point of view it can be characterized as the most advanced form of ensuring homeostasis; from a philosophical point of view it can be characterized as the crucial means by which the richness of reality unfolds for us. These different descriptions do not

v

vi

Series Preface

constitute a contradiction; they rather articulate the complementary perspectives of asking for the function and of asking for the sense of thinking. Logic, which has long been seen as the core feature of thinking, is an important, but nevertheless rather small part of what thinking really is. It refers only to the coherence of explicitly reviewable linkages made by thinking operations. In contrast, metaphors and analogies constitute a highly content related way of connecting mental content that is extremely important for thinking, though they often escape a rigorous logical analysis.

The Relevance of the Phenomenon of Thinking Complex thinking skills are probably the most characteristic feature of humans, and the following four appear to be of particular importance: 1. Thinking is the crucial mechanism through which the richness, interrelatedness and coherence of reality unfold for us. Thinking can be seen as the “crown” of evolutionary sophistication and it is crucial for answering the question: “What makes us human?” 2. Thinking and what we refer to as reality shape themselves mutually. Major breakthroughs in many of today’s most fascinating scientiﬁc issues (from trying to grasp how consciousness works to bridging the conceptual gap between quantum physics and gravity) require a better understanding of how thinking shapes reality and how reality shapes thinking. 3. Ever-increasing complexity and a self-accelerating pace of change characterize our modern world. The highly complex, interrelated dynamics of technological, economical, political and sociocultural developments constitute new challenges that require further advancements in our thinking skills in order to cope with them. 4. In an increasingly knowledge based economy, thinking as the process by which new knowledge is generated will become the main value-generation process. Being aware of the importance of thinking, it is astonishing how little we understand about how complex thinking actually works and how it is implemented in the human brain. The task of the Parmenides Foundation is to enable advanced, interdisciplinary research on this topic.

The Parmenides Foundation and its Research Agenda The overall purpose of the Parmenides Foundation is to advance our understanding of one of the most fascinating, characteristic and relevant faculties of human beings: complex thinking. The Foundation was established in 2000 as a charitable institution for basic research.

Series Preface

vii

The main activity of the Parmenides Foundation is to run the Parmenides Center for the Study of Thinking, which was established in co-operation with the Ludwig Maximilian University of Munich. The Parmenides Center is organized similarly to a Max Planck Institute. It tries to provide optimal conditions for basic research and interdisciplinary co-operation, with minimized bureaucratic distractions and optional teaching activities at the university. The work of the Parmenides Foundation is based on an interdisciplinary core team of approximately 15 scientists at present, a guest fellow program and an international faculty of about 30 members. The faculty unites outstanding experts from the neurosciences, neuroinformatics, philosophy, cognitive psychology, linguistics and evolutionary biology. At present we focus on the following areas of basic and applied research on thinking. The main topics of basic research are: • To develop a conceptual framework (or taxonomy) for the understanding of thinking • To identify and analyze the neural and neurobiological correlates of thinking • To understand the complementary features of human cognition such as syntactic language and (self-)consciousness • To become able to reconstruct key aspects of complex thinking by modeling • To learn more about the ontogenesis of complex thinking in childhood • To learn more about the phylogenesis of complex thinking during evolution • To study the structural constraints of thinking and their relation to problems in the categorical foundations of science The main topics of applied research are: • To develop new approaches and methodologies for supporting the acquisition of thinking skills in early and later childhood • To develop new approaches and methodologies for supporting the human brain in dealing with tasks of high complexity • To develop new approaches and methodologies for analyzing and improving the knowledge metabolism of institutions • To develop new approaches and methodologies for supporting strategy development and decision making in a brain-adequate way • To develop new approaches and methodologies for the medical reconstruction or restitution of advanced thinking skills The book series On Thinking was established to present new insights and ﬁndings, as well as ongoing discussions to a wider readership. The volumes are edited by authors from the Parmenides Foundation and faculty as well as by guest authors and present the progress in this important ﬁeld for society. Munich March 2008

Ernst P¨oppel Albrecht von M¨uller

Preface

The idea of the present book emerged on the island of Elba in the summer of 2006 during an enjoyable and very fruitful workshop on thinking with the participation of most of the contributors of the present volume. The main intention behind the book is to address thinking by surveying the contribution of various functional neuroimaging methods to our understanding of the neural underpinnings of thinking. The major focus is on the methods applicable to the neurobiological study of human thinking, since much of what we consider complex thinking has to be considered as a part of the distinctive features of human nature. Despite the fact that we are far from a full understanding of the modularity of the human brain, the use of functional imaging techniques is obviously based on the premise that brain functions are modular. We are grateful to the distinguished authors, coming from different backgrounds, for their commitment to this project, which represents a true interdisciplinary approach, as is mandatory for this fascinating and challenging topic. We are also proud to have been able to recruit an outstanding worldwide team of contributors. We also wish to thank Anette Lindqvist and Dieter Czechlik from Springer Science+Business Media for their enthusiasm and constant support. Without their optimism and tireless efforts this volume would not have been possible. Munich, Germany Stockholm, Sweden Munich, Germany June 2008

Eduard Kraft Bal´azs Guly´as Ernst P¨oppel

ix

Contents

Introduction Neural Correlates of Thinking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eduard Kraft, Bal´azs Guly´as, and Ernst P¨oppel

3

Part I The Methods: State of the Art and Prospects Functional MRI Limitations and Aspirations . . . . . . . . . . . . . . . . . . . . . . . . 15 Peter A. Bandettini Studying Cognition with Positron Emission Tomography . . . . . . . . . . . . . . 39 Alain Dagher Investigating the Neural Correlates of Percepts Using Magnetoencephalography and Magnetic Source Imaging . . . . . . . . . 51 Thomas Hartmann, Nathan Weisz, Winfried Schlee, and Thomas Elbert EEG and Thinking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 ¨ Michael Ollinger Near-Infrared Spectroscopy for Studying Higher Cognition . . . . . . . . . . . . 83 Yoko Hoshi Integration of EEG and fMRI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Christoph Mulert and Ulrich Hegerl

Part II Neural Correlates of Key Components of Higher Cognition The Neural Implementation of Working Memory . . . . . . . . . . . . . . . . . . . . . 109 Oliver Gruber

xi

xii

Contents

Current Perspectives on Imaging Language . . . . . . . . . . . . . . . . . . . . . . . . . 123 Joseph T. Devlin Functional Neuroimaging and the Logic of Conscious and Unconscious Mental Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Bal´azs Guly´as Knowledge Systems of the Brain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 Josef Ilmberger Neural Representation of Time and Timing Processes . . . . . . . . . . . . . . . . . 187 Elsbieta Szelag, Joanna Dreszer, Monika Lewandowska, and Aneta Szymaszek

Part III From Conceptual Considerations to Neural Correlates Fractionating the System of Deductive Reasoning . . . . . . . . . . . . . . . . . . . . 203 Vinod Goel Human Thought and the Lateral Prefrontal Cortex . . . . . . . . . . . . . . . . . . . 219 Kalina Christoff Neural Correlates of Insight Phenomena . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 Jing Luo, Gunther Knoblich, and Chongde Lin Brain-Based Mechanisms Underlying Causal Reasoning . . . . . . . . . . . . . . . 269 Jonathan Fugelsang and Kevin N. Dunbar Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

Contributors

Peter A. Bandettini Laboratory of Brain and Cognition, National Institute of Mental Health, Section on Functional Imaging Methods Building 10, Room 1D80B, 10 Center Dr. MSC 1148, Bethesda, MD 20892-1148, USA, [email protected] Kalina Christoff Department of Psychology, University of British Columbia, Cognitive Neuroscience of Thought Lab, 2136 West Mall, Vancouver, BC, Canada, V6T 1Z4, [email protected] Alain Dagher McConnell Brain Imaging Center, McGill University, 3801 University St, Montreal, QC, Canada H3A 2B4, [email protected] Joseph T. Devlin Department of Psychology, University College London, Gower Street, London, WC1E 6BT, UK, [email protected] Joanna Dreszer Laboratory of Neuropsychology, Nencki Institute of Experimental Biology, 3 Pasteur Street, 02-093 Warsaw, Poland, [email protected] Kevin N. Dunbar Department of Psychology, University of Toronto Scarborough, 1265 Military Trail, Toronto, ON, Canada M1C 1A4, [email protected] Thomas Elbert Department of Psychology, University of Konstanz, Universit¨atsstraße 10, 78464 Konstanz, Germany, [email protected]

xiii

xiv

Contributors

Jonathan Fugelsang Department of Psychology, University of Waterloo, 200 University Avenue West, Waterloo, ON, Canada N2L 3G1, [email protected] Vinod Goel Department of Psychology, York University, 4700 Keele St., Toronto, ON, Canada M3J 1P3, [email protected] Oliver Gruber Department of Psychiatry, University of G¨ottingen, von-Siebold-Str. 5, 37075 G¨ottingen, Germany, [email protected] Bal´azs Guly´as Karolinska Institutet, Department of Clinical Neuroscience, Psychiatry Section, 171 76 Stockholm, Sweden, [email protected] Thomas Hartmann Department of Psychology, University of Konstanz, Universit¨atsstraße 10, 78464 Konstanz, Germany, [email protected] Ulrich Hegerl Department of Psychiatry, University of Leipzig, Semmelweisstr. 10, 04103 Leipzig, Germany, [email protected] Yoko Hoshi Integrated Neuroscience Research Team, Tokyo Institute of Psychiatry, 2-1-7 Kamikitazawa, Setagaya-ku, Tokyo 156-8585, Japan, [email protected] Josef Ilmberger Department of Physical Medicine and Rehabilitation, Ludwig-Maximilian University, Neuropsychology Unit, Marchioninistr.15, 81337 Munich, Germany, [email protected] Gunther Knoblich Department of Psychology, School of Psychology, Birmingham University, Edgbaston, Birmingham B15 2TT, UK, [email protected] Eduard Kraft Parmenides Foundation, Parmenides Center for the Study of Thinking, Kardinal Faulhaber Str. 14a, 80333 Munich, Germany, [email protected] Monika Lewandowska Laboratory of Neuropsychology, Nencki Institute of Experimental Biology, 3 Pasteur Street, 02-093 Warsaw, Poland, [email protected] Chongde Lin Institute of Developmental Psychology, School of Psychology, Beijing Normal University, Beijing 100875, P. R. China [email protected]

Contributors

xv

Jing Luo Chinese Academy of Sciences, Department of Psychology, Da-tun Road 10, Chao-yang District, Beijing 100101, Peoples Republic of China, [email protected] Christoph Mulert Functional Brain Imaging Branch, Department of Psychiatry, Ludwig-Maximilian University, Nussbaumstr. 7, 80336 Munich, Germany, [email protected] Albrecht von M¨uller Parmenides Foundation, Parmenides Center for the Study of Thinking, Kardinal Faulhaber Str. 14a, 80333 Munich, Germany, [email protected] ¨ Michael Ollinger Parmenides Foundation, Parmenides Center for the Study of Thinking, Kardinal Faulhaber Str. 14a, 80333 M¨unchen, Germany, [email protected] Ernst P¨oppel Parmenides Foundation, Parmenides Center for the Study of Thinking, Kardinal Faulhaber Str. 14a, 80333 Munich, Germany, [email protected] Winfried Schlee Department of Psychology, University of Konstanz, Universit¨atsstraße 10, 78464 Konstanz, Germany, [email protected] Elsbieta Szelag Laboratory of Neuropsychology, Nencki Institute of Experimental Biology, 3 Pasteur Street, 02-093 Warsaw, Poland, [email protected] Aneta Szymaszek Laboratory of Neuropsychology, Nencki Institute of Experimental Biology, 3 Pasteur Street, 02-093 Warsaw, Poland, [email protected] Nathan Weisz Department of Psychology, University of Konstanz, Universit¨atsstraße 10, 78464 Konstanz, Germany, [email protected]

Neural Correlates of Thinking Eduard Kraft(¬), Bal´azs Guly´as, and Ernst P¨oppel

1 Introduction In April 1918, Korbinian Brodmann moved to Munich to join the Deutsche Forschungsanstalt f¨ur Psychiatrie, the ﬁrst interdisciplinary brain research institute in the world.1 Brodmann published what is now regarded as one of the major “classics” of neurological literature, a monograph entitled Vergleichende Lokalisationslehre der Grosshirnrinde. (An English translation by Laurence Garey was published in 1994: Brodmann’s Localisation in the Cerebral Cortex.) Although the cortical map Brodmann described was purely based on histomorphological criteria, it was an important landmark for future work on functional localization. From our present perspective Brodmann’s work remains a seminal landmark for localizing activity in neuroimaging research, since most functional imaging studies still refer to Brodmann’s areas when they describe the localization of peak activities and the extent of the activated ﬁelds in the human brain. In the light of all this, a historical reference to Brodmann seems to be an appropriate starting point when launching the ﬁrst book of a series promoted by the Parmenides Foundation, in particular when the book addresses the question of how modern imaging techniques can contribute to our understanding of complex thinking.

E. Kraft Parmenides Foundation, Parmenides Center for the Study of Thinking, Kardinal Faulhaber Str. 14a, 80333 Munich, Germany [email protected] 1

The founding of this institute was almost a logical consequence of a scientiﬁc tradition, which has been known as the “Munich School of Neuroanatomy and Neuroscience”, and was started in the late nineteenth century by Bernhard von Gudden (Danek 2006). An upcoming volume of the Parmenides book series will be dedicated to this historical and fascinating period of brain research.

E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

3

4

E. Kraft et al.

2 Localization and Functional Integration Over more than a century the two seemingly opposing concepts of localized and holistic organization of cerebral functions have alternatingly dominated the scientiﬁc debate on how particular cognitive functions are represented in the brain (Kert´esz 1994). Despite the longstanding historical dimension and the ongoing controversies concerning the dispute, these two hypotheses might be reconciled (P¨oppel 1989; Umilta 2003), since at present a complementary synthesis of these two principles may be described as functional specialization and functional integration (Friston 2002). In a taxonomy of function a distinction has been made between content functions which are “localized” and logistical functions which take care of integration in the time domain on at least two different levels (P¨oppel 1989). The localization concept claims that well-circumscribed anatomical regions or neuronal populations in the brain are responsible for well-determined brain functions. This assumption dates back to Franz Joseph Gall’s phrenology in the early nineteenth century, but the ﬁrst rigorous scientiﬁc investigations were performed by Jean Pierre Flourens (1825). The hypothesis received more scientiﬁc support from the famous reports presented by Paul Broca (1861,1863,1865) and Carl Wernicke (1874) on patients with speciﬁc language deﬁcits and from the electrical stimulation experiments pioneered by Fritsch and Hitzig (1870). World War I provided investigators with ample patient data soldiers with circumscribed brain regions due to gunshot wounds, resulting in functional brain deﬁcits. Gordon Holmes (1945) Walther Poppelreuter (1917, 1918) and later on, during World War II, Alexander Luria (1963, 1970) performed pioneering work by establishing meticulous correlations between the anatomical localization of a cortical deﬁcit and the resulting functional deﬁcits, supporting the theory of functional localization of brain functions. With the advent of highly reﬁned neurosurgery and anaesthesia techniques, Otfried F¨orster and Wilder Penﬁeld pioneered the direct brain stimulation method which paved the way towards the direct or invasive cortical localization approach (Jasper and Penﬁeld 1954). Functional integration refers to the interaction between neuronal populations, but in contrast to its historical roots in Lashley’s concept of mass action (Lashley 1943) and the equipotentiality hypothesis ﬁrst advocated by Flourens, it is nowadays based upon a more advanced computational background (Friston 2002; McIntosh 2004). Functional integration can be addressed by using electrophysiological recordings by electroencephalography (EEG) or magnetoencephalography (MEG) studying synchronization (Singer 1993, see also the chapters in Part I on EEG and MEG in this book). Using positron emission tomography (PET) or functional MRI (fMRI), one can compute and quantify data for functional interactions of cortical areas by modelling functional and effective connectivity. Several conceptual and methodological approaches have been proposed for these computations (structural equation modelling, dynamic causal modelling, principal component analysis, independent

Neural Correlates of Thinking

5

component analysis) (Friston 2002). However, the very precise nature of the synchronization, coordination and interaction of different brain regions is yet to be determined and there is still a very poor understanding of the computational processes underlying those interactions. An important conceptual tool of neuropsychology which was adopted in neuroimaging is the exploration of double dissociation of functions in patients with focal lesions. This has been a central concept of neuropsychology ever since such a double dissociation was found in language disorders and described by Paul Broca and Carl Wernicke. Double dissociation can be described by the following situation. In a given patient a lesion in region X results in a deﬁcit of function A but not function B. Then, another patient has a lesion in region Y, resulting in a deﬁcit of function B but not function A. Such observations lend further support to the notion of modular (localized) cortical functions. However, as recent neuroimaging techniques have revealed, the loss of a single area does not necessarily mean that this very area is solely responsible for a given brain function. It can in fact be a crucial part of a cortical network, a “hub” in the network, the loss of which may – or may not – result in deﬁciency of network activities. The advances in neuroimaging in the last three decades represent a major methodological development in the neurosciences and have provided us with unprecedented possibilities to study brain activity in vivo, in real time and in a noninvasive manner. In particular, they have enabled researchers to test for double dissociations in healthy controls. Neuroimaging has become a leading research method in neuroscience and has great potential for the identiﬁcation of the functions that are actually computed (Goel 2004). It may serve that purpose by means of the empirical and experimental evaluation of cognitive theories and claims. However, the rise of these powerful imaging technologies has sometimes led to unlimited enthusiasm about the possibilities to explore cognitive and mental activities in the brain. That has prompted critical and thoughtful comments concerning the underlying assumptions of the experimental procedures and other methodological pitfalls of these techniques. The editors are well aware of many conceptual and methodological shortcomings of current experimental paradigms, procedures and interpretations. Some of these issues have been addressed elsewhere (Kosslyn 1999; Uttal 2001; Logothetis 2008), but will also be addressed in this book on several occasions (e.g. chapters in Part I). The present volume is an attempt to summarize our state-of-the-art knowledge about the possibilities and limitations of studying thinking using functional neuroimaging techniques. This is not an easy undertaking, since there is a serious lack of understanding of the basic principles of computational information processing at different levels. At present, current neuroimaging tools are not sensitive enough to measure the activity of medium-sized neuronal assemblies. Notwithstanding that, appropriate techniques may be available in the future (Logothetis 2008).

6

E. Kraft et al.

3 Deﬁning Thinking Despite the difﬁcult and controversial topic of providing an accurate deﬁnition, thinking is a core cognitive capacity and has traditionally been conceptualized into reasoning, problem solving and decision making (Holyoak and Morrison 2005). These are closely interconnected ﬁelds, although historically they have represented distinct perspectives on thinking. Reasoning, which in a broad description is drawing inferences from given information, can be subdivided into many special instances, including relational reasoning, causal reasoning, conditional reasoning, analogical reasoning, and deductive and inductive reasoning. Problem solving has been deﬁned as a goal-driven process of overcoming obstacles that obstruct the path to a solution (Simon 1999). Thinking is a polymorphous term, as has been emphasized by Bennett and Hacker (2003), who argued that for this very reason the term may not be amenable to fruitful scientiﬁc investigation. However, it is owing to this polymorphous nature that it may be used as a relevant conceptual term referring to all facets of higher cognitive processing. Additionally, thinking would also incorporate into its traditional nomenclature terms such as “intuition”, “insight”, “spontaneous thought processes” and “free ﬂoating thoughts.” One could consider a group of thinking operations (decision making, reasoning, problem solving) as explicit domains and another group (intuition, insight, spontaneous thought) as implicit domains in a taxonomy of thinking processes. Two theories about reasoning have dominated the cognitive literature: mental model and mental logic (Braine and O’Brien 1998). Mental model is a semantic theory claiming that the central concept by which we perform reasoning operations relates to spatially organized mental models (Johnson-Laird 1983). Mental model would have predicted primarily right-hemisphere regions, especially parietal and occipital regions. In contrast mental logic claims that deductive reasoning is based on the application of formal deductive rules according to formal syntactic operations. Thus, one would expect that left-sided prefrontal and temporal regions would be implicated in formal, rule-based operations. Over the last few years alternative and more integrative concepts have been formulated by dual-mechanism theories. These theories are presented in different versions, for instance intuitive versus deliberate (Tversky and Kahneman 1986), associative versus rule-based, formal and heuristic processes (Newell and Simon 1972). These dual-mechanism concepts come closest to what one might consider as a general theory of thinking. Most of them would predict the presence of broadly distributed neural systems (Barbey and Barsalou 2006). However, despite all these approaches, a coherent theory of thinking is lacking, as is a proper taxonomy for all the different ﬂavours of its components. All in all, given the lack of data and knowledge about neuroscientiﬁc investigation into thinking on the one hand, and the missing coherent theory and taxonomy on the other hand, a book exclusively dedicated to the present state of the art of neuroimaging techniques for gaining insight into the process and organization of thinking seems warranted. This is also justiﬁed by the impression that central domains of thinking have neither participated in nor beneﬁted that much from the interaction of cognitive science

Neural Correlates of Thinking

7

and neuroscience which has generated the new ﬁeld of cognitive neuroscience (Goel 2004). Because the fruitful interaction of cognitive science and neuroscience has been fostered especially by the emergence of modern neuroimaging techniques, it appears necessary to reach for a more intensive interaction of these ﬁelds for the study of thinking.

4 Neural Correlates of Thinking Our motivation for compiling this book lies in our interest in thinking on a conceptual and neurophysiological level. Therefore, by editing Neural Correlates of Thinking, we would like to provide the reader with an overview of present efforts to map the neural mechanisms underlying thinking on a systems level. The underlying assumption is that neuroscience research using neuroimaging methods can contribute to the identiﬁcation and the veriﬁcation of cognitive theories and help to specify the computational architecture of the mind (Goel 2004). However, our aim is not to give a historic introduction to, nor to provide the reader with a comprehensive overview of this burgeoning ﬁeld. Instead, our intention with the present book is to offer a concise up-to-date survey of some key elements of the neurobiological foundations of thinking, and its analysis using functional neuroimaging techniques. The present volume is organized into three parts: (1) The aim of the ﬁrst part is to provide an up-to-date account of the present possibilities and limitations of the different imaging techniques to shed light on the neural correlates of thinking. (2) The objective of the second part is to zoom in on some key components of higher cognition and their neural underpinnings as well as to discuss relevant connections to thought processes. (3) The purpose of the third part is to elaborate on the interrelationship between conceptual approaches to thinking and the neural mechanism at a systems level. The ﬁrst part, a general methodological overview, comprising six chapters that describe the basic principles underlying the most common imaging techniques, so a clearer impression of the strengths and limitations of the individual methods can be obtained. The common denominator of these chapters is the “technique”. This part gives an overview of the recently available functional imaging techniques with the help of which thinking processes can be studied. In line with this, Peter Bandettini gives a concise account of the current state of the art and prospects of fMRI, with special regard to the use of the fMRI technique in cognitive sciences. Alain Dagher, using the unique capacity of PET to visualize CNS receptors with the help of dedicated radioligands and describing studies on addictive behaviour, opens a vista into the understanding of the interaction between impulsive behaviour and deliberate reasoning. Thomas Hartmann and colleagues discuss the usefulness of the MEG technique and its scope of application in

8

E. Kraft et al.

the cognitive sciences, starting from a simple perceptional model and reaching the assessment of synchronicity as an important concept of cerebral activity. Michael Oellinger summarizes the basic principles and applications of EEG, emphasizing the important role this method had before the emergence of newer methods such as PET or fMRI. His review indicates that at present EEG still remains a relevant tool for analysing brain function given its excellent temporal resolution. In addition, it allows the powerful analysis of synchronization effects around the cortex. This is an important tool for examining functional integration. Yoko Hoshi introduces into the basic theory underlying near-infrared spectroscopy (NIRS). This is a completely non-invasive new imaging tool, and does not require motion restriction. Since this technique can be applied in a daily-life environment, it is expected that NIRS will provide a new direction for cognitive neuroscience research. Christoph Mulert and Ulrich Hegerl tackle the promising approach of combining the high spatial resolution of fMRI with the high temporal resolution of electrophysiological methods such as EEG. They explain some of the promising results which this combination can generate. Since the present book is not intended to be a handbook on the use of functional neuroimaging techniques in cognitive neurosciences, the aforementioned chapters give the reader a vista into the respective ﬁelds, but they do not cover the entire role of functional neuroimaging in cognitive research. A body of important theoretical concepts and practical aspects of the ﬁeld such as attention, episodic and semantic memory are not speciﬁcally covered in our book. In this context we refer to excellent handbooks written for that purpose (e.g. Handbook of Functional Neuroimaging of Cognition by Cabeza and Kingstone 2006 or Human Brain Function by Frackowiak et al. 2004). The second part comprises ﬁve chapters concerning fundamental cognitive domains associated with reasoning and thinking. The ﬁrst chapter, by Oliver Gruber, is about one of the most inﬂuential, but also still evolving, concepts in cognitive neuroscience: working memory. Gruber elaborates on the evolution of this concept since the ﬁrst model proposed by Baddeley provided rich ground for further research. Working memory is an excellent example of the fruitful interaction between a cognitive model and its empirical elaboration based on a growing number of neuroimaging studies. Obviously one of the major cognitive domains of relevance to thinking is language. Joseph Devlin discusses the brain areas responsible for language comprehension in skilled reading, again demonstrating how cognitive theories and neurobiological investigations can be mutually informative and lead to novel explanations framed in terms of neural information processing. Bal´azs Guly´as reviews the use of PET in revealing the functional logic of human brain activation and gives some examples from the non-conscious–conscious cognitive domain. Josef Ilmberger deals with the difﬁcult topic of deﬁning terms concerning knowledge and examines the taxonomic approaches currently available. He discusses the difﬁculties of the variety of approaches used so far. For anybody using functional imaging methods it is an important exercise not to generate more confusion by introducing new terms into a ﬁeld already ill-deﬁned by the terminology of mental activities. He proposes a taxonomy of knowledge systems based on the distinction of iconic, lexical and symbolic representations and elaborates how these concepts may

Neural Correlates of Thinking

9

be organized into domains which are accessible to cognitive testing and how they are related to some of the taxonomies proposed over the last few years. Elsbieta Szelag and colleagues tackle a basic feature of cognitive processing by reporting on the current understanding of neural correlates of time processing and time perception in the brain. This might be the underlying platform of cognitive processing which has been neglected for many years, but considerable experimental evidence has been collected in recent years to outline a consistent taxonomy of time perception. The third part provides the reader with state-of-the-art examples on the experimental and conceptual approaches to thinking and reasoning. This part of the book comprises four chapters, each dealing with the evaluation of different cognitive theories using neuroimaging methods. To start with, Vinod Goel, in his review about the neural structures of deductive reasoning, summarizes a dual-mechanism concept which might more adequately reﬂect the cognitive architecture of deductive reasoning and would overcome some obvious ﬂaws of the two most prominent theories on reasoning by being more compatible with the empirical data collected in the last decade. He nicely demonstrates how using the double dissociation approach in behavioural studies on patients with focal lesions and in imaging studies with healthy controls, one can extract dissociated ﬁndings. By these means the empirical evidence for a fractionation of deductive reasoning into different systems on the basis of content sensitivity can be provided. Kalina Christoff integrates the mounting evidence for a hierarchical organization of the lateral part of the prefrontal cortex by reporting on a series of experiments nicely indicating an anterior to posterior aligned functional subdivision of the lateral prefrontal cortex, which is based on abstraction level. This has potential implications for the cognitive models of cognition and thought as well as for the neuropsychological investigations of executive functions. Jing Luo and colleagues present the challenges and difﬁculties of performing brain-imaging studies on insight problem solving. They suggest an array of criteria that would characterize the ideal experimental paradigm for studying the neural correlates of insight by functional imaging techniques. They continue to analyse recent experimental studies concerning these proposed criteria. Studies involving the reinterpretation of meaning in riddles and puzzles are used to clarify the role of the anterior cingulate cortex within insight problem solving. They further investigate experiments integrating meaning in the Remote Associates Test paradigm and evaluate the approach of perceptual reorganization by chunk decomposition. Jonathan Fugelsang and Kevin Dunbar discuss neuronal mechanisms underlying causal reasoning. They report on investigations probing the brain in order to uncover the mechanisms mediating people’s conception of causality. They also describe their recent study on the nature of how people interpret and reason about causality. We hope that this volume provides some insight into the state-of-the-art neurobiological theories and techniques related to research on thinking. Further volumes of the present book series will provide readers with further aspects of thinking research, including theoretical and practical ones. There is no question that, although much progress has been made in recent years, neuroscientiﬁc research on thinking is still in its infancy. Our ﬁnal goal is to gain better understanding of how people think and how to improve this crucial skill. The beneﬁts will be visible in many

10

E. Kraft et al.

different ﬁelds (education, economic and political decision making, social interaction). For that purpose, starting to explore the full potential of modern neuroimaging techniques seems to be a promising path.

References Barbey A, Barsalou LW (2006) Intelligence: models of reasoning. In: Squire L, Albright T, Bloom F, Gage F, Spitzer N (eds) New encyclopedia of neuroscience. Elsevier, Oxford Bennett MR, Hacker PMR (2003) Philosophical foundations of neuroscience. Blackwell, Oxford Braine MDS, O’Brien DP (eds) (1998) Mental logic. Erlbaum, Mahwah Broca P (1863) Localization of cerebral functions. Location of articulate language. Bulletin of the Society of Anthropology (Paris) 4:200–203 Broca P (1865) On the location of the faculty of articulate language in the left hemisphere of the brain. Bulletin of the Society of Anthropology 6:377–393 Cabeza R, Kingstone A (eds) (2006) Handbook of Functional Neuroimaging of Cognition. MIT Press, Cambridge Danek A (2006) Bernhard von Gudden, neuro-ophthalmology and the Munich School of Neuroanatomy and Psychiatry. Strabismus 4:211–216 Flourens JP (1825) Experiences sur le syst`eme nerveux. Paris Frackowiak RSJ, Ashburner JT, Penny WD, Zeki S (2004) Human Brain Function. Academic, New York Friston K (2002) Beyond phrenology: what can neuroimaging tell us about distributed circuitry? Annu Rev Neurosci 25:221–250 ¨ Fritsch E, Hitzig GT (1870) Uber die elektrische Erregbarkeit des Grosshirns. Archiv f¨ur Anatomie, Physiologie und wissenschaftliche Medicin. Arch. F. Anat., Physiol. und wissenschaftl. Mediz., Leipzig, 37:300–332 Goel V (2004) Can there be a cognitive neuroscience of central cognitive systems? In: Johnson D, Erneling C (eds) Mind as a scientiﬁc object: between brain and culture. Oxford University Press, Oxford Holmes G (1945) Ferrier lecture. The organisation of the visual cortex in man. Proc R Soc Ser B 132:348–361 Holyoak KJ, Morrison RG (eds) (2005) The Cambridge handbook of thinking and reasoning. Cambridge University Press, Cambridge Jasper H, Penﬁeld W (1954) Epilepsy and the functional anatomy of the human brain. Little, Brown, Boston Johnson-Laird PN (1983) Mental models. Towards a cognitive science of language, inference and consciousness. Harvard University Press, Cambridge Kert´esz A (1994) Localization and neuroimaging in neuropsychology. Academic, San Diego Kosslyn SM (1999) If neuroimaging is the answer, what is the question? Philos Trans R Soc Lond B Biol Sci 354:1283–1294 Lashley KS (1943) Studies of Cerebral Function in Learning. Journal of Comparative Neurology, volume 79 Logothetis NK (2008) What we can do and what we cannot do with fMRI, Nature 453:869–878 Luria AR (1963) Restoration of function after brain injury. Pergamon, Oxford Luria AR (1970) Traumatic aphasia: its syndromes, psychology, and treatment. de Gruyter, Berlin McIntosh AR (2004) Contexts and catalysts: a resolution of the localization and integration of function in the brain. Neuroinformatics 2:175–182 Newell A, Simon HA (1972) Human problem solving. Prentice Hall, Englewood Cliffs P¨oppel E (1989) Taxonomy of the subjective: an evolutionary perspective. In: Brown JW (ed) Neuropsychology of visual perception. Erlbaum, Hillsdale

Neural Correlates of Thinking

11

Poppelreuter W (1917) Die psychischen Sch¨adigungen durch Kopfschuß im Kriege 1914/17: mit besonderer Ber¨ucksichtigung der pathopsychologischen, p¨adagogischen, gewerblichen und sozialen Beziehungen, vol 1. Die St¨orungen der niederen und h¨oheren Sehleistungen durch Verletzungen des Okzipitalhirns. Leipzig Poppelreuter W (1918) Die psychischen Sch¨adigungen durch Kopfschuß im Kriege 1914/17: mit besonderer Ber¨ucksichtigung der pathopsychologischen, p¨adagogischen, gewerblichen und sozialen Beziehungen, vol 2. Die Herabsetzung der k¨orperlichen Leistungsf¨ahigkeit und des Arbeitswillens durch Hirnverletzung im Vergleich zu Normalen und Psychogenen. Leipzig Simon HA (1999) Problem solving. In: Wilson RA, Keil FC (eds) The MIT encyclopedia of the cognitive sciences. MIT Press, Cambridge, 674–676 Simon HAA (1999) Time for Talk and a Time for Silence. In: Streitz NA, Siegel J, Hartkopf V, Konomi S (eds) Cooperative Buildings, Integrating Information, Organization, and Architecture, Second International Workshop, CoBuild’99, Pittsburgh, USA, October 1–2, Proceedings. Lecture Notes in Computer Science 1670 Springer 1999 Singer W(1993) Synchronization of cortical activity and its putative role in information processing and learning. Annu Rev Physiol 55:349–374 Tversky A, Kahneman D (1986) Rational Choice and the Framing of Decisions. Journal of Business, 59:251–278 Umilta C (2003) Modularity in neural systems and localization of function. In: Encyclopedia of cognitive science, vol III. Nature, London Uttal WR (2001) The new phrenology: the limits of localizing cognitive processes in the brain. MIT Press, Cambridge Wernicke C (1874) Der aphasische Symptomencomplex. Cohn & Weigert, Breslau. p. 72

Functional MRI Limitations and Aspirations Peter A. Bandettini

Abstract Most would agree that knowing precisely what was happening in the brain during the act of thinking would help in our pursuit to understand what thinking really is. This chapter describes the basics, limits, and future directions of one of the more effective tools we have to observe the human brain while it is functioning – functional MRI. Functional MRI emerged in the early 1990s, and has since grown explosively in utility. In this chapter, an in-depth exploration is carried out of what limits functional MRI to a spatial resolution of millimeters and a temporal resolution of seconds. In addition, issues of how sensitive functional MRI is in detecting brain activity and how deeply we can interpret the signal changes are explored. Lastly, the chapter ends with a discussion on how imaging might be essential, or perhaps irrelevant, to understanding thinking.

1 Introduction Before 1991, the thought that one could use magnetic resonance imaging (MRI) to map human brain activation noninvasively, rapidly, with full brain coverage, and with relatively high spatial and temporal resolution was pure fantasy. This fantasy became reality with a rapidity and decisiveness that surprised almost everyone, causing the neuroscience community to rapidly readjust itself as it embraced this new modality. Many researchers tailored many of their ongoing behavioral, electrophysiological, or other imaging modality studies to the MRI scanner environment. Since then, functional MRI (fMRI) has proven to be a powerful and robust technique. Some argue that it is so easy to obtain eye-catching maps of “brain activation” that the quality of science performed with fMRI can be lacking at times. P.A. Bandettini Laboratory of Brain and Cognition, National Institute of Mental Health, Section on Functional Imaging Methods Building 10, Room 1D80B, 10 Center Dr. MSC 1148, Bethesda, MD 20892-1148, USA [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

15

16

P.A. Bandettini

However, truly innovative experiments and unique, insightful ﬁndings have indeed been obtained using fMRI. The reality is that fMRI is an improving, powerful, but sometimes misused and overinterpreted technique that is the tool of choice for a growing community of researchers, clinicians, and thinkers. In this chapter, the limits and aspirations of fMRI are explored. First, the basics of fMRI are introduced: the history, the physiologic processes behind the signal, and the current state of the ﬁeld. This section is concluded with a list of some of the advances that have helped to deﬁne fMRI. Section 2 focuses on the limits of temporal and spatial resolution, sensitivity, and signal interpretation. The last section is an attempt to put fMRI in the context of understanding thinking – an objective of the Parmenides Foundation. In this last section, the question of “What really does fMRI contribute to our understanding of thinking?” is addressed.

2 Basics 2.1 History and Functional Contrast The use of MRI to map brain activation in humans was introduced with a groundbreaking paper by Belliveau et al. (1991) in November of 1991 which described a technique involving two sequential bolus injections of the susceptibility contrast agent gadolinium-DTPA, to map blood volume during rest and activation. About the time that work was published, this approach was rendered obsolete (as far as functional activation imaging is concerned) by a completely noninvasive MRI-based technique utilizing endogenous functional contrast associated with localized changes in blood oxygenation during activation. Between the early spring and late fall of 1991, the ﬁrst successful experiments in endogenous functional contrast fMRI were carried out. The ﬁndings of these ﬁrst experiments were published within 2 weeks of each other in the early summer of 1992 (Bandettini et al. 1992; Kwong et al. 1992; Ogawa et al. 1992). The mechanism of endogenous contrast by which these early results were based was pioneered in animal and phantom studies by Ogawa et al. (Ogawa and Lee 1990; Ogawa et al. 1990a,b), who coined the term “blood oxygenation level dependent” (BOLD), as well as by Turner et al. (1991), who further demonstrated this contrast in cat models. The basics of the contrast mechanism are as follows. Hemoglobin is more paramagnetic (lower magnetic susceptibility) than tissue when deoxygenated, and has the same susceptibility as tissue when fully oxygenated. When it is deoxygenated and within a magnetic ﬁeld, microscopic ﬁeld inhomogeneities exist as a result of the different susceptibilities, leading to an attenuated magnetic resonance signal. During rest, venous blood is slightly deoxygenated. With activation, blood ﬂow locally increases more than what is required by an increase in oxidative metabolic rate, causing an increase in blood oxygenation and a decrease in the amount of deoxygenated hemoglobin, therefore creating a small magnetic resonance signal increase.

Functional MRI Limitations and Aspirations

17

Another noninvasive fMRI technique that emerged almost simultaneously with BOLD fMRI is known as arterial spin labeling (ASL) (Williams et al. 1992). The contrast in ASL arises from blood perfusion, independent of blood oxygenation. With ASL, blood is “labeled” with a radiofrequency (RF) pulse. This RF-labeled inﬂowing blood changes the signal in the plane being imaged as a function of blood perfusion in each voxel. ASL is unique in that maps of baseline and active-state perfusion may be made, whereas with BOLD contrast, only maps of changes can be obtained. Details of other techniques exist which allow noninvasive assessment of activation-induced changes in blood volume (Lu et al. 2003) and oxidative metabolic rate (Davis et al. 1998; Hoge et al. 1999), temperature (Yablonskiy et al. 2000a,b), and diffusion coefﬁcient (Le Bihan et al. 2006) have since been published. Currently, efforts are being made to develop a successful approach to directly imaging neuronal activity using MRI (Bandettini et al. 2005). So far, no results have been convincingly demonstrated, as models suggest that the sensitivity required is an order of magnitude higher than what is currently available when imaging humans. BOLD fMRI is the brain activation mapping method of choice for almost all neuroscientists because it is easiest to implement and the functional contrast to noise (deﬁned as the signal change magnitude divided by the background ﬂuctuation magnitude – ranging from 2 to about 6 with BOLD contrast) is generally a factor of 2–4 higher than for the other MRI-based methods. The need for sensitivity and ease of use outweighs, most of the time, the advantages in speciﬁcity, quantitation, or baseline information inherent to ASL. Picking up in 1992, only a handful of laboratories could perform fMRI because it required not only an MRI scanner but also the capability of performing highspeed MRI – known as echo planar imaging (EPI). EPI is a technique by which an entire image (or “plane”) is collected with the use of a single RF pulse and a single subsequent signal “echo” – hence the name “echo planar imaging.” Collecting an entire image in 30 ms (as with EPI) “freezes” physiologic processes that contribute to nonrepeatable artifacts in slower MRI methods, leading to a signiﬁcantly higher temporal stability – critical for fMRI. Until about 1996, the hardware for performing EPI was not available on clinical systems. Now, practically every standard MRI scanner is equipped to perform EPI. After about 1996, with rapid proliferation of EPI-capable MRI scanners incorporating whole-body gradients, the standard platform for fMRI reached a plateau that is still mostly in use today. In addition, the number of people able to perform fMRI increased dramatically. Figure 1 shows the increase in the number of articles and reviews published (using the Scopus search engine) dealing with fMRI. This standard platform pulse sequence typically used is gradient-echo EPI: echo time 40 ms, matrix size 64 × 64, ﬁeld of view 24 cm, slice thickness 4 mm (voxel dimension of about 4 × 4 × 4 mm). For studies incorporating spatial normalization and multisubject spatial averaging, going to any higher resolution gives no gains and a loss in sensitivity. (In fact, perhaps the optimal matrix size to use when performing multisubject averaging is 32 × 32 since sensitivity is increased, and the acquisition resolution approximately matches the resolution that spatial smoothing,

18

P.A. Bandettini

Fig. 1 The number of articles or reviews published per year, obtained from the Scopus search engine. The production of papers in functional magnetic resonance imaging (fMRI) shows no signs of slowing down

normalization, and multisubject averaging reduces functional images to.) Typically, whole brain volume coverage using a repetition time (TR) of 2 s is performed. Time series are collected, lasting on the order of 5–8 min. A typical experiment involves the collection of about seven time series per subject scanning session. Multisubject studies usually settle on assessing about 12 such sessions. Regarding hardware, in about the year 2002, the “standard” ﬁeld strength increased from 1.5 to 3 T, thus improving sensitivity. Currently, a new standard in RF coils has begun to take hold. In the past, the standard was the use of a quadrature RF coil. Currently, the trend is to use multireceiver coils, ranging from eight channels to 32 channels. The use of these coils translates into either an increase in sensitivity (smaller coils have greater sensitivity at the expense of less coverage – hence the need for more RF coils) or an increase in resolution (and/or speed) that comes with a novel strategy known as “sensitivity encoding.” Typical paradigm design methods are either “box-car,” involving steady-state activation periods for 10 s or more, or more commonly “even-related” designs enjoying the ﬂexibility inherent to brief activation periods interspersed within the time series. For postprocessing, SPM is the most common processing software program, but platforms such as Brain Voyager, FSL, and AFNI are almost as common (information regarding these platforms is at the end of the chapter). The fundamental concept in all of functional imaging creation is the statistical comparison of what is expected to happen in the hemodynamic response, as deﬁned by a “reference” function or a “regressor,” with the data, on a voxelwise basis.

Functional MRI Limitations and Aspirations

19

2.2 Advances An approximately chronological list of a few of the many signiﬁcant developments in fMRI is shown below. Neither this list nor the references associated with each topic are comprehensive. The goal is to provide a quick perspective of some of the highlights over the past 15 or so years and to give a sense that fMRI is a method that is very much in the hands of the users, as they drive many of the most innovative advances. • Parametric manipulation of brain activation demonstrated that BOLD contrast approximately followed the level of brain activation: visual system (Kwong et al. 1992), auditory system (Binder et al. 1994), and motor system (Rao et al. 1996). • Event-related fMRI was ﬁrst demonstrated (Blamire et al. 1992). The application of event-related fMRI to cognitive activation was shown (Buckner et al. 1996; McCarthy et al. 1997). Development of mixed event-related and block designs was put forward: (Donaldson et al. 2003). Paradigms were demonstrated in which the activation timing of multiple brain systems was orthogonal, allowing multiple conditions to be cleanly extracted from a single run (Courtney et al. 1997). • High spatial resolution maps were created: For spatial resolution ocular dominance columns (Cheng et al. 2001; Menon et al. 1997; Yacoub et al. 2006, 2007) and cortical layer activation maps (Logothetis et al. 2002) were created. Figure 2 illustrates graphically the spatial scales of brain organization that are able to be imaged with fMRI. First able to be imaged was the large-scale organization (i.e. V1, V2, etc..), then the ocular dominance column scale, then the layer-speciﬁc scale, and then the smallest scale – the orientation column scale.

FMRI-Accessible Scales of Visual Cortex Organization Parietal Lobes

Frontal Lobe Lateral Geniculate Nucleus (LGN)

Columns with direction selectivity

1

Occipital Lobes 2 and 3

V7 MT/V5

V3A LO

V2 V8 Temporal Lobe inferotemporal cortex (ITC)

Blobs

V3

V1 VP V4v Cerebellum

Input from M channel Input from P-IB channel

4A 4B 4Ca 4Cb 5 and 6 Input from right eye Input from left eye

Fig. 2 The spatial scales of cortical organization that are accessible to fMRI measures: functional units such as V1 ( left), cortical columns such as ocular dominance columns, cortical layers, and orientation columns (right). (Obtained from http://www.thebrain.mcgill.ca)

20

P.A. Bandettini

• High temporal resolution fMRI developed: relative onset timings from milliseconds to hundreds of milliseconds were extracted (Bellgowan et al. 2003; Henson et al. 2002; Menon et al. 1998; Ogawa et al. 2000). • The development of “deconvolution” methods allowed for rapid presentation of event-related stimuli (Dale and Buckner 1997). • Early BOLD contrast models were put forward (Buxton and Frank 1997; Ogawa et al. 1993). More sophisticated models were published that more fully integrated the latest data on hemodynamic and metabolic changes (Buxton et al. 2004). • The use of continuous variation of visual stimuli parameters as a function of time was proven a powerful method for fMRI-based retinotopy. (Deyoe et al. 1994; Engel et al. 1994; Sereno et al. 1995). • The development of “clustered volume” acquisition was put forth as a method to avoid scanner acoustic noise artifacts (Amaro et al. 2002; Edmister et al. 1999). • The ﬁndings of functionally related resting state correlations (Biswal et al. 1995) and regions that consistently show deactivation (Binder et al. 1999; Raichle et al. 2001) were described. This exploration of resting state connectivity has currently emerged as a major new research area in fMRI (Raichle and Snyder 2007). The very recent, explosive growth of this area of fMRI is illustrated in Fig. 3. • Observation of the pre-undershoot in fMRI (Hennig et al. 1997; Hu et al. 1997; Menon et al. 1995) and correlation with optical imaging (Malonek and Grinvald 1996) was reported. This is still highly controversial as the effect is very subtle

Fig. 3 The number of articles published per year which discuss resting state ﬂuctuations in fMRI. While this effect was ﬁrst discovered in 1995, this aspect of fMRI took off dramatically in 2006 and 2007

Functional MRI Limitations and Aspirations

• •

• •

•

•

•

21

and the hypothesized mechanisms producing it (rapid change in blood volume or ∆CMRO2 ) remain hotly debated. Structural equation modeling was developed in the context of fMRI time series analysis (Buchel and Friston 1998). Simultaneous use of fMRI and direct electrophysiological recording in nonhuman primate brains during visual stimulation has elucidated the relationship between fMRI and BOLD contrast (Logothetis et al. 2001), suggesting that BOLD contrast is more correlated with synaptic activity (local ﬁeld potentials) than with spiking activity. Simultaneous electrophysiological recordings in animal models revealed a correlation between negative signal changes and decreased neuronal activity (Shmuel et al. 2002). Simultaneous electrophysiological recordings in animal models provided evidence that inhibitory input could cause an increase in cerebral blood ﬂow (Matheiesen et al. 1998). A technique known as vascular space occupancy (VASO) has emerged as a way to noninvasively map blood volume changes (Lu et al. 2003). Recently, this technique has also come under some scrutiny. Extraction of information at high spatial frequencies within regions of activation was ﬁrst demonstrated (Haxby et al. 2001). This approach, which focuses on information extraction rather than mapping (Kriegeskorte et al. 2006), has developed rapidly as many groups are making efforts at “brain reading” rather than brain mapping (Cox and Savoy 2003; Haynes et al. 2005; Haynes and Rees 2005a,b; Kriegeskorte 2007a,b). The ﬁeld of “fMRI decoding” has also become a major direction of research in fMRI (Haynes et al. 2004; Haynes and Rees 2005a; Kamitani and Tong 2005; Kay et al. 2008). Several papers have been published showing the potential for direct neuronal current imaging with MRI (Bandettini et al. 2005; Buracas et al. 2008; Cassara et al. 2008; Kraus et al. 2008; Mandelkow et al. 2007; Matlachov et al. 2007; Park and Lee 2007; Park et al. 2004; Parkes et al. 2007; Singh and Sungkarat 2005; Truong et al. 2008; Xiong et al. 2003). No paper has been published showing convincing neuronal current activation maps in humans. While real-time fMRI has been in existence since at least 1995 (Cox et al. 1995), a paper was recently published (DeCharms et al. 2005) demonstrating that realtime feedback of brain activation to subjects experiencing chronic pain not only allowed them to modulate their activation, but that this resulted in their perception of the pain – opening up a potentially rich area of real time fMRI based therapy. The uses of parallel imaging and high ﬁeld strength have been the major developments on the technical side of fMRI (Bellgowan et al. 2006; De Zwart et al. 2004; Moeller et al. 2006; Preibisch et al. 2008). Parallel imaging allows for higher resolution, or a more rapid acquisition, or higher sensitivity. Higher ﬁeld allows for higher resolution and higher sensitivity. Interestingly, the progression of ﬁeld strength used for humans has been linear since clinical human MRI began around 1984. Figure 4 shows this progression. Scanners with ﬁeld strengths of 11.7 T have been proposed for 2011, keeping the trend linear.

22

P.A. Bandettini

Fig. 4 The progression of human MRI ﬁeld strength as a function of year. The trend is surprisingly linear. The ﬁnal data point is a proposed 11.7 T human scanner in Paris (Neurospin) in 2011

3 Limitations and Aspirations 3.1 Temporal Resolution An echo planar image has an acquisition window that is about 20–30 ms in duration. In general, about 15 echo planar images can be collected in 1 s. For volume collection, typically consisting of 30 slices, a TR of 2 s is therefore required. It is also possible to collect one image (as opposed to multiple images in a volume) at a rate of 15 images per second (TR = 1, 000 ms/15 images = 66.7 ms per image). Relative to the limits in temporal resolution imposed by the sluggishness and variability of the hemodynamic response in fMRI, the image acquisition rate is quite fast. The dynamics, location, and magnitude of the signal are highly inﬂuenced by the vasculature as it is sampled in each voxel. If a voxel happen to capture large vessel effects, the magnitude of the signal may be large (up to an order of magnitude larger than capillary effects), the timing a bit more delayed than average (up to 4 s delayed from capillary effects), and the location of the signal somewhat distal (up to 1 cm) from the true region of activation. The problem of variable vasculature and hemodynamic coupling in fMRI remains to some extent at all ﬁeld strengths and poses signiﬁcant limits on the depth and range of questions that can be addressed using fMRI.

Functional MRI Limitations and Aspirations

23

On average, the fMRI signal begins to increase approximately 2 s after neuronal activity begins, and plateaus in the “on” state after about 7–10 s. A “pre-undershoot” in the signal is sometimes observed at about 0.2–1.0 s and a post-undershoot is much more commonly observed, lasting up to 1 min. These more subtle dynamics are still not fully understood, but are likely due to temporal mismatches among the hemodynamic factors which most inﬂuence the signal: ﬂow, blood volume, or CMRO2 (Buxton and Frank 1997; Buxton et al. 2004). The hemodynamic response can be thought of as behaving like a low-pass ﬁlter for neuronal activity (Bandettini 1999; Kim et al. 1997; Richter et al. 1997). At on/off frequencies of 6 s on/6 s off (0.08 Hz), BOLD responses begin to be attenuated relative to longer on/off times. At on/off frequencies of 2 s on/2 s off (0.25 Hz) the BOLD response is almost completely attenuated. Even though BOLD signal is attenuated by these rapid on/off responses, activity of very brief duration can be observed. Activity durations as low as 16 ms have been shown to cause robust BOLD signal changes, suggesting that there is no apparent limit to the briefness of detectible activation (Birn and Bandettini 2005). When repeated experiments are performed, the hemodynamic response in each voxel shows variability of only about 100 ms (Bandettini 1999). A strong desire of those who use functional brain imaging is to determine the precise timing of activation between different regions of the brain – either relative to the stimulus or input or relative to each other. The temporal resolution required for this type of assessment is on the order of at least tens of milliseconds. With BOLD contrast, the latency of the hemodynamic response has a range of 4 s across voxels owing to spatial variations in underlying hemodynamics or neurovascular coupling dynamics from voxel to voxel even within the same region of activity (Bandettini 1999). If a voxel contains mostly larger venous vessels, the response is typically more delayed than if the voxel captures predominantly capillaries. This observation is only approximate. The precise reasons for latency variations have still not been completely determined. Methods have been proposed to alleviate this problem of spatial heterogeneity of the latency of the hemodynamic response. The most direct is to try to identify and remove larger vessel effects by thresholding based on the percentage signal change or temporal ﬂuctuation characteristics. The accuracy of these methods remains undetermined and is likely to be low since high percentage signal changes may occur quite proximal to an active area (and therefore should not be eliminated), and draining veins may have low ﬂuctuations (and therefore be missed, while they should be eliminated). Another solution is to use pulse sequences sensitive only to capillary effects. ASL techniques are more sensitive to capillaries, but the practical limitations of lower functional contrast to noise and longer interimage waiting time (owing to the additionally required time to excite the inﬂowing blood and to wait for it to arrive in the plane – about 1.5 s) make this unworkable for most studies. An alternative strategy to push temporal resolution is to focus on localized changes in latency and width of the hemodynamic response with task timing changes. As mentioned, within a voxel, the hemodynamic response, while exhibiting a delay of 4 s from the mean, still only shows a variation (with repeated, identical

24

P.A. Bandettini

trials) on the order of 100 ms, allowing signiﬁcantly more accurate assessment if activation timing were to be varied within a region (Bellgowan et al. 2003; Henson et al. 2002). One other unique method for probing very rapid neuronal interactions was pioneered by Ogawa et al. (2000). In this approach, paired pulses of stimuli either activate the same region or activate two different regions that are connected by inhibitory or excitatory synaptic input. The time between the stimuli pairs is modulated and the amplitude modulation of the second response is observed. This method has demonstrated a 50-ms optimal inhibitory timing between left and right forepaw in rat – as indicated by maximal reduction of the second BOLD response at that timing, and also has shown a 100-ms optimal inhibition in human visual cortex in the same manner. The precise neuronal mechanisms behind these ﬁndings are still not fully understood, but the method itself is potentially quite useful for probing the timing and connectivity between either excitatory or inhibitory processing nodes in the brain using fMRI. A summary of current temporal limits of fMRI as well as speculations on improvements are given below. Temporal limits: • Able to detect transient activity as short as 16 ms. • Able to create a functional image within 20 s for more robust activation (more an issue of sensitivity, but can be considered practical temporal resolution). • Able to detect differences in brain activation timing across regions of no less than 4–6 s because across voxels the temporal lag varies by 4 s. • Able to detect modulations in brain activation timing within the same voxel or region of the brain that are no smaller than 100 ms. • Able to detect inhibitory interaction, with no temporal limit, between connected and interacting nodes of activity in the brain, provided that these interactions are able to cause detectible modulations in BOLD signal in the area being affected. Temporal aspirations: • Calibration methods (identiﬁcation and removal of draining vein effects) are being developed to reduce physiologic ﬂuctuations such that the temporal resolution would be most inﬂuenced by temporal signal to noise, perhaps pushing all temporal resolutions to about 50 ms. • Neuronal current imaging aspires to bypass hemodynamics altogether, aiming to detect neuronal transients on the order of 10 ms. This will take a breakthrough to be accomplished, since, so far, no robust results at all have been observed using neuronal current imaging in humans.

3.2 Spatial Resolution Single-shot (one excitation RF pulse per imaging plane) imaging is easily the most robust, stable, and common imaging procedure in fMRI. The primary drawback

Functional MRI Limitations and Aspirations

25

is that the spatial resolution and overall image quality of echo planar images are signiﬁcantly less that those of clinical anatomical scans. The upper in-plane resolution of standard single-shot EPI is about 2 mm2 . One of the most promising developments in fMRI scanner technology has been the use of multiple parallel RF receive coils to help spatially encode the data, thus allowing for much higher resolution with a single excitation pulse. This approach can allow functional image resolutions of about 1 mm3 . Nevertheless, because the voxel volume directly determines functional signal to noise, the signal to noise of these high-resolution images is considerably lower than the signal to noise of lower-resolution images, requiring functional imaging to be performed at 3 T or preferably higher (because of greater image signal to noise, and larger functional contrast at higher ﬁeld strengths) to produce useful data in a workable amount of time (Murphy et al. 2007). A fundamentally important caveat to improving spatial resolution that is worth mentioning at this point is that most fMRI studies involve spatial smoothing, spatial normalization, and multisubject averaging – effectively reducing the spatial resolution to, at best, 10 mm3 and completely nullifying any advantages of collecting data at high resolution, and at high ﬁeld for that matter. The main purpose of high-resolution studies is single-subject assessment that is not subsequently spatially smoothed, transformed into a normalized space, and averaged with 20+ other brains. Individual subject assessment is a growing area of fMRI as methods are being developed to extract ever more subtle and useful information on a subjectwise basis (Kriegeskorte and Bandettini 2007a,b). There is no compelling reason to perform fMRI at resolution higher than 4 mm3 or lower if multisubject averaging is part of the processing path. As with temporal resolution limits, the spatial resolution limits are predominantly determined not by limits in the acquisition method but by the relatively wide spatial spread of oxygenation and perfusion changes that accompany focal brain activation. This “hemodynamic point spread function” has been empirically determined to be on the order of 3.5 mm (Engel et al. 1997). At 7 T, more sensitive to microvessels, Shmuel et al. (2007) found the point spread function to be about 2.3 mm. Interestingly, they also found that the point spread function was narrower for BOLD signal change obtained during the third (1.52 mm) and fourth (1.99 mm) seconds of stimulation. The approaches for dealing with the hemodynamic smoothing function are similarly applicable as those for temporal resolution limits. The primary effort has been to eliminate large vessel effects while preserving sufﬁcient functional contrast for creating functional images in a reasonable amount of time. Approaches to eliminate large vessel effects have generally included: (1) The use of spin-echo imaging. A spin-echo image (as opposed to the more typically used gradient-echo image) is more sensitive to small susceptibility variations but is still sensitive to red blood cells in large vessels (Jochimsen et al. 2004) – except at high ﬁelds where intravascular signal is almost completely gone because of T2 shortening of deoxygenated blood.

26

P.A. Bandettini

(2) The use of high ﬁeld. At high ﬁeld, there is slightly more sensitivity to small vessels and less to large vessel intravascular signal since both T2∗ and T2 of blood decrease. (3) The use of diffusion weighting (otherwise known as “velocity nulling” in this context). Diffusion weighting removes rapidly ﬂowing large vessel intravascular signal but not extravascular effects. (4) The use of ASL. Imaging capillary perfusion completely bypasses the large vessel problem. This method has a contrast to noise that is about a factor of 2–4 lower than T2∗ BOLD signal except for long duration activation paradigms where the BOLD baseline tends to drift, whereas the ASL baseline is steady (Wang et al. 2003). The combination of approaches 1 and 3 could work in theory but, in practice, there is no functional signal left owing to signal-to-noise limitations. The combination of approaches 1 and 2 has been used successfully for mapping ocular dominance columns. Another approach to increasing functional spatial resolution is calibration of the hemodynamic factors which inﬂuence BOLD signal change. Spatial calibration methods have been proposed involving hypercapnia (Bandettini and Wong 1997; Cohen et al. 2004; Thomason et al. 2007). CO2 stress (Chiarelli et al. 2007; Handwerker et al. 2007), and, recently, even the resting state ﬂuctuation data (Birn et al. 2008). The general idea in calibration is to create a map of the “potential magnitude of BOLD” by giving a global hemodynamic stress that is evenly distributed throughout the brain (this maps resembles closely a map of gray and white matter combined with a venous angiogram), then to divide activation-induced signal changes, on a voxelwise basis, by this map of signal change to a global stress. In spite of the limitations in spatial resolution, ocular dominance column (1 mm3 ) (Cheng et al. 2001; Goodyear and Menon 2001), cortical layer (less than 0.5 mm3 ) (Logothetis et al. 2002), and orientation column (Yacoub et al. 2006) delineation have been achieved. An ongoing issue with regard to the upper resolution of fMRI is whether or not ﬁne delineation necessarily translates to accurate delineation (Kriegeskorte and Bandettini 2007a,b) – meaning that detailed activation maps may not be precisely registered with underlying function. This remains to be demonstrated using a gold standard, but compelling data suggesting ﬁne and accurate delineation have been presented in an animal model in which optical imaging data were compared with hemodynamic changes as measured by fMRI (Fukuda et al. 2006; Moon et al. 2007a,b). Lastly, a method involving neuronal adaptation paradigms may be able to selectively image neuronal populations on a subvoxel scale. This approach has been termed “fMR-adaptation” (Grill-Spector and Malach 2001), and relies on the relatively rapid adaptation and recovery properties of speciﬁc neuronal pools, and the reﬂection of these properties in fMRI signal, to characterize and differentiate subvoxel populations of neurons that are sensitive to subtle differences in stimulus or general paradigm properties. Grill-Spector and Malach described this method as a paradigm that proceeds in two stages: ﬁrst, a neuronal population is adapted by repeated presentation of a single stimulus; second, a property of the stimulus is

Functional MRI Limitations and Aspirations

27

varied and the recovery from adaptation (manifest as an increase in fMRI signal) is assessed. If the signal remains adapted, it indicates that the neurons are invariant to that attribute. However, if the fMRI signal recovers from the adapted state, it implies that the neurons are sensitive to the property that was varied. A summary of current spatial limits of fMRI as well as speculation on improvements is given below. Spatial limits: • At 3 T, only enough sensitivity to practically achieve 1.5-mm3 resolution. The functional point spread function is about 3.5 mm. • At 7 T, enough sensitivity to practically achieve 0.5-mm3 resolution. The functional point spread function can be has high as 1.5 mm. • At 7 T, using spin-echo sequences, the smallest resolved functional unit was orientation columns (on the order of 0.5-mm width). • With fMR-adaptation paradigms, the highest resolution is unknown, as very small pools of neurons within voxels may be selectively modulated. • With calibration methods using a global hemodynamic stress, it is speculated that the functional point spread function can be reduced to 1.5 mm at all ﬁeld strengths. Spatial aspirations: • It’s not clear that there is a need to go to a resolution higher than 0.5 mm. With methods for improving sensitivity (RF coil addition being most promising), and selectivity to capillaries, and neuronal subpopulations, the functional resolution may be readily approaching 0.5 mm. It is not clear how abundant functional units smaller than 0.5 mm are. • Averaging or pooling of multi-subject high-resolution data remains a challenge. I believe that normalization algorithms have room for improvement – as more information about the principles of brain variability may be incorporated. It’s hard to speculate on how high a resolution multisubject averaging may achieve. It may reach 2 mm as algorithms become more focused on particular brain structures.

3.3 Sensitivity Currently, the functional contrast to noise ratio in fMRI is about 4:1 at typical resolutions at 3 T. This means that the functional signal change is approximately 4 times larger than the underlying noise levels. Increases in sensitivity can directly translate into being able to extract more subtle functional information either in space or in time, and as shown in Fig. 5 can translate into creating a usable functional image in signiﬁcantly less time – extremely important in a clinical setting. In general, it is perhaps the most desired commodity in fMRI, as most researchers are willing to sacriﬁce temporal resolution, spatial resolution, and higher speciﬁcity in order to maximize sensitivity.

28

P.A. Bandettini

Fig. 5 Simulated data illustrating the point that to detect a 1% signal change at p = 5 × 10−10 , one needs to scan for 50 min when at a signal-to-noise ratio (SNR) of 50, for 21 min when the SNR improves by 50% to 75, and for 10 min when the SNR doubles from that of the 50-min scan. This is a highly nonlinear beneﬁt obtained from an improvement in SNR. (Adapted from Murphy et al. 2007.)

Sensitivity to brain activation related hemodynamic changes can be increased in the following ways: (1) Increasing image signal-to-noise ratio – either by reducing RF coil size (and adding more for whole brain coverage), or increasing ﬁeld strength (which improves both signal change magnitude and signal-to-nose ratio). (2) Increasing the activation-induced signal change – by increasing ﬁeld strength, or, in animal studies, using an intravascular paramagnetic contrast agent (Smirnakis et al. 2007). (3) Better modeling and accounting for variations in activation-induced signal changes, as much sensitivity is lost when the temporal models do not match the signal change dynamics exactly. The hemodynamic response varies considerably in shape, latency, and with across regions, individuals, and voxels. (4) Better modeling of and accounting for the noise – mostly the physiologic noise (which includes respiration, cardiac, respiration changes, etc.). This noise is additionally problematic since it sets an upper bound on the temporal signal-to-noise ratio (Bodurka et al. 2007; Triantafyllou et al. 2005). This upper bound is about 100:1 (1% ﬂuctuations). Sensitivity limits: • Even though image signal-to-noise ratio can be as high as 800:1, we are currently limited to a temporal signal-to-noise ratio of about 100:1 across all ﬁeld strengths (Bodurka et al. 2007). This limit is most strongly inﬂuenced by physiologic

Functional MRI Limitations and Aspirations

29

Fig. 6 The relationship between temporal SNR (y-axis) and image SNR (x-axis). If no physiologic ﬂuctuations exist, the relationship is a straight line – they would equal each other as shown in the phantom data. In reality, the temporal SNR in gray matter plateaus around 100:1, as image SNR continues to increase. The vertical lines indicate what is the “optimal” resolutions to scan at – where temporal and spatial SNRs are highest and closest to each other. If one collects a time series of images in which the temporal SNR is greater than 100, that person is throwing away signal that could be translated into perhaps higher resolution. WM white matter, GM gray matter, CSF cerebrospinal ﬂuid. (Adapted from Bodurka et al. 2007.)

ﬂuctuations that occur over time. A graphic illustration of this effect is shown in Fig. 6, in which the temporal signal-to-noise ratio in gray matter is shown to plateau, even as image signal-to-noise ratio continues to increase. • The functional contrast-to-noise ratio is about 4:1 at 3 T and up to 5:1 at 7 T. Sensitivity aspirations: • The goal is to achieve a temporal signal-to-noise ratio that matches the image signal-to-noise ratio above 100:1. While many physiologic ﬂuctuations can be accounted for, to account for all nonneuronal ﬂuctuations would require a much higher sampling rate and better spatial and temporal modeling of the noise. A temporal signal-to-noise ratio of 200:1 and a functional contrast-to-noise ratio of 10:1 is likely to be achievable relatively soon. • Processing methods that take into account “patterns” of activation rather than individual voxels as independent measures show considerable promise not only for fMRI decoding efforts but also for increasing sensitivity (Kriegeskorte and Bandettini 2007b; Kriegeskorte et al. 2006). These methods are still in their infancy, and their potential is yet to be fully realized. Fundamentally, this will be a paradigm shift in fMRI since, instead of looking for 1-cm “blobs” of activation, we will start looking for unique patterns of activation, across several spatial scales – down to individual voxelwise patterns.

30

P.A. Bandettini

3.4 Interpretation A fundamental goal in fMRI is to be able to infer precisely where, when, and how much neuronal activity is taking place in the brain on the basis of the measured BOLD signal. This goal is problematic since BOLD changes depend on variables other than neuronal activity itself, including hemodynamic coupling and volume in each voxel. The hemodynamics vary from voxel to voxel, so even if studies demonstrate that within a region there is a relationship between neuronal activity and BOLD signal, this does not get any closer, in practice, to being able to say that “neuronal activity is x in this voxel.” To do this, spatial calibration (voxelwise calibration) of the hemodynamic response is necessary. Progress has been made in at least conﬁrming the BOLD signal change is a reliable and a high enough ﬁdelity measure of neuronal activation to be widely used and depended on. Strategies for characterizing the relationship between neuronal activity and BOLD signal changes have included (1) animal models and the simultaneous use of other measures of neuronal activity such as multiunit electrodes or more precise measures of hemodynamic changes, such as optical imaging: (2) parametric modulation of magnitude or timing of activation in humans with corresponding measurement of fMRI signal changes; (3) simultaneous measures of neuronal activity (implanted electrode or electroencephalography, EEG) and fMRI signal changes; (4) nonsimultaneous measures of neuronal activity (magnetoencephalography, MEG; EEG) and fMRI signal changes; and (5) modeling of the hemodynamic response and comparison with precise activation magnitude, timing, or pharmacological manipulations. Primary ﬁndings of these efforts have been BOLD signal changes appear to be driven by synaptic activity, as indicated by ﬁeld potentials, rather than spiking itself (Viswanathan and Freeman 2007; Logothetis et al. 2001; Niessing et al. 2005), and MEG coherence changes in the gamma frequency range correlate spatially with fMRI signal changes (Muthukumaraswamy and Singh 2008; Singh et al. 2002). Interpretation limits: • Cannot differentiate inhibitory from excitatory activity. • BOLD signal change is not a quantitative measure. Hemodynamic factors (baseline blood volume, neurovascular coupling) inﬂuence location, magnitude, and dynamics. • Interpretation aspirations. • Work has been progressing rapidly in the area of calibration. It will likely be possible to perform a spatial calibration of BOLD signal using the resting state ﬂuctuation data (accompanied by a measurement of breathing depth using a chest strap). This will not only increase spatial speciﬁcity but will also reduce intrasubject variability and increase statistical power when averaging multisubject data. • Convergent evidence from multimodal studies will continue to increase the conﬁdence in the ﬁdelity of the relationship between BOLD signal and underlying neuronal activity. This will have direct impact on the clinical applications of fMRI.

Functional MRI Limitations and Aspirations

31

4 What About Thinking? What about thinking? A major theme in this book is the quest to understand thinking. The question that most reading this chapter will want to know the answer to is: “What can fMRI, or more generally, neuroimaging, contribute to our pursuit of an understanding of thinking?” Does it really help to be able to look into the brain? To borrow an analogy, can one really truly understand how computers work by opening up a computer chassis and probing the components with a heat gun? Can identifying the when, where, and how much in the brain provide enough information so that we can begin, from this information, to derive principles of thinking? Even if we had a perfect picture at inﬁnite spatial and temporal resolution of what was actually happening in the brain during thought, would we even then begin to understand thinking? Does it really matter what the limits of fMRI are with regard to answering questions about thinking? It seems apparent that to truly understand the brain, a much wider context (physical and evolutionary factors) needs to be considered. Thinking itself might someday be deconstructed into simple algorithms that can be carried out within different media other than brains. Perhaps a simple model of interacting layers of neuronal networks may emerge as being able to explain thought (Hawkins and Blakeslee 2004). It is my feeling that because thinking is a subjective process, it tends to be shrouded in mystery, and potentially elevated to a status, either correctly or incorrectly, that deﬁes understanding. fMRI has been an extremely effective tool with regard to deepening our understanding of speciﬁc aspects of thinking. Speciﬁc regions, networks, dynamics, and patterns have been revealed as they are associated with, among many other processes, learning (Cabeza and Nyberg 2000; Peissig and Tarr 2007; Poznanski and Riera 2006), working memory (Wager and Smith 2003), emotional processing (Beauregard 2007; Singewald 2007; Wildgruber et al. 2006; Yurgelun-Todd and Ross 2006; Zald 2003), moral judgment (Heekeren et al. 2003), sense of free will (Brass and Haggard 2007; Goldberg et al. 2007), theory of mind (Gallagher et al. 2000; Saxe and Kanwisher 2003; Vogeley et al. 2001), deception (Langleben 2008; Langleben et al. 2006; Phillips 2004), social interaction (Montague et al. 2002), humor (Berns 2004; Moran et al. 2004; Watson et al. 2007; Wild et al. 2006), and introspection (Goldberg et al. 2006). At the end of the day, we might be able to then say that x network, on x spatial scale, is directly related to say, theory of mind, willed action, and humor. So fMRI reveals the functions of speciﬁc processing modules. Does this really tell us anything that will help our understanding of thinking? Do we need to know what modules overlap in function or how large they are or where they are located in the brain? Does this information really matter? What spatial scale in the brain is the most critical for the understanding of thinking? While all of our tools are able to probe many different spatial scales, there are also many which have not been investigated yet. Does this matter? I believe that more will be understood about thinking once we can integrate data across all temporal and spatial scales and use this information to construct testable

32

P.A. Bandettini

models of thinking. The fMRI speciﬁc spatial scale of millimeters and a temporal resolution, in most cases, of seconds is a relatively narrow temporal/spatial niche to be studying how humans think. The human thinking process has evolved as a strategy for human survival in very speciﬁc context. One might say that how we see the world and, therefore, how we think about the world, is highly tuned to the physical and social environment in which we evolved. fMRI can tell us how the brain works on a very speciﬁc temporal and spatial scale. It can certainly contribute to but not provide the whole story of how we think. To answer this we need to draw upon not only the vast array of imaging and behavioral measures, but also on data-driven models of how functional units in the brain interact across spatial and temporal scales to create such emergent activity such that human beings can see, hear, feel, move, react, solve problems, create, learn, and introspect.

5 Further Information MRI and fMRI basics: • http://www.simplyphysics.com/MAIN.HTM • http://deﬁant.ssc.uwo.ca/Jody web/fmri4dummies.htm Processing software: • http://afni.nimh.nih.gov/afni: Analysis of Functional NeuroImages by Bob Cox, NIMH • http://www.bic.mni.mcgill.ca/software/: from the Brain Imaging Center at McGill University • http://grommit.lrdc.pitt.edu/ﬁswidgets/: a Java graphical user interface for a number of neuroimaging analysis packages • http://brainmapping.loni.ucla.edu/BMD HTML/SharedCode/SharedSoftware. html: general analysis tools from UCLA brain imaging center • http://www.mayo.edu/bir/Software/Analyze/Analyze.html: from the Mayo Clinic • http://www.brainvoyager.com/: a commercial product from Brain Innovation (Rainer Goebel) • http://www.math.mcgill.ca/keith/fmristat/: a set of useful MATLAB programs (Keith Worsley) • http://www.fmrib.ox.ac.uk/fsl/: a comprehensive set of analysis programs (Steve Smith, Oxford University) Books: • Introduction to Functional Magnetic Resonance Imaging: Principles and Techniques, by Richard Buxton, Cambridge University Press, Cambridge, 2001 • Functional Magnetic Resonance Imaging, by Scott A. Huettel, Allen W. Song, and Gregory McCarthy, Sinauer Associates, Sunderland, 2004

Functional MRI Limitations and Aspirations

33

• Functional MRI: An Introduction to Methods (Eds. Peter Jezzard, Paul M. Mattthews, and Stephen M. Smith), Oxford University press, New York, 2003 • Functional MRI (Eds. Chrit Moonen and Peter A. Bandettini), Springer, Berlin, 1999 • Functional MRI: Basic Principles and Clinical Applications (Eds. Scott H. Faro, and Feroze B. Mohamed), Springer, Berlin, 2005 fMRI course Web sites: • http://www.nmr.mgh.harvard.edu/fmrivfp/ • http://www.ﬁrc.mcw.edu/course/

References Amaro E, Williams SCR, Shergill SS, Fu CHY, MacSweeney M, Picchioni MM, Brammer MJ, McGuire PK (2002) Acoustic noise and functional magnetic resonance imaging: current strategies and future prospects. J Magn Reson Imaging 16:497–510 Bandettini PA (1999) The temporal resolution of functional MRI. In: Moonen C, Bandettini P (eds) Functional MRI. Springer, Berlin Heidelberg New York, pp 205–220 Bandettini PA, Wong EC (1997) A hypercapnia-based normalization method for improved spatial localization of human brain activation with fMRI. NMR Biomed 10:197–203 Bandettini PA, Wong EC, Hinks RS, Tikofsky RS, Hyde JS (1992) Time course EPI of human brain-function during task activation. Magn Reson Med 25:390–397 Bandettini PA, Petridou N, Bodurka J (2005) Direct detection of neuronal activity with MRI: fantasy, possibility, or reality? Appl Magn Reson 29:65–88 Beauregard M (2007) Mind does really matter: evidence from neuroimaging studies of emotional self-regulation, psychotherapy, and placebo effect. Prog Neurobiol 81:218–236 Bellgowan PSF, Saad ZS, Bandettini PA (2003) Understanding neural system dynamics through task modulation and measurement of functional MRI amplitude, latency, and width. Proc Natl Acad Sci U S A 100:1415–1419 Bellgowan PSF, Bandettini PA, Van Gelderen P, Martin A, Bodurka J (2006) Improved BOLD detection in the medial temporal region using parallel imaging and voxel volume reduction. Neuroimage 29:1244–1251 Belliveau JW, Kennedy DN, McKinstry RC, Buchbinder BR, Weisskoff RM, Cohen MS, Vevea JM, Brady TJ, Rosen BR (1991) Functional mapping of the human visual-cortex by magneticresonance-imaging. Science 254:716–719 Berns GS (2004) Something funny happened to reward. Trends Cogn Sci 8:193–194 Binder JR, Rao SM, Hammeke TA, Frost JA, Bandettini PA, Hyde JS (1994) Effects of stimulus rate on signal response during functional magnetic-resonance-imaging of auditory-cortex. Cogn Brain Res 2:31–38 Binder JR, Frost JA, Hammeke TA, Bellgowan PSF, Rao SM, Cox RW (1999) Conceptual processing during the conscious resting state: a functional MRI study. J Cogn Neurosci 11:80–93 Birn RM, Bandettini PA (2005) The effect of stimulus duty cycle and “off” duration on BOLD response linearity. Neuroimage 27:70–82 Birn RM, Smith MA, Jones TB, Bandettini PA (2008) The respiration response function: the temporal dynamics of fMRI signal ﬂuctuations related to changes in respiration. Neuroimage 40:644–654 Biswal B, Yetkin FZ, Haughton VM, Hyde JS (1995). Functional connectivity in the motor cortex of resting human brain using echo-planar MRI. Magn Reson Med 34:537–541

34

P.A. Bandettini

Blamire AM, Ogawa S, Ugurbil K, Rothman D, McCarthy G, Ellermann JM, Hyder F, Rattner Z, Shulman RG (1992) Dynamic mapping of the human visual-cortex by high-speed magneticresonance-imaging. Proc Natl Acad Sci U S A 89:11069–11073 Bodurka J, Ye F, Petridou N, Murphy K, Bandettini A (2007) Mapping the MRI voxel volume in which thermal noise matches physiological noise – implications for fMRI. Neuroimage 34:542–549 Brass M, Haggard P (2007) To do or not to do: the neural signature of self-control. J Neurosci 27:9141–9145 Braver T, Reynolds JR, Donaldson DI (2003) Neural mechanisms of transient and sustained cognitive control during task switching, Neuron 39:713–726 Buchel C, Friston KJ (1998) Dynamic changes in effective connectivity characterized by variable parameter regression and Kalman ﬁltering. Hum Brain Mapp 6:403–408 Buckner RL, Bandettini PA, Ocraven KM, Savoy RL, Petersen SE, Raichle ME, Rosen BR (1996) Detection of cortical activation during averaged single trials of a cognitive task using functional magnetic resonance imaging. Proc Natl Acad Sci U S A 93:14878–14883 Buracas GT, Liu TT, Buxton RB, Frank LR, Wong EC (2008) Imaging periodic currents using alternating balanced steady-state free precession. Magn Reson Med 59:140–148 Buxton RB, Frank LR (1997) A model for the coupling between cerebral blood ﬂow and oxygen metabolism during neural stimulation. J Cereb Blood Flow Metab 17:64–72 Buxton RB, Uludag K, Dubowitz DJ, Liu TT (2004) Modeling the hemodynamic response to brain activation. Neuroimage 23:S220–S233 Cabeza R, Nyberg L (2000) Imaging cognition II: an empirical review of 275 PET and fMRI studies. J Cogn Neurosci 12:1–47 Cassara AM, Hagberg GE, Bianciardi M, Migliore M, Maraviglia B (2008) Realistic simulations of neuronal activity: a contribution to the debate on direct detection of neuronal currents by MRI. Neuroimage 39:87–106 Cheng K, Waggoner RA, Tanaka K (2001) Human ocular dominance columns as revealed by highﬁeld functional magnetic resonance imaging. Neuron 32:359–374 Chiarelli PA, Bulte DP, Wise R, Gallichan D, Jezzard P (2007) A calibration method for quantitative BOLD fMRI based on hyperoxia. Neuroimage 37:808–820 Cohen ER, Rostrup E, Sidaros K, Lund TE, Paulson OB, Ugurbil K, Kim SG (2004) Hypercapnic normalization of BOLD fMRI: comparison across ﬁeld strengths and pulse sequences. Neuroimage 23:613–624 Courtney SM, Ungerleider BG, Keil K, Haxby JV (1997) Transient and sustained activity in a distributed neural system for human working memory. Nature 386:608–611 Cox DD, Savoy RL (2003) Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19:261–270 Cox RW, Jesmanowicz A, Hyde JS (1995) Real-time functional magnetic-resonance-imaging. Magn Reson Med 33:230–236 Dale AM, Buckner RL (1997) Selective averaging of rapidly presented individual trials using fMRI. Hum Brain Mapp 5:329–340 Davis TL, Kwong KK, Weisskoff RM, Rosen BR (1998) Calibrated functional MRI: mapping the dynamics of oxidative metabolism. Proc Natl Acad Sci U S A 95:1834–1839 DeCharms RC, Maeda F, Glover GH, Ludlow D, Pauly JM, Soneji D, Gabrieli JDE, Mackey SC (2005) Control over brain activation and pain learned by using real-time functional MRI. Proc Natl Acad Sci U S A 102:18626–18631 Deyoe EA, Bandettini P, Neitz J, Miller D, Winans P (1994) Functional magnetic-resonanceimaging (FMRI) of the human brain. J Neurosci Methods 54:171–187 De Zwart JA, Ledden PJ, Van Gelderen P, Bodurka J, Chu R, Duyn JH (2004) Signal-to-noise ratio and parallel imaging performance of a 16-channel receive-only brain coil array at 3.0 tesla. Magn Reson Med 51:22–26 Edmister WB, Talavage TM, Ledden PJ, Weisskoff RM (1999) Improved auditory cortex imaging using clustered volume acquisitions. Hum Brain Mapp 7:89–97

Functional MRI Limitations and Aspirations

35

Engel SA, Rumelhart DE, Wandell BA, Lee AT, Glover GH, Chichilnisky EJ, Shadlen MN (1994) FMRI of human visual-cortex. Nature 369:525 Engel SA, Glover GH, Wandell BA (1997) Retinotopic organization in human visual cortex and the spatial precision of functional MRI. Cereb Cortex 7:181–192 Fukuda M, Moon CH, Wang P, Kim SG (2006) Mapping iso-orientation columns by contrast agentenhanced functional magnetic resonance imaging: reproducibility, speciﬁcity, and evaluation by optical imaging of intrinsic signal. J Neurosci 26:11821–11832 Gallagher HL, Happe F, Brunswick N, Fletcher PC, Frith U, Frith CD (2000) Reading the mind in cartoons and stories: an fMRI study of ‘theory of mind’ in verbal and nonverbal tasks. Neuropsychologia 38:11–21 Goldberg II, Harel M, Malach R (2006) When the brain loses its self: prefrontal inactivation during sensorimotor processing. Neuron 50:329–339 Goldberg I, Ullman S, Malach R (in press) Neuronal correlates of “free will” are associated with regional specialization in the human intrinsic/default network. Consciousness Cogn Goodyear BG, Menon RS (2001) Brief visual stimulation allows mapping of ocular dominance in visual cortex using fMRI. Hum Brain Mapp 14:210–217 Grill-Spector K, Malach R (2001) fMR-adaptation: a tool for studying the functional properties of human cortical neurons. Acta Psychol 107:293–321 Handwerker DA, Gazzaley A, Inglis BA, D’Esposito M (2007) Reducing vascular variability of fMRI data across aging populations using a breathholding task. Hum Brain Mapp 28:846–859 Hawkins J, Blakeslee S (2004) On intelligence. Holt, New York Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425–2430 Haynes JD, Rees G (2005a) Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat Neurosci 8:686–691 Haynes JD, Rees G (2005b) Predicting the stream of consciousness from activity in human visual cortex. Curr Biol 15:1301–1307 Haynes JD, Lotto RB, Rees G (2004) Responses of human visual cortex to uniform surfaces. Proc Natl Acad Sci U S A 101:4286–4291 Haynes JD, Driver J, Rees G (2005) Visibility reﬂects dynamic changes of effective connectivity between V1 and fusiform cortex. Neuron 46:811–821 Heekeren HR, Wartenburger I, Schmidt H, Schwintowski HP, Villringer A (2003) An fMRI study of simple ethical decision-making. Neuroreport 14:1215–1219 Hennig J, Janz C, Speck O, Ernst T (1997) Is there a different type of MR-contrast in the early phase of functional activation? Adv Exp Med Biol 2:35–42 Henson RNA, Price CJ, Rugg MD, Turner R, Friston KJ (2002) Detecting latency differences in event-related BOLD responses: application to words versus nonwords and initial versus repeated face presentations. Neuroimage 15:83–97 Hoge RD, Atkinson J, Gill B, Crelier GR, Marrett S, Pike GB (1999) Investigation of BOLD signal dependence on cerebral blood ﬂow and oxygen consumption: the deoxyhemoglobin dilution model. Magn Reson Med 42:849–863 Hu XP, Le TH, Ugurbil K (1997) Evaluation of the early response in fMRI in individual subjects using short stimulus duration. Magn Reson Med 37:877–884 Jochimsen TH, Norris DG, Mildner T, Moller HE (2004) Quantifying the intra- and extravascular contributions to spin-echo fMRI at 3 T. Magn Reson Med 52:724–732 Kamitani Y, Tong F (2005) Decoding the visual and subjective contents of the human brain. Nat Neurosci 8:679–685 Kay KN, Naselaris T, Prenger RJ, Gallant JL (2008) Identifying natural images from human brain activity. Nature 452:352–355 Kim SG, Richter W, Ugurbil K (1997) Limitations of temporal resolution in functional MRI. Magn Reson Med 37:631–636 Kraus RH Jr, Volegov P, Matlachov A, Espy M (2008) Toward direct neural current imaging by resonant mechanisms at ultra-low ﬁeld. Neuroimage 39:310–317 Kriegeskorte N, Bandettini P (2007a) Analyzing for information, not activation, to exploit highresolution fMRI. Neuroimage 38:649–662

36

P.A. Bandettini

Kriegeskorte N, Bandettini P (2007b) Combining the tools: activation- and information-based fMRI analysis. Neuroimage 38:666–668 Kriegeskorte N, Goebel R, Bandettini P (2006) Information-based functional brain mapping. Proc Natl Acad Sci U S A 103:3863–3868 Kwong KK, Belliveau JW, Chesler DA, Goldberg IE, Weisskoff RM, Poncelet BP, Kennedy DN, Hoppel BE, Cohen MS, Turner R, Cheng HM, Brady TJ, Rosen BR (1992) Dynamic magneticresonance-imaging of human brain activity during primary sensory stimulation. Proc Natl Acad Sci U S A 89:5675–5679 Langleben DD (2008) Detection of deception with fMRI: Are we there yet? Leg Criminol Psychol 13:1–9 Langleben DD, Dattilio FM, Guthei TG (2006) True lies: delusions and lie-detection technology. J Psychiatry Law 34:351–370 Le Bihan D, Urayama SI, Aso T, Hanakawa T, Fukuyama H (2006) Direct and fast detection of neuronal activation in the human brain with diffusion MRI. Proc Natl Acad Sci USA 103:8263– 8268 Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A (2001) Neurophysiological investigation of the basis of the fMRI signal. Nature 412:150–157 Logothetis NK, Merkle H, Augath M, Trinath T, Ugurbil K (2002) Ultra high-resolution fMRI in monkeys with implanted RF coils. Neuron 35:227–242 Lu HZ, Golay X, Pekar JJ, van Zijll PCM (2003) Functional magnetic resonance Imaging based on changes in vascular space occupancy. Magn Reson Med 50:263–274 Malonek D, Grinvald A (1996) Interactions between electrical activity and cortical microcirculation revealed by imaging spectroscopy: implications for functional brain mapping. Science 272:551–554 Mandelkow H, Halder P, Brandeis D, Soellinger M, de Zanche N, Luechinger R, Boesiger P (2007) Heart beats brain: the problem of detecting alpha waves by neuronal current imaging in joint EEG-MRI experiments. Neuroimage 37:149–163 Matheiesen C, Caesar K, Akgoren N, Lauritzen M (1998) Modiﬁcation of activity-dependent increases of cerebral blood ﬂow by excitatory synaptic activity and spikes in rat cerebellar cortex. J Physiol 512:555–566 Matlachov AN, Volegov PL, Zotev VS, Espy MA, Mosher JC, Kraus RH Jr (2007) Using ultralow ﬁeld nuclear magnetic resonance for direct neural current measurements. Int Congr Ser 1300:582–585 McCarthy G, Luby M, Gore J, GoldmanRakic P (1997) Infrequent events transiently activate human prefrontal and parietal cortex as measured by functional MRI. J Neurophysiol 77:1630–1634 Menon RS, Ogawa S, Ugurbil K (1995) High-temporal-resolution studies of the human primary visual-cortex at 4 T – teasing out the oxygenation contribution in FMRI. Int J Imaging Syst Technol 6:209–215 Menon RS, Ogawa S, Strupp JP, Ugurbil K (1997) Ocular dominance in human V1 demonstrated by functional magnetic resonance imaging. J Neurophysiol 77:2780–2787 Menon RS, Luknowsky DC, Gati JS (1998) Mental chronometry using latency-resolved functional MRI. Proc Natl Acad Sci U S A 95:10902–10907 Moeller S, Van De Moortele PF, Goerke U, Adriany G, Ugurbil K (2006) Application of parallel imaging to fMRI at 7 tesla utilizing a high 1D reduction factor. Mag Reson Med 56:118–129 Montague PR, Berns GS, Cohen JD, McClure SM, Pagnoni G, Dhamala M, Wiest MC, Karpov I, King RD, Apple N, Fisher RE (2002) Hyperscanning: simultaneous fMRI during linked social interactions. Neuroimage 16:1159–1164 Moon C.-H, Fukuda M, Park S.-H, Kim S.-G (2007a) Neural interpretation of blood oxygenation level-dependent fMRI maps at submillimeter columnar resolution. J Neurosci 27(26): 6892–6902 Moon CH, Fukuda M, Park SH, Kim SG (2007b) Neural interpretation of blood oxygenation leveldependent fMRI maps at submillimeter columnar resolution. J Neurosci 27:6892–6902 Moran JM, Wig GS, Adams RB Jr, Janata P, Kelley WM (2004) Neural correlates of humor detection and appreciation. Neuroimage 21:1055–1060

Functional MRI Limitations and Aspirations

37

Murphy K, Bodurka J, Bandettini PA (2007) How long to scan? The relationship between fMRI temporal signal to noise ratio and necessary scan duration. Neuroimage 34:565–574 Muthukumaraswamy SD, Singh KD (2008) Spatiotemporal frequency tuning of BOLD and gamma band MEG responses compared in primary visual cortex. Neuroimage 40:1552–1560 Niessing J, Ebisch B, Schmidt KE, Niessing M, Singer W, Galuske RAW (2005) Hemodynamic signals correlate tightly with synchronized gamma oscillations. Science 309:948–951 Ogawa S, Lee TM (1990) Magnetic-resonance-imaging of blood-vessels at high ﬁelds – invivo and invitro measurements and image simulation. Magn Reson Med 16:9–18 Ogawa S, Lee TM, Kay AR, Tank DW (1990a) Brain magnetic-resonance-imaging with contrast dependent on blood oxygenation. Proc Natl Acad Sci U S A 87:9868–9872 Ogawa S, Lee TM, Nayak AS, Glynn P (1990b) Oxygenation-sensitive contrast in magneticresonance image of rodent brain at high magnetic-ﬁelds. Magn Reson Med 14:68–78 Ogawa S, Tank DW, Menon R, Ellermann JM, Kim SG, Merkle H, Ugurbil K (1992) Intrinsic signal changes accompanying sensory stimulation – functional brain mapping with magneticresonance-imaging. Proc Natl Acad Sci U S A 89:5951–5955 Ogawa S, Menon RS, Tank DW, Kim SG, Merkle H, Ellermann JM, Ugurbil K (1993) Functional brain mapping by blood oxygenation level-dependent contrast magnetic-resonance-imaging – a comparison of signal characteristics with a biophysical model. Biophys J 64:803–812 Ogawa S, Lee TM, Stepnoski R, Chen W, Zhuo XH, Ugurbil K (2000) An approach to probe some neural systems interaction by functional MRI at neural time scale down to milliseconds. Proc Natl Acad Sci U S A 97:11026–11031 Park TS, Lee SY (2007) Effects of neuronal magnetic ﬁelds on MRI: numerical analysis with axon and dendrite models. Neuroimage 35:531–538 Park TS, Lee SY, Park JH, Lee SY (2004) Effect of nerve cell currents on MRI images in snail ganglia. Neuroreport 15:2783–2786 Parkes LM, De Lange FP, Fries P, Toni I, Norris DG (2007) Inability to directly detect magnetic ﬁeld changes associated with neuronal activity. Magn Reson Med 57:411–416 Peissig JJ, Tarr MJ (2007) Visual object recognition: Do we know more now than we did 20 years ago? Annu Rev Psychol 58:75–96 Phillips H (2004) Private thoughts, public property. New Sci 183:38–41 Poznanski RR, Riera JJ (2006) fMRI models of dendritic and astrocytic networks. J Integr Neurosci 5:273–326 Preibisch C, Wallenhorst T, Heidemann R, Zanella FE, Lanfermann H (2008) Comparison of parallel acquisition techniques generalized autocalibrating partially parallel acquisitions (GRAPPA) and modiﬁed sensitivity encoding (mSENSE) in functional MRI (fMRI) at 3T. J Magn Reson Imaging 27:590–598 Raichle M, Snyder A (2007) A default mode of brain function: a brief history of an evolving idea. Neuroimage 37:1083–1090 Raichle ME, MacLeod AM, Snyder AZ, Powers WJ, Gusnard DA, Shulman GL (2001) A default mode of brain function. Proc Natl Acad Sci U S A 98:676–682 Rao SM, Bandettini PA, Binder JR, Bobholz JA, Hammeke TA, Stein EA, Hyde JS (1996) Relationship between ﬁnger movement rate and functional magnetic resonance signal change in human primary motor cortex. J Cereb Blood Flow Metab 16:1250–1254 Richter W, Ugurbil K, Georgopoulos A, Kim SG (1997) Time-resolved fMRI of mental rotation. Neuroreport 8:3697–3702 Saxe R, Kanwisher N (2003) People thinking about thinking people: the role of the temporoparietal junction in “theory of mind”. Neuroimage 19:1835–1842 Sereno MI, Dale AM, Reppas JB, Kwong KK, Belliveau JW, Brady TJ, Rosen BR, Tootell RBH (1995) Borders of multiple visual areas in humans revealed by functional magnetic-resonanceimaging. Science 268:889–893 Shmuel A, Yacoub E, Pfeuffer J, Van de Moortele PF, Adriany G, Hu XP, Ugurbil K (2002) Sustained negative BOLD, blood ﬂow and oxygen consumption response and its coupling to the positive response in the human brain. Neuron 36:1195–1210

38

P.A. Bandettini

Shmuel A, Yacoub E, Chaimow D, Logothetis NK, Ugurbil K (2007) Spatio-temporal point-spread function of fMRI signal in human gray matter at 7 tesla. Neuroimage 35:539–552 Singewald N (2007) Altered brain activity processing in high-anxiety rodents revealed by challenge paradigms and functional mapping. Neurosci Biobehav Rev 31:18–40 Singh M, Sungkarat W (2005) A novel approach to image neural activity directly by MRI. Proc SPIE 5746 (I), art. no. 15, 109–118 Singh KD, Barnes GR, Hillebrand A, Forde EME, Williams AL (2002) Task-related changes in cortical synchronization are spatially coincident with the hemodynamic response. Neuroimage 16:103–114 Smirnakis SM, Schmid MC, Weber B, Tolias AS, Augath M, Logothetis NK (2007) Spatial speciﬁcity of BOLD versus cerebral blood volume fMRI for mapping cortical organization. J Cereb Blood Flow Metab 27:1248–1261 Thomason ME, Foland LC, Glover GH (2007) Calibration of BOLD fMRI using breath holding reduces group variance during a cognitive task. Hum Brain Mapp 28:59–68 Triantafyllou C, Hoge RD, Krueger G, Wiggins CJ, Potthast A, Wiggins GC, Wald LL (2005) Comparison of physiological noise at 1.5 T, 3 T and 7 T and optimization of fMRI acquisition parameters. Neuroimage 26:243–250 Truong TK, Avram A, Song AW (2008) Lorentz effect imaging of ionic currents in solution. J Magn Reson 191:93–99 Turner R, Lebihan D, Moonen CTW, Despres D, Frank J (1991) Echo-planar time course MRI of cat brain oxygenation changes. Magn Reson Med 22:159–166 Viswanathan A, Freeman RD (2007) Neurometabolic coupling in cerebral cortex reﬂects synaptic more than spiking activity. Nat Neurosci 10:1308–1312 Vogeley K, Bussfeld P, Newen A, Herrmann S, Happe F, Falkai P, Maier W, Shah NJ, Fink GR, Zilles K (2001) Mind reading: neural mechanisms of theory of mind and self-perspective. Neuroimage 14:170–181 Wager TD, Smith EE (2003) Neuroimaging studies of working memory: a meta-analysis. Cogn Affect Behav Neurosci 3:255–274 Wang JJ, Aguirre GK, Kimberg DY, Roc AC, Li L, Detre JA (2003) Arterial spin labeling perfusion AM with very low task frequency. Magn Reson Med 49:796–802 Watson KK, Matthews BJ, Allman JM (2007) Brain activation during sight gags and languagedependent humor. Cereb Cortex 17:314–324 Wild B, Rodden FA, Rapp A, Erb M, Grodd W, Ruch W (2006) Humor and smiling: cortical regions selective for cognitive, affective, and volitional components. Neurology 66:887–893 Wildgruber D, Ackermann H, Kreifelts B, Ethofer T (2006) Cerebral processing of linguistic and emotional prosody: fMRI studies. Prog Brain Res 156:249–268 Williams DS, Detre JA, Leigh JS, Koretsky AP (1992) Magnetic-resonance-imaging of perfusion using spin inversion of arterial water. Proc Natl Acad Sci U S A 89:212–216 Xiong J, Fox PT, Gao JH (2003) Directly mapping magnetic ﬁeld effects of neuronal activity by magnetic resonance imaging. Hum Brain Mapp 20:41–49 Yablonskiy DA, Ackerman JJH, Raichle ME (2000a) Erratum: Coupling between changes in human brain temperature and oxidative metabolism during prolonged visual stimulation. Proc Natl Acad Sci U S A 97:9819 Yablonskiy DA, Ackerman JJH, Raichle ME (2000b) Coupling between changes in human brain temperature and oxidative metabolism during prolonged visual stimulation. Proc Natl Acad Sci o U S A 97:7603–7608 Yacoub E, Ugurbil K, Harel N (2006) Detection of orientation speciﬁc activation zones in human V1 using fMRI. Paper presented at the ISMRM 14th Scientiﬁc Meeting, Seattle Yacoub E, Shmuel A, Logothetis N, Ugurbil K (2007) Robust detection of ocular dominance columns in humans using Hahn spin echo BOLD functional MRI at 7 tesla. Neuroimage 37:1161–1177 Yurgelun-Todd DA, Ross AJ (2006) Functional magnetic resonance imaging studies in bipolar disorder. CNS Spectr 11:287–297 Zald DH (2003) The human amygdala and the emotional evaluation of sensory stimuli. Brain Res Rev 41:88–123

Studying Cognition with Positron Emission Tomography Alain Dagher

Abstract Positron emission tomography (PET) is essentially the only method for directly measuring neurotransmitter function in vivo in the human brain. While PET has been supplanted by functional magnetic resonance imaging for the purpose of brain mapping, much information about cognitive function can still be gleaned from studies of speciﬁc neurotransmitter function. We will introduce the methodology and give examples of studies involving one neurotransmitter (dopamine). Dopamine has been linked to motivation and reward, processes that belong entirely to the realm of what we refer to as thinking.

1 Introduction Positron emission tomography (PET) was the ﬁrst method used for mapping neuronal function in the living human. Initial studies measured metabolic activity, consisting of changes in regional cerebral blood ﬂow (rCBF) or glucose metabolism, on the basis of the premise that neuronal activity is necessarily accompanied by changes in these biological variables. Basically, neurons need energy, which is supplied by oxygen and glucose from the systemic vasculature. Another premise of functional brain imaging is that the brain is modular. Although this is likely an ancient concept, it was stated in the early nineteenth century by Franz Gall (1825), in his four theses, namely: moral and intellectual qualities are innate; their functioning depends upon organic supports; the brain is the organ of all faculties, tendencies, and feelings; the brain is composed of as many organs as there are faculties, tendencies, and feelings. Although the initial evidence supporting this modular arrangement of the brain came from postmortem studies such as those of A. Dagher McConnell Brain Imaging Center, McGill University, 3801 University St, Montreal, QC, Canada H3A 2B4 [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

39

40

A. Dagher

Paul Broca, in vivo brain mapping may have begun as early as 1880 in the Turin laboratory of Angelo Mosso. Mosso observed an interesting phenomenon in a patient of his, Michele Bertino, who had a skull defect over his frontal lobes as a result of a traumatic injury: whenever Bertino engaged in cognitive activity, blood would be seen to pulsate in his exposed frontal cortex. Importantly, these changes were unaccompanied by changes in systemic blood pressure or heart rate, leading Mosso to conclude that he was observing a change in focal cerebral blood ﬂow (CBF) secondary to cerebral activity (Raichle 2000). This was echoed in 1890 by Roy and Sherrington, who famously proposed that “the brain possesses an intrinsic mechanism by which its vascular supply can be varied locally in correspondence with local variations of functional activity.” Changes in rCBF related to changes in neuronal activity were eventually measured in living humans undergoing craniotomy for the treatment of epilepsy (Penﬁeld et al. 1939). The ﬁrst noninvasive measurements of rCBF were carried out using nuclear medicine techniques. Here a radioactive molecule is injected into the bloodstream, and its distribution is mapped by detector arrays surrounding the brain. Ingvar and Risberg (1967) were the ﬁrst to show increased rCBF in motor and supplementary motor areas during the execution and imagination of hand movements. Although current cutting-edge brain imaging research is directed at measuring whole brain network interactions, it is fair to say that the ﬁrst 20 years of functional brain imaging consisted in recapitulations of these experiments using ever more sophisticated technology. The recent development of functional magnetic resonance imaging (fMRI), based on the blood-oxygen-level-dependent signal, ﬁrst described by Ogawa et al. (1990), has allowed brain mapping to be performed without the injection of radioactive tracers. fMRI has several advantages over PET for brain mapping, including much greater temporal resolution (on the order of 100 ms using event-related experimental designs), greater spatial resolution, and possibly greater sensitivity. While fMRI has taken over from PET for mapping neuronal activity, PET has become the tool of choice for mapping the activity of neurotransmitter systems in the human brain.

2 Methods Used in PET 2.1 Brain Mapping Brain mapping refers to the imaging of neuronal activity using PET or fMRI. The basic premise of brain mapping is that synaptic neuronal activity leads to a proportionate increase in CBF (Raichle 1987). Typically subjects are scanned in at least two cognitive states, which may be referred to as baseline and activation. By looking for regional changes in CBF between the two states, one can detect brain areas where neuronal ﬁring is increased by the activation task. The second important principle in brain mapping has been the development of sophisticated image processing and statistical methods that allow the generation of so-called statistical parametric

Studying Cognition with Positron Emission Tomography

41

maps. Because the signal changes that result from activation are typically quite small (as low as 4%), data from multiple subjects must be combined. In order to do this, the PET and/or magnetic resonance imaging images are transformed into a standard coordinate space, based on the neuroanatomical atlas of Talairach and Tournoux (1988). This allows statistical tests to be applied at each voxel, for example, by dividing the mean change in CBF over all subjects by an estimate of the standard deviation, to yield a z or t statistical map. One can then search this 3D Gaussian statistical ﬁeld for areas where z or t values exceed a certain threshold (for example, a threshold corresponding to a P value of 0.05 corrected for multiple comparisons). The areas thus uncovered will correspond to brain regions where there was a statistically signiﬁcant change in CBF during performance of the activation task compared with the baseline. The ﬁrst method used for brain mapping was PET with the tracer [15 O]H2 O. Radiolabeled water molecules injected into the venous circulation as a bolus are distributed in the brain in a manner proportional to CBF. A single [15 O]H2 O PET scan yields a measurement of relative CBF over a period of approximately 60 s. Usually up to 12 measurements can be performed in a single session. Another PET tracer that can be used for mapping neuronal activity is [18 F]deoxyglucose, which is used to measure the neuronal metabolic rate of glucose. Like CBF, glucose metabolism is most likely an index of synaptic activity.

2.2 Neurotransmitter Imaging Although PET is being used to image a range of neurotransmitter systems, we will focus on the dopamine system, by far the most studied to date. There are three presynaptic targets for functional neuroimaging tracers within the dopamine neuron (Booij et al. 1999). FDOPA is taken up by dopamine neurons, converted to [18 F]dopamine by the enzyme dopa decarboxylase, and stored in synaptic vesicles. It was the ﬁrst marker to be used in the diagnosis of Parkinson’s disease (Garnett et al. 1984), a disease characterized by degeneration of dopamine neurons. More recently, labeled analogues of cocaine that target the synaptic dopamine transporter have been developed for PET and single photon emission computed tomography (Marek et al. 2003). Finally, [11 C]dihydrotetrabenazine, a ligand that binds to the vesicular monoamine transporter (VMAT2), is also sensitive to loss of dopamine neurons (Frey et al. 1996).

2.3 PET Imaging of Neurotransmitter Release While fMRI and [15 O]H2 O PET are used to map neuronal activity, the signal changes are probably not speciﬁc to the actions of any single neurotransmitter.

42

A. Dagher

However, it is possible to map the activity of dopamine neurons using PET. Numerous studies have shown that benzamide dopamine D2 receptor antagonists such as [11 C]raclopride are sensitive to endogenous dopamine levels. Drugs that increase synaptic dopamine lead to a reduction in [11 C]raclopride binding (Smith et al. 1997; Laruelle 2000), while pharmacologic depletion of dopamine has the opposite effect (Ginovart et al. 1997). Combined in vivo microdialysis and PET studies in primates have demonstrated that the increase in extrasynaptic dopamine is proportional to the reduction in benzamide binding (Endres et al. 1997; Laruelle et al. 1997). Following amphetamine administration, the reduction in [11 C]raclopride binding in the striatum lasts for as long as 5 h (Laruelle et al. 1997). The evidence points to receptor internalization as the mechanism underlying this phenomenon (Sun et al. 2003). When dopamine receptors bind dopamine, they undergo internalization from the cell membrane via endocytosis. While internalized they no longer bind benzamide ligands such as raclopride, possibly because of the low lipophilicity of these ligands, although other factors such as endosomal sodium concentration and pH may play a role (Laruelle 2000). This has the effect of reducing the apparent quantity of binding sites for the tracer. Using compartmental modeling, one can dissociate the effects of changes in CBF from the effects of changes in dopamine levels (Aston et al. 2000). The main disadvantages of the PET [11 C]raclopride technique are the very low temporal resolution (only a single measurement is obtained, and it likely represents the mean dopamine concentration over a 1–3-h period), and the fact that only signals from the striatum are detectable with adequate sensitivity. It is therefore not possible with this tracer to map dopamine release outside the striatum, although a new tracer from the same family may eventually overcome this problem (Mukherjee et al. 2002). This method has also been used to study the placebo effect in Parkinson’s disease patients (de la Fuente-Fernandez et al. 2001). Placebo caused a 16–21% reduction in [11 C]raclopride binding, comparable to the effect of a therapeutic dose of L-dopa. [11 C]Raclopride has also been used to visualize dopamine release from embryonic mesencephalic cell grafts in a patient transplanted 10 years earlier (Piccini et al. 1999).

3 The Neural Correlates of Thinking 3.1 Introduction Most of the neuroimaging research on cognitive function at present uses the brain mapping techniques described earlier, almost exclusively with fMRI. The unique capacity of PET is the ability to measure levels of neurotransmitters, and to measure neurotransmitter release in the living human brain. A group of neurotransmitters of particular interest are neuromodulators, such as dopamine and serotonin, which play a role in determining interindividual differences in cognitive function. In this

Studying Cognition with Positron Emission Tomography

43

section I will discuss some human neuroimaging data on the role of dopamine in cognitive and emotional function. The interest in dopamine stems from its known role in disorders such as drug addiction and schizophrenia. Understanding disordered thinking is one way of approaching the problem of the neural correlates of thinking.

3.2 Addiction as a Model To Understand Thinking Although there are many clinical features that make up the addictive syndromes, one of the most consistent and important is a loss of control over behavior. Addiction, in this sense, can be thought of as a battle between a cognitive or reﬂexive system on the one hand and an impulsive system on the other (Bechara 2005). In these models, the impulsive system, which assigns hedonic and motivational signiﬁcance to drug rewards and drug cues (previously neutral stimuli that have acquired incentive properties as a result of being paired repeatedly with the drug), is in conﬂict with a cognitive system that weighs the future consequences of a speciﬁc behavior. A key component of the impulsive system is the neurotransmitter dopamine. For over 50 years scientists have been studying the role of the dopamine system in drug addiction in animal models. This research has implicated the neurotransmitter in the processing of both natural rewards, such as food and sex, and drugs of abuse. Several notions regarding the role of dopamine have emerged. First, studies using the technique of brain stimulation reward (Milner 1991) have shown that the self-administration of electrical stimulation by rodents was dependent on brain dopamine levels. Second, numerous experiments have shown that all known drugs of abuse have the ability to release dopamine in the striatum, and most prominently in its ventral part (Di Chiara and Imperato 1988). Blockade of dopamine receptors reduces drug self-administration in animals. It also reduces the rewarding and reinforcing effects of food, which has led to the suggestion that addictive drugs act on neural circuitry that originally evolved to serve feeding behavior (Wise and Rompre 1989). Although initial conceptualizations of dopamine had it functioning as a pleasure or reward signal, these have been recently reﬁned to implicate dopamine in attention to, learning about, and motivation to obtain rewarding stimuli in the environment. Almost all the research implicating the dopamine system in addiction has been conducted in animals (typically rodents). There has been little research in human subjects until recently when the advent of functional neuroimaging has allowed us to study human neurotrasnmitter function safely and in vivo. The questions to be addressed here will be the following. Are the responses of the dopamine system to “natural” and drug rewards similar? Is the dopamine system changed by drug taking or addiction? Are there differences in the dopamine system in individuals vulnerable to addiction?

44

A. Dagher

Fig. 1 Anatomy of the striatum. Main projection sites of the dopamine system that are imaged using the tracer [11 C]raclopride

3.3 Anatomy and Physiology of the Dopamine System Dopamine neurons are located in the midbrain and project to most forebrain structures, with the highest density of projections in the striatum. The striatum is part of the basal ganglia (Fig. 1) and receives projections from the entire cerebral cortex. These projections form part of a network of corticostriatal loops, such that all of the neuronal information gets fed back to the cortex, via the globus pallidus and thalamus (Alexander et al. 1986). Each set of corticostriatal loops is thought to have a different function, based on the function of the cortical area involved. For example, a motor loop involves projections from the primary motor cortex, supplementary motor area, and lateral premotor cortex to the putamen, which return to the same cortical areas via the ventrolateral thalamus. A cognitive loop consists of projections from association and prefrontal cortex to the caudate nucleus. Finally, the limbic loop includes projections from limbic areas such as orbitofrontal cortex and amygdala to the ventral striatum, and is involved in motivation and emotion (Alexander et al. 1986). One of the key distinctions then is between ventral and dorsal striatum, the former being involved in motivation and emotion (including addiction) and the latter in cognitive and motor functions.

3.4 Dopamine Release Measured in Humans Almost all known drugs of abuse release dopamine in the human brain, and especially in the ventral striatum (Di Chiara and Imperato 1988). Dopamine release in this brain region is thought to be the crucial event in reinforcement, leading to addiction. An action is said to be reinforced if neural processes accompanying the action make it likely that the individual will repeat it. In animals, dopamine release can be measured with in vivo microdialysis; however, this method is obviously too invasive for human use. We have used PET scanning to perform similar measurements in humans. We scanned healthy subjects following a single oral dose of amphetamine or placebo (Leyton et al. 2002). We generated t maps of the statistical likelihood of

Studying Cognition with Positron Emission Tomography

45

Fig. 2 Amphetamine-induced dopamine release. Top: Statistical parametric map of dopamine release following a low dose of oral amphetamine (0.3 mg kg−1 ) in healthy volunteers demonstrating an effect in the ventral striatum. Bottom: Dopamine release correlates with drug wanting and with novelty-seeking personality score, suggesting that it may represent a vulnerability-to-addiction factor

dopamine release, and showed that amphetamine, in low doses, caused dopamine release in the ventral striatum, conﬁrming animal microdialysis studies (Fig. 2). Moreover, the quantity of amphetamine-induced dopamine release correlated with two variables: (1) the degree of “drug-wanting” experienced by subjects, as determined by a visual analogue scale (“how much would you want to have this drug again?”); (2) a measure of subjects’ personality referred to as “novelty seeking.” Novelty seeking, as assessed by the tridimensional personality questionnaire, is a measure of a subject’s impulsivity and response to novel and exciting situations, and is a risk factor for addiction (Cloninger 1994). Individuals who score high on this measure are more likely to become addicts, and more likely to relapse after abstaining from drugs. This initial result then suggests that dopaminergic responsivity is a biological risk factor for addiction. Similar results were obtained with alcohol (Boileau et al. 2003). We then wished to test the theory that addictive drugs act on brain circuitry that originally evolved to subserve natural behaviors such as feeding (Wise and Rompre 1989). This theory is based on the ﬁnding that blockade of dopamine receptors in animals reduces the reinforcing properties of food. Our group had previously shown that chocolate, in chocolate lovers, activated a network of brain regions that overlapped to a high degree with regions activated during a cocaine rush in cocaine addicts (Breiter et al. 1997; Small et al. 2001). We now wished to see if a pleasant meal resulted in release of dopamine in areas similar to those for dopamine release induced by amphetamine and alcohol.

46

A. Dagher

Surprisingly we found that food caused dopamine release in the dorsal portion of the striatum (Small et al. 2003), as opposed to ethanol and amphetamine, which targeted the ventral portion of the striatum (Fig. 2). To test whether this indicated a major difference between natural and drug rewards we then tested the effect of nicotine in smokers. The reason for this study is that our food and drug stimuli differed in other ways than the natural/drug dichotomy. For one, subjects in our drug studies were drug-na¨ıve, had no anticipation of drug effects, and had no conditioned cues related to the drugs that were administered to them. In the food study, subjects were anticipating their meal, and were exposed to conditioned cues associated with food reward (the taste, smell, and appearance of their favorite foods). These factors are relevant because both conditioned cues and reward anticipation can themselves cause dopamine release in the brain (Schultz 1998). Interestingly, we found that nicotine caused dopamine release in the dorsal striatum of habitual smokers (Barrett et al. 2004), thus resembling food more than amphetamine or ethanol. In other words, a drug, nicotine, had very similar effects to a natural reward (food) when conditioned cues (the smell and taste of tobacco, the physical presence of the cigarette) were present. This ﬁnding is consistent with a theory of drug addiction according to which the initial pharmacological effects of a stimulant drug target the ventral striatum, but, with habitual use, there is a shift in dopamine release towards the dorsal portion of the striatum (Everitt et al. 2001), a process that is thought to parallel the development of the addictive habit. The imaging research described here suggests that the intensity of the dopminergic response to drugs may be related to vulnerability to addiction (Leyton et al. 2002; Boileau et al. 2003). This is consistent with a considerable amount of work in rodents (Piazza et al. 1990) and primates (Morgan et al. 2002) linking intrinsic differences in striatal dopamine function with vulnerability to addiction. It is also consistent with the link between addiction and novelty-seeking personality, a personality type long thought to depend on dopaminergic function (Cloninger 1994). However, a question remains: What factors make the dopamine system hyperresponsive? Clearly genetics play a role: heredity is a major risk factor for addiction, and several genes have been linked to addiction, several of which, interestingly, are dopamine genes (Noble 2000). However, environmental factors also contribute. Two such causative factors are stress and poor maternal care. Children who are brought up in a stressful uncaring environment are at increased risk of a host of psychiatric disorders, including addiction. We have used our PET paradigm to measure dopamine release in response to psychosocial stress in young adults (Pruessner et al. 2004). We showed that the stress task caused dopamine release in the ventral striatum. This ﬁnding is interesting for several reasons. First, it shows that dopamine is not only released by appetitive rewarding stimuli. Second, it may provide an explanation for the well-known observation that stress can lead to relapse among abstinent drug addicts. It is plausible that dopamine release during stress causes increased drug seeking and craving, as shown in animal models (Everitt et al. 2001). We also found that the amount of dopamine released during stress correlated with maternal care; individuals with self-described low maternal care during childhood had the highest dopamine release during the stress task. Thus, one possible

Studying Cognition with Positron Emission Tomography

47

mechanism linking poor early-life experiences to addiction is hyperreactivity to stress within the mesolimbic dopamine system. Finally, a second pathway to hyperdopaminergia is sensitization. Sensitization, a phenomenon long studied in animals, refers to the increased psychomotor response to stimulant drugs following repeated exposure (Paulson and Robinson 1995). Until recently sensitization had never been demonstrated in humans, although it is suspected to play a role in addiction, schizophrenia, and other brain disorders. We measured dopamine release in drug-na¨ıve individuals after they had received a ﬁrst and a third dose of amphetamine, and demonstrated a sensitized response consisting of enhanced dopamine release in the ventral striatum (Boileau et al. 2006). When subjects were tested again 1 year later they had an even greater response (Fig. 3), now involving the dorsal portion of the striatum, consistent with the idea mentioned earlier

Fig. 3 Sensitization to amphetamine. Progressive augmentation and dorsal migration of the dopaminergic response to amphetamine in healthy subjects tested on day 1, day 21 (dose 4), and day 365 (dose 5)

48

A. Dagher

that repeated exposure to rewards is associated with a migration of response from the limbic striatum to the dorsal striatum, paralleling the development of habitual drug seeking. In a subsequent experiment we showed that the sensitized response could be attributed in large part to conditioned effects of the drug-taking environment (Boileau et al. 2007). Interestingly the degree of sensitization correlated with novelty-seeking personality score, further establishing the link between this personality variable and vulnerability to addiction. Sensitization has subsequently been demonstrated, using a similar imaging paradigm, in a subtype of Parkinson’s disease patients who develop addictive behaviors when treated with dopaminergic agents (Evans et al. 2006).

4 Conclusion The examples were restricted to a single neurotransmitter system; however, they illustrate the current usefulness of PET in studying cognitive function. While there is no doubt that fMRI is the modality of choice for studying neuronal activity in the living human brain, PET gives us the ability to measure neurotransmitter function. In particular, it is useful for measuring the effect of modulatory neurotransmitters, such as the monoamines and acetylcholine, on normal and abnormal cognition. At present, fMRI cannot measure neuroreceptor binding. Ultimately, the ability to combine the two modalities might provide a powerful tool for in vivo investigations of human brain function.

References Alexander GE, DeLong MR, Strick PL (1986) Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu Rev Neurosci 9:357–381 Aston JA, Gunn RN, Worsley KJ, Ma Y, Evans AC, Dagher A (2000) A statistical method for the analysis of positron emission tomography neuroreceptor ligand data. Neuroimage 12:245–256 Barrett SP, Boileau I, Okker J, Pihl RO, Dagher A (2004) The hedonic response to cigarette smoking is proportional to dopamine release in the human striatum as measured by positron emission tomography and [11C]raclopride. Synapse 54:65–71 Bechara A (2005) Decision making, impulse control and loss of willpower to resist drugs: a neurocognitive perspective. Nat Neurosci 8:1458–1463 Boileau I, Assaad JM, Pihl RO, Benkelfat C, Leyton M, Diksic M, Tremblay RE, Dagher A (2003) Alcohol promotes dopamine release in the human nucleus accumbens. Synapse 49:226–231 Boileau I, Dagher A, Leyton M, Gunn RN, Baker GB, Diksic M, Benkelfat C (2006) Modeling sensitization to stimulants in humans: an [11C]raclopride/positron emission tomography study in healthy men. Arch Gen Psychiatry 63:1386–1395 Boileau I, Dagher A, Leyton M, Welfeld K, Booij L, Diksic M, Benkelfat C (2007) Conditioned dopamine release in humans: a positron emission tomography [11C]raclopride study with amphetamine. J Neurosci 27:3998–4003 Booij J, Tissingh G, Winogrodzka A, van Royen EA (1999) Imaging of the dopaminergic neurotransmission system using single-photon emission tomography and positron emission tomography in patients with parkinsonism. Eur J Nucl Med 26:171–182

Studying Cognition with Positron Emission Tomography

49

Breiter HC, Gollub RL, Weisskoff RM, Kennedy DN, Makris N, Berke JD, Goodman JM, Kantor HL, Gastfriend DR, Riorden JP, Mathew RT, Rosen BR, Hyman SE (1997) Acute effects of cocaine on human brain activity and emotion. Neuron 19:591–611 Cloninger CR (1994) Temperament and personality. Curr Opin Neurobiol 4:266–273 de la Fuente-Fernandez R, Ruth TJ, Sossi V, Schulzer M, Calne DB, Stoessl AJ (2001) Expectation and dopamine release: mechanism of the placebo effect in Parkinson’s disease. Science 293:1164–1166 Di Chiara G, Imperato A (1988) Drugs abused by humans preferentially increase synaptic dopamine concentrations in the mesolimbic system of freely moving rats. Proc Natl Acad Sci USA 85:5274–5278 Endres CJ, Kolachana BS, Saunders RC, Su T, Weinberger D, Breier A, Eckelman WC, Carson RE (1997) Kinetic modeling of [11C]raclopride: combined PET-microdialysis studies. J Cereb Blood Flow Metab 17:932–942 Evans AH, Pavese N, Lawrence AD, Tai YF, Appel S, Doder M, Brooks DJ, Lees AJ, Piccini P (2006) Compulsive drug use linked to sensitized ventral striatal dopamine transmission. Ann Neurol 59:852–858 Everitt BJ, Dickinson A, Robbins TW (2001) The neuropsychological basis of addictive behaviour. Brain Res Brain Res Rev 36:129–138 Frey KA, Koeppe RA, Kilbourn MR, Vander Borght TM, Albin RL, Gilman S, Kuhl DE (1996) Presynaptic monoaminergic vesicles in Parkinson’s disease and normal aging. Ann Neurol 40:873–884 Gall FJ (1825) Sur les fonctions du cerveau et sur celles de chacune de ses parties: avec des observations sur la possibilit´e de reconnaitre les instincts, les penchans, les talens, ou les dispositions morales et intellectuelles des hommes et des animaux, par la conﬁguration de leur tˆete. Balliere, Paris Garnett ES, Nahmias C, Firnau G (1984) Central dopaminergic pathways in hemiparkinsonism examined by positron emission tomography. Can J Neurol Sci 11:174–179 Ginovart N, Farde L, Halldin C, Swahn CG (1997) Effect of reserpine-induced depletion of synaptic dopamine on [11C]raclopride binding to D2-dopamine receptors in the monkey brain. Synapse 25:321–325 Ingvar DH, Risberg J (1967) Increase of regional cerebral blood ﬂow during mental effort in normals and in patients with focal brain disorders. Exp Brain Res 3:195–211 Laruelle M (2000) Imaging synaptic neurotransmission with in vivo binding competition techniques: a critical review. J Cereb Blood Flow Metab 20:423–451 Laruelle M, Iyer RN, al-Tikriti MS, Zea-Ponce Y, Malison R, Zoghbi SS, Baldwin RM, Kung HF, Charney DS, Hoffer PB, Innis RB, Bradberry CW (1997) Microdialysis and SPECT measurements of amphetamine-induced dopamine release in nonhuman primates. Synapse 25:1–14 Leyton M, Boileau I, Benkelfat C, Diksic M, Baker G, Dagher A (2002) Amphetamine-induced increases in extracellular dopamine, drug wanting, and novelty seeking. A PET/[11C]raclopride study in healthy men. Neuropsychopharmacology 27:1027–1035 Marek K, Jennings D, Seibyl J (2003) Single-photon emission tomography and dopamine transporter imaging in Parkinson’s disease. Adv Neurol 91:183–191 Milner PM (1991) Brain-stimulation reward: a review. Can J Psychol 45:1–36 Morgan D, Grant KA, Gage HD, Mach RH, Kaplan JR, Prioleau O, Nader SH, Buchheimer N, Ehrenkaufer RL, Nader MA (2002) Social dominance in monkeys: dopamine D2 receptors and cocaine self-administration. Nat Neurosci 5:169–174 Mukherjee J, Christian BT, Dunigan KA, Shi B, Narayanan TK, Satter M, Mantil J (2002) Brain imaging of 18F-fallypride in normal volunteers: Blood analysis, distribution, test-retest studies, and preliminary assessment of sensitivity to aging effects on dopamine D-2/D-3 receptors. Synapse 46:170–188 Noble EP (2000) The DRD2 gene in psychiatric and neurological disorders and its phenotypes. Pharmacogenomics 1:309–333 Ogawa S, Lee TM, Kay AR, Tank DW (1990) Brain magnetic resonance imaging with contrast dependent on blood oxygenation. Proc Natl Acad Sci USA 87:9868–9872

50

A. Dagher

Paulson PE, Robinson TE (1995) Amphetamine-induced time-dependent sensitization of dopamine neurotransmission in the dorsal and ventral striatum: a microdialysis study in behaving rats. Synapse 19:56–65 Penﬁeld W, Von Santha K, Cipriani A (1939) Cerebral blood ﬂow during induced epileptiform seizures in animals and man. J Neurophysiol 2:257–267 Piazza PV, Deminiere JM, Maccari S, Mormede P, Le Moal M, Simon H (1990) Individual reactivity to novelty predicts probability of amphetamine self-administration. Behav Pharmacol 1:339–345 Piccini P, Brooks DJ, Bjorklund A, Gunn RN, Grasby PM, Rimoldi O, Brundin P, Hagell P, Rehncrona S, Widner H, Lindvall O (1999) Dopamine release from nigral transplants visualized in vivo in a Parkinson’s patient. Nat Neurosci 2:1137–1140 Pruessner JC, Champagne F, Meaney MJ, Dagher A (2004) Dopamine release in response to a psychological stress in humans and its relationship to early life maternal care: a positron emission tomography study using [11C]raclopride. J Neurosci 24:2825–2831 Raichle ME (1987) Circulatory and metabolic correlates of brain function in normal humans. In: Mountcastle VB (ed) Handbook of physiology, sect 1, vol 5. The nervous system. American Physiological Society, Bethesda, pp 643–674 Raichle ME (2000) A brief history of human functional brain mapping. In: Toga AW, Mazziotta JC (eds) Brain mapping: the systems. Academic, San Diego, pp 33–77 Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80:1–27 Small DM, Zatorre RJ, Dagher A, Evans AC, Jones-Gotman M (2001) Changes in brain activity related to eating chocolate: From pleasure to aversion. Brain 124:1720–1733 Small DM, Jones-Gotman M, Dagher A (2003) Feeding-induced dopamine release in dorsal striatum correlates with meal pleasantness ratings in healthy human volunteers. Neuroimage 19:1709–1715 Smith GS, Dewey SL, Brodie JD, Logan J, Vitkun SA, Simkowitz P, Schloesser R, Alexoff DA, Hurley A, Cooper T, Volkow ND (1997) Serotonergic modulation of dopamine measured with [11C]raclopride and PET in normal human subjects. Am J Psychiatry 154:490–496 Sun W, Ginovart N, Ko F, Seeman P, Kapur S (2003) In vivo evidence for dopamine-mediated internalization of D(2)-receptors after amphetamine: differential ﬁndings with [(3)H]raclopride versus [(3)H]spiperone. Mol Pharmacol 63:456–462 Talairach J, Tournoux P (1988) Co-planar stereotaxic atlas of the human brain. Thieme, Stuttgart Wise RA, Rompre PP (1989) Brain dopamine and reward. Annu Rev Psychol 40:191–225

Investigating the Neural Correlates of Percepts Using Magnetoencephalography and Magnetic Source Imaging Thomas Hartmann, Nathan Weisz, Winfried Schlee, and Thomas Elbert(¬)

Abstract Magnetoencephalography (MEG) has become an important tool for neuroscientists. The high temporal resolution and the low signal-to-noise ratio of MEG provide advantages that other neuroscientiﬁc methods do not. Owing to recent ﬁndings concerning the relationship between perception and neuronal oscillations, more attention is being drawn to the importance of MEG. This chapter provides an introduction to oscillatory brain dynamics and outlines the fundamental and recent research on this topic. It also includes an overview of the basic principles of MEG and compares MEG with other neuroscientiﬁc methods such as imaging techniques like functional magnetic resonance imaging, positron emission tomography and electroencephalography. Finally, as an example of the application of MEG in current research, a short review of our work on tinnitus is provided, including links to current research on general perception.

1 Perception in Relation to Oscillatory Brain Dynamics Imagine for a moment a barking dog. It is easy to conjure up an image of the dog in our mind and to vividly add sensory and emotional content to this image. The image is a percept reconstructed from memory and it may be so vivid that we might perceive changes in peripheral physiological response such as sweating or a racing heart beat. Elements of the underlying complex network of this cognitive process have been formed by our own unique experiences in life and are modiﬁed each time we retrieve the information and each time we re-experience similar situations. Multiple complex components of the brain are involved in this process and various pieces of sensory information related to this memory are utilized. For instance, the barking dog may be perceived acoustically at ﬁrst, i.e., processed by the auditory T. Elbert Department of Psychology, University of Konstanz, Universit¨atsstraße 10, 78464 Konstanz, Germany [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

51

52

T. Hartmann et al.

system, which decodes sounds by segmenting and integrating temporal bits from the incoming stream of sound. Depending on the memory or image being recalled, noises such as the zoom of cars passing by, which are unrelated to the focus of attention (in this case the barking dog), might be suppressed. On another level, the visual level, we might try to determine the location of the dog and to retrieve additional information by trying to visualize the dog and its whereabouts. The brain might then perform cross-modal connections between the visual percept of the dog and the auditory percept of the dog’s barking to create a whole picture, or Gestalt. But how does this happen? How does the brain code and process incoming information? How does it recognize a certain object like a dog? How does the Gestalt of a dog come into our awareness? One of the key processes related to mentally conjuring an object is one’s attention devoted to the corresponding sensory input that evokes the neural pattern underlying the Gestalt. This requires that attention be devoted to only one object at a time. However, one’s attention may switch quickly between different objects or ongoing input processes and an individual may be aware of two different objects simultaneously, such as the dog and the car passing by. In this case, how can we avoid mixing the ﬁring patterns of one certain type of movement (columns in V5) or a color (V4) of one object with those of another object? In other words, one of the fundamental problems in the neurophysiology of cortical sensory coding is understanding how local cortical activity, which occurs in clusters of neurons with similar properties (columns, barrels, areas), leads to a unique and globally coherent percept of objects. In other words, how does the brain process large amounts of information simultaneously and in such a way that individual objects are recognized and transformed into a recognizable “Gestalt”? This problem seems particularly intriguing as there is no single area in the cortex where all processing pathways converge. The problem of how a subset of sensory information is selected to form the representation of a given object, the so-called binding problem, is complicated by the fact that for normal visual processing multiple objects must be represented simultaneously and in a hierarchical structure. Thus, any mechanism designed to solve the binding problem must be able to selectively “tag” feature-selective neurons that code for one particular object and, additionally, to demarcate the responses to one object from any simultaneous responses to other objects in order to avoid the illusory conjunction of features (von der Malsburg 1981). A solution to the binding problem has been proposed by a number of researchers (Abeles 1982; von der Malsburg and Schneider 1986; von der Malsburg 1981) who suggest that neurons responding to the same object might synchronize their discharges with a temporal precision of a few milliseconds. In contrast, no synchronization should occur between cells encoding features of different objects. This concept complements and extends the classical notion of object representation by distributed neuronal assemblies (Hebb 1949). As in the Hebb model, representations are generated in a highly ﬂexible and economic manner because any neuron can, at different times, participate in a number of different assemblies. Thus, new objects can readily be encoded by new patterns of activity in the same set of neurons. In principle, each of these neurons individually needs to encode primitive object features only. As the

Investigating the Neural Correlates of Percepts

53

temporal aspects of activity become available as an additional coding dimension, the binding mechanism combines these advantages with the possibility of coactivating multiple object representations. Experimental studies with humans and animals provide support for this concept of binding by synchronization. Several studies have demonstrated that spatially separate cells within the primary visual area can synchronize their spike discharges in both anaesthetized and awake cats (Eckhorn et al. 1988; Engel et al. 1990; Gray and Singer 1989; Gray et al. 1989; Michalski et al. 1983; Ts’o et al. 1986; summarized in Pantev et al. 1994). In most cases, the recorded cells synchronize with zero phase lag, which holds even if the recording sites are separated by more than 7 mm (Engel et al. 1990; Gray and Singer 1989). It was found that synchronization over these larger distances within the visual cortex of the cat only occurs when the respective neurons engage in oscillatory ﬁring with a frequency of approximately 40 Hz, and not when neurons ﬁre more irregularly. Therefore, it has been suggested that gamma oscillations may function as a carrier for long-range synchronization (Engel et al. 1992). An important ﬁnding is that both within and across sensory areas response synchronization depends critically on the stimulus conﬁguration. It was recently demonstrated that spatially separate cells in the visual cortex of the cat show strong synchronization only if they respond to the same visual stimulus. However, if responding to two independent stimuli, the cells ﬁre in a less correlated manner or even without any ﬁxed temporal relationship (Engel et al. 1991a, b; Gray and Singer 1989). Correlated ﬁring has also been reported to occur between neurons of the primary and the secondary visual area (Eckhorn et al. 1993; Munk et al. 1993). Importantly, the stimulus dependence of neuronal interactions was conﬁrmed in awake monkeys (Kreiter and Singer 1992). Testing this hypothesis in humans, Melloni et al. (2007) found increases in long-range gamma synchronization when subjects were presented with visible words, compared with a condition of subliminal word processing. An important extension to this concept has been suggested recently (Supp et al. 2007). The authors investigated the directionality of widespread cortical networks while familiar and unfamiliar pictures were viewed. Unfamiliar pictures entailed only a small number of unilateral connections. In contrast, familiar pictures entailed a widespread network of reciprocal (feed-forward and feed-backward) connections. These observations support the hypothesis that correlated ﬁring between remote brain areas could provide a dynamic binding mechanism which permits the formation of assemblies in a ﬂexible manner. When we return to the seemingly “simple” example of a barking dog, we realize that multiple pieces of information need to be simultaneously analyzed by many specialized subsystems. Even though neuronal synchrony as a crucial mechanism in the formation of object-related cell assemblies was theoretically recognized by the early 1980s, it long awaited empirical validation. Recording from area 17, Gray and Singer (1989) observed synchronous ﬁring of groups of neurons within a cortical column when presented with an optimal stimulus (slowly moving bars). The rate of simultaneous discharge lasting a few hundred milliseconds was periodic at approximately 30–60 Hz, which falls into the so-called gamma frequency range (the boundaries of which are vaguely deﬁned but usually fall within 30–100 Hz).

54

T. Hartmann et al.

Fig. 1 Top: The multiunit spike recordings and the local ﬁeld potentials (LFP) at the same recording sites. The vertical dashed line shows the stimulus onset. Bottom: Expanded plot of the 200 ms marked in the top plot showing phase locking between multiunit spike recordings and the LFP. (Adapted from Siegel and K¨onig 2003.)

Importantly, local ﬁeld potentials (LFPs) recorded with the same electrodes show an oscillatory modulation at the same rate, where a distinct phase of an oscillatory cycle coincides with the discharges (Fig. 1). This was later conﬁrmed in several other studies (Fries et al. 2001; Gruber et al. 1999; Keil et al. 1999; M¨uller et al. 1996; Pulverm¨uller et al. 1999; Siegel and K¨onig 2003) and is of great importance in understanding the multitude of perception-related gamma-band results reported in magnetoencephalography (MEG) and electroencephalography (EEG) literature (Keil et al. 2001). If such ﬂexible synchrony between cell assemblies actually existed in the brain it might be possible to track it using MEG-based technology, as outlined in more detail later in this chapter. In their 1995 review, Singer and Gray (1995) differentiated ﬁve spatial scales at which synchronization putatively occurs: (1) (2) (3) (4) (5)

Same cortical column Different cortical columns Different cortical areas Two hemispheres Different sensory and motor modalities

With the use of EEG and MEG, macroscopic forms of synchronization can be studied, corresponding approximately to scales 2–5 from the above list. Regarding scale 2, of course a sufﬁcient number of cortical columns must be involved because more than 10,000 neurons have to synchronize their activity so that the LFP is strong enough to generate an externally recordable signal. Scale 2 would be what EEG/MEG researchers call “local” synchrony (even though from an invasive viewpoint this would already be quite distant), with the single-column level being inaccessible by their methods. From the list it can be furthermore taken that

Investigating the Neural Correlates of Percepts

55

synchrony exceeds the “local” level to include so-called long-range connections (scales 3–5). Indeed such long-range synchronies, putatively mediating integration of information from different brain regions, have been experimentally shown to exist, measured by intracranial and also noninvasive approaches such as EEG and MEG (Melloni et al. 2007; reviewed in Varela et al. 2001). Just recently, it was suggested that synchronization between neuronal cell assemblies not only binds the activity of neuronal cell assemblies but also leads to spike-time-dependent plasticity in the cortex and thus governs long-term effects on cognitive functions (Womelsdorf et al. 2007). This being said, in order to gain a relatively “complete” picture of electrophysiological processes, the strengths from intracranial EEG and noninvasive EEG/MEG must be combined. Whereas intracranial recordings can score out the details regarding spatial accuracy and also the relation between oscillatory activity and discharges, MEG and EEG provide the “big picture.” In this chapter we will concentrate on the MEG method. We will compare it with other established neuropsychological procedures like functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) to show the fundamental differences between them. Finally, we will exemplify the utility of MEG in investigating oscillatory brain activity with an example from our own research.

2 Magnetoencephalography 2.1 Basics of MEG The excited portion of the dendritic tree represents a local source of current. The intracellular current ﬂow produces a magnetic ﬁeld that can be measured as magnetoencephalographic signals. At different locations the current penetrates through the cell membrane such that the circuit can be closed over the volume conductor i.e., by current pathways through extracellular body tissue. The bioelectric potentials that originate from the volume currents are recorded as an electroencephalogram, which refers to the voltage derived from two electrodes attached to the surface of the scalp. In a homogenous volume conductor, the magnetic ﬁelds produced by volume currents sum up to zero. As body tissue is not homogenous, there may be some contribution from volume currents, referred to as secondary sources. Usually such contributions are, however, small and can be neglected (Elbert 1998). The magnetic ﬁelds produced by intracellular currents ﬂowing in neighboring dendritic trees of pyramidal neurons towards the soma mostly have the same orientation and therefore the sum is of a measurable size. If enough neighboring cells show this synchronized behavior (about 20,000–50,000), the emitted ﬁeld is strong enough to be measured by MEG. To measure the biomagnetic ﬁelds, sensors are needed to detect ﬁelds as small as femtotesla. MEG uses so-called superconducting quantum interference device (SQUIDs) to detect the current that is induced by the magnetic ﬁeld and convert it

56

T. Hartmann et al.

to voltage. The SQUIDs are located inside a larger apparatus that encompasses the skull during measurement. This helmet-like apparatus is constantly cooled by liquid helium to maintain the superconductor effect. To decrease the effects of urban magnetic activity, such as from trafﬁc or other large magnetic ﬁelds, the entire MEG instrument is installed inside a magnetically shielded room. Furthermore, additional sensors are installed within the outer layer of the helmet in order to measure remaining noise that can later be subtracted from the measurements of the internal sensors which are measuring close to the skull of the subject. One serious limitation of MEG and EEG is that the localization of the neural generators cannot be directly and unequivocally derived from the measured signal. Because the recorded data is a 2D projection of activity that has three spatial dimensions, source activity has to be properly modeled. Source modeling faces similar issues as the visual system that has to derive coherent objects and their location in three dimensions is based on a 2D retinotopic representation. Generally there is no unique solution to this problem, as proven by Helmholtz. But the widely held assumption that the lack of a reliable method of source localization prevents one from making conﬁdent inferences about the underlying neural generators cannot be supported. In the cases of both MEG and EEG, additional information, e.g., derived from neuroanatomical knowledge, can be used to constrain the solution space, making it possible to deduce the best possible solution to the problem.

2.2 Advantages of MEG over Imaging Methods From the initial enthusiasm that resulted when modern neuroimaging techniques like fMRI and PET were developed, one would have though that these techniques would soon replace EEG/MEG or that electrophysiological methods would have to be combined with neuroimaging methods. This was mainly owing to the spatial resolution provided by neuroimaging methods, especially in deep regions of the brain. However, in recent years EEG and MEG have witnessed a true renaissance stemming from three developments: (1) the inherent limits regarding the temporal resolution of neuroimaging methods are more than 1 s, whereas electrophysiological methods reﬂect neuronal activity in real time (millisecond range); (2) the widespread availability of high-density EEG/MEG systems, which combined with advances in localization techniques (see later) have improved the spatial resolution of noninvasive electrophysiological methods; (3) even though there appear to be some correlations between signals recorded by fMRI and LFPs (Logothetis 2007), it is far from clear what aspects related to neuronal activity may actually modulate hemodynamic changes (Burke and B¨uhrle 2006). In contrast to MEG, PET and single photon emission computed tomography both require the use of radioactive substances. Compared with fMRI, MEG has the advantage that no magnetic ﬁeld is emitted by the machine; therefore, there is no risk of harm to the subjects or staff from metal parts that are brought close to the machine.

Investigating the Neural Correlates of Percepts

57

Instead of being offered up as an alternative to one another, or dismissing the usability of a speciﬁc technique such as MEG or EEG, the various neuroimaging methods should not be seen as mutually exclusive; rather, they should be seen as complementary. For example, fMRI and PET are of great value in clinical settings and have been shown to be very useful with patients who have suffered strokes. In these cases, fMRI and PET can be used to visualize nonfunctional areas of the brain and to identify cortical areas that can be rehabilitated through speciﬁc intervention.

2.3 Comparison of MEG and EEG While theory suggests that similar physiological processes underlie EEG and MEG, the signals generally provide different information when used in real measurements and therefore are ideally combined to extract maximal information. This may be counterintuitive since the neuromagnetic or MEG signals can be similar in appearance to those of EEG. The EEG electrodes cover only a relatively small area compared with the distance between two electrodes and, thus, EEG is vulnerable to spatial aliasing in which physiologically distinct signals become indistinguishable in the measurement. For shallow sources that have the highest spatial frequencies, 100 or more electrodes are needed (when equally spaced across the head’s surface) in order to avoid spatial aliasing (Jungh¨ofer et al. 1997). When comparing highresolution (approximately 128 electrodes) EEG and MEG, the following differences can be noted: • Improper ﬁxation or location of EEG electrodes produces artifacts or errors in the source estimation (e.g., distortion of the interpolated surface potential and consequently erroneous “ghost sources”). • Artifacts are more common in EEG than in MEG as EEG may be affected by movement of electrodes, electrode drift and volume-conducted electrocardiography. Similarly, ocular artifacts are also more remarkable for EEG than for MEG. But the greatest difference stems from the selectivity of MEG with respect to the orientation of sources. As mentioned, the electroencephalogram results from the extracellular volume currents triggered mainly by postsynaptic potentials. The magnetoencephalogram, in contrast, arises from the intracellular branch of this process, i.e., from the currents that ﬂow within the dendrite to the soma. Thereby, MEG is mainly sensitive to currents ﬂowing tangentially to the surface of the scalp and to a lesser degree – about 10% – to radial sources. As a consequence EEG and MEG are affected differently by averaging: If sources vary across trials and appear in different cerebral regions from trial to trial their impact on the event-related brain responses will be suppressed by averaging. Thereby, this “biological noise” is more strongly reduced for tangential sources than for radial ones (as tangential sources in opposing walls of a sulcus may partially cancel each other out, leaving only the radially directed currents in the average). Sources in the primary and secondary sensory projection areas, such as Brodmann areas 3b (somatosensory), 41/42 (auditory) or

58

T. Hartmann et al.

17 (visual), are primarily tangentially oriented and are consistently evoked in each trial. Consequently, for such sources activated early in the information processing, the signal-to-noise ratio is considerably higher for MEG measurements than for EEG measurements. A high-resolution EEG system costs about A C200,000. An MEG system, which starts at about A C1,000,000, is not only considerably more expensive to purchase, but also requires around 10 l of liquid helium per day for operation, which totals about another A C30,000 per year. The preparation for EEG, on the other hand, is more labor-intensive and, owing to a greater sensitivity to artifacts, requires larger sample sizes. Another advantage of MEG is the fact that the magnetic ﬁelds penetrate tissue mostly undistorted, whereas the volume current of EEG must penetrate the cerebrospinal ﬂuid, the meninges, the skull and the skin in a correspondingly complicated spatial pattern. Moreover, in order to realize low impedances between the skin and the electrodes, conductive agents such as gels or creams are needed. These agents might lead to a blurring of signals if two or more adjacent electrodes connect to each other through the conductive agent. This is particularly true with EEG systems that utilize a large number of electrodes. The magnetic ﬁeld measured by MEG passes through the outer layers of the head almost unaffected. This is also an advantage for source modeling as underlying models can be much simpler than with EEG (indeed EEG source solutions can be very sensitive to head-model misspeciﬁcations (Plis et al. 2007). In addition, there is no need for conductive agents that might blur the spatial information. Furthermore, as we have already explained, MEG and EEG measure different properties. While EEG measures the volume conduction, meaning the current that counterbalances the internal current ﬂow of many different neurons, MEG is able to directly measure the intracellular currents. As volume conduction spreads out across the surface, and is inﬂuenced by other electric signals that may lie at some distance, the currents can be greatly disturbed. The intracellular current is immune to these inﬂuences and does not blur at the surface and/or in the brain, leading to a higher spatial accuracy. Another advantage further increasing the signal-to-noise ratio stems from (1) the columnar organization and folded layout of the cortex and (2) the fact that MEG is almost blind to radial sources. Two thirds of the cortex lie in sulci, which means that the chance of the desired signal being oriented tangentially is higher than that of the desired signal being exclusively radially oriented. This can be regarded as a kind of spatial ﬁlter, leading to more focal effects. In contrast to EEG, which records this signal, MEG is blind to this part of the signal, thus enhancing the signal-to-noise ratio. The remaining sources of noise are mostly tangentially oriented and thus lie in the walls of the sulci. As each potential noise source has another noise source at the opposite wall of the sulci, these two most likely cancel each other out. These enhancements in signal quality make it possible to detect signals with MEG that would not be detectable or that would be difﬁcult to separate from other parallel processes using EEG.

Investigating the Neural Correlates of Percepts

59

3 Unraveling the Neuronal Correlates of Tinnitus with MEG In this section we present, as an example, a series of studies in which EEG and MEG were used to investigate abnormalities of the cortical activity in chronic tinnitus subjects. Since subjective tinnitus is described as a conscious perception of a sound in the absence of a physical sound source, research on this topic has general implications regarding neuronal activity underlying percepts. The perceived sound is typically described by the patients as a tone, a hissing, a roaring noise or some combination of these sounds. Transient tinnitus is quite common in the general population and typically lasts a few seconds to a few hours or even a few days. However, a diagnosis of chronic tinnitus is made when the tinnitus lasts uninterrupted for more than 3 months. About 5–15% of the population in Western societies (Heller 2003) report chronic tinnitus and in 1–3% the tinnitus affects the quality of life. This can occur in the form of sleep disturbances, impaired ability to concentrate at work, difﬁculties in social interactions and psychiatric distress (Dobie 2003). As tinnitus is a percept without a physical stimulus, the question arises whether it is possible to ﬁnd changes in the neuronal oscillations which might reﬂect synchronized cortical activity as described at the beginning of the chapter. Of the ﬁve spatial scales of synchronization mentioned previously in this chapter, scales 2–5 are most relevant for MEG-based research and for which functional neuroimaging techniques are agnostic because of their low temporal resolution. We believe that a deafferentation of auditory brain regions is necessary in order for tinnitus to develop (Eggermont and Roberts 2004; Saunders 2007; Weisz et al. 2006). However, we also know that deafferentation is not a sufﬁcient condition for the development of tinnitus. This is owing to the fact that not all patients with a profound hearing loss (and thus deafferantation of the auditory system) also suffer from tinnitus (K¨onig et al. 2006). Furthermore, treatment approaches aiming at altering the changed tonotopy have proven to be efﬁcient in reducing the perception of tinnitus (Dohrmann et al. 2007a, b; Flor et al. 2004). As with the barking dog, the conscious perception of the tinnitus sound is most likely associated with a long-range interaction of remote brain areas. We think that the critical condition for evoking such a widespread network is a reduction of cortical inhibition (Weisz et al. 2007a). Such a loss of inhibition would lead to spontaneously synchronized brain activity within the auditory cortex as well as between brain regions relevant for the processing of attentional and emotional aspects of tinnitus. When we speak of neuronal oscillations we must be more speciﬁc and will classify them according to the scale outlined at the beginning of this chapter (Singer and Gray 1995). We ﬁrst examined whether there was altered local synchrony (scale 2 on the Singer and Gray scale) by acquiring a resting-state MEG measurement where subjects were asked to lie still with their eyes open. The data were then analyzed in the frequency domain and revealed an enhancement in lower frequencies (below 4 Hz) accompanied by a decrease of energy in the alpha band (8–12 Hz) (Weisz et al. 2005). Both changes were located in temporal regions that are relevant for auditory processing. The extent of these changes showed a high correlation with decrease

60

T. Hartmann et al.

in tinnitus distress. The functional importance of the frequency bands is supported by an EEG neurofeedback study showing a strong relationship between the extent patients were able to normalize their spontaneous activity spectra and the reduction of tinnitus intensity. In this study there were even two cases in which the tinnitus sensation completely disappeared (Dohrmann et al. 2007b). In Sect. 1, we mentioned that perception is accompanied by local and global synchronization of ﬁring expressed by the gamma-band rhythm of the LFP (more than 30 Hz). It thus seems logical to investigate whether tinnitus, being a phantom percept, leads to changes in the gamma oscillations (Weisz et al. 2007b). Our study examining this effect yielded three important results: (1) In subjects with tinnitus, the overall gamma-band activity was increased. (2) Individuals with unilateral tinnitus showed more gamma-band activity on the contralateral side to the tinnitus than on the ipsilateral side. (3) Particularly in control subjects we found a strong correlation in the time course between slow-wave and gamma activity. These results are of great relevance for the understanding of tinnitus as we were able to show alterations of local synchronization and thus an objective measurement for an otherwise subjective percept. Generally these ﬁndings underline the importance of gamma-band activity in the generation of conscious percepts. We then moved to a more global approach to assess the connectivity between different cortical areas in tinnitus subjects. Schlee et al. (2007) stimulated tinnitus and control subjects with 40-Hz amplitude-modulated tones. The highest one was close to the tinnitus frequency, while the other two were 1.1 and 2.2 octaves lower. While every stimulus evokes a transient reaction that lasts for about 500 ms in EEG and MEG, amplitude-modulated or frequency-modulated stimuli evoke responses at the modulation frequency that last until stimulus offset. While other studies have only dealt with the amplitude and thus local synchronization, Schlee et al. (2007) examined whether changes in the global synchronization could be found. For this purpose they applied a source montage on MEG data consisting of eight ﬁxed regional sources each at a prominent location of the cortex (anterior cingulate cortex, posterior cingulate cortex, left and right frontal, temporal and parietal) and calculated the phase coherence between these sources for the 40-Hz response. Phase coherence is a measurement to assess whether two sources are synchronously activated in each trial. It is calculated by subtracting the phases of source A from the phases of source B for each trial. If the variance of these differences is low, it can be assumed that these two sources are synchronously active and form a functionally related network. The phase coherence of each individual was then correlated with the individual’s tinnitus distress as assessed with the German version of the Tinnitus Questionnaire (G¨obel und Hiller). This revealed a strong positive correlation between tinnitus intrusiveness and phase coherence between the right parietal and anterior cingulate regions (Fig. 2). The inverse was found for the correlation between tinnitus intrusiveness and the connection of right frontal and anterior cingulate regions. It is important to note that these effects were only present for the stimulus whose frequency was close to the tinnitus tone reported by the subjects.

Investigating the Neural Correlates of Percepts

61

Tinnitus Tone r = 0.76

10 8 6 4 2

0.05 0.10 0.15 0.20 0.25 0.30 0.35

10 8

r = −0.65

6 4 2 0.10

0.20

0.30

0.40

phase locking Fig. 2 Interregional connectivities with an association between tinnitus intrusiveness and phase synchronization. The top plot shows the scatterplot of the interregional connectivity between the right parietal and the anterior cingulate cortex across all stimulation conditions. Subjective ratings were positively correlated with the interregional phase synchronization when stimulated with the tinnitus tone. The bottom plot depicts the connectivity between right frontal and anterior cingulate cortex. The correlation between tinnitus intrusiveness and phase synchrony was negative. (Adapted from Schlee et al. unpublished results.)

Interestingly, these ﬁndings are in accordance with new results made with an advanced magnetic resonance tomography based imaging technique called diffusion tensor imaging. Lee et al. (2007) were able to show differences in the connections between temporal and frontal regions between tinnitus and control subjects. They also linked their ﬁndings to other functional imaging studies that were able to show contributions of emotional and memory systems, both being located in frontal regions. These ﬁndings also agree with those reported here. Although these ﬁndings are of great importance for understanding tinnitus, it is also possible to link these ﬁndings to research regarding general perception. In a recent article by our research workgroup, a model integrating current knowledge

62

T. Hartmann et al.

about oscillatory brain activity (Miller 2007) and abnormalities in tinnitus patients was proposed (Weisz et al. 2007a). The effect that perception of any kind leads to a decrease of energy in the alpha band and an increase in the gamma band is in accordance with ﬁndings in tinnitus patients who seem to show a chronic increase in these frequency bands. Although we are still far from a deep understanding of the mechanisms behind tinnitus, we have shown here that MEG has provided insights that would not have otherwise been impossible to attain. These insights will give rise to new therapeutic approaches that might ultimately lead to effective treatment modalities.

4 Summary Neuronal oscillations have become a valuable and important tool for neuroscientists to research neuronal activity which is too fast to be measured using methods that depend on blood ﬂow or blood oxygenation. Tinnitus research is one of the best examples of the usefulness of MEG as there is great potential for new and important insights that will increase our future understanding of the condition and which may result in viable treatment options. MEG, although more expensive than EEG, provides scientists with more accurate measurements – in the time and the spatial domain – and an enhanced signal-to-noise ratio, thus providing even greater insights into our understanding and knowledge of these topics.

References Abeles M (1982) Role of the cortical neuron: integrator or coincidence detector? Isr J Med Sci 18:83–92 Burke M, B¨uhrle C (2006) BOLD response during uncoupling of neuronal activity and CBF. Neuroimage 32:1–8 Dobie RA (2003) Depression and tinnitus. Otolaryngol Clin North Am 36:383–388 Dohrmann K, Elbert T, Schlee W, Weisz N (2007a) Tuning the tinnitus percept by modiﬁcation of synchronous brain activity. Restor Neurol Neurosci 25:371–378 Dohrmann K, Weisz N, Schlee W, Hartmann T, Elbert T (2007b) Neurofeedback for treating tinnitus. Prog Brain Res 166:473–485 Eckhorn R, Bauer R, Jordan W, Brosch M, Kruse W, Munk M, Reitboeck HJ (1988) Coherent oscillations: a mechanism of feature linking in the visual cortex? Multiple electrode and correlation analyses in the cat. Biol Cybern 60:121–130 Eckhorn R, Frien A, Bauer R, Woelbern T, Kehr H (1993) High frequency (60-90 Hz) oscillations in primary visual cortex of awake monkey. Neuroreport 4:243–246 Eggermont JJ, Roberts LE (2004) The neuroscience of tinnitus. Trends Neurosci 27:676–682 Elbert T (1998) Neuromagnetism. In: Andr¨a W, Nowak H (eds) Magnetism in medicine. Wiley, New York, pp 190–262 Engel AK, K¨onig P, Gray CM, Singer W (1990) Stimulus-dependent neuronal oscillations in cat visual cortex: inter-columnar interaction as determined by cross-correlation analysis. Eur J Neurosci 2:588–606

Investigating the Neural Correlates of Percepts

63

Engel AK, Kreiter AK, K¨onig P, Singer W (1991a) Synchronization of oscillatory neuronal responses between striate and extrastriate visual cortical areas of the cat. Proc Natl Acad Sci USA 88:6048–6052 Engel AK, K¨onig P, Singer W (1991b) Direct physiological evidence for scene segmentation by temporal coding. Proc Natl Acad Sci USA 88:9136–9140 Engel AK, K¨onig P, Kreiter AK, Schillen TB, Singer W (1992) Temporal coding in the visual cortex: new vistas on integration in the nervous system. Trends Neurosci 15:218–226 Flor H, Hoffmann D, Struve M, Diesch E (2004) Auditory discrimination training for the treatment of tinnitus. Appl Psychophysiol Biofeedback 29:113–120 Fries P, Reynolds JH, Rorie AE, Desimone R (2001) Modulation of oscillatory neuronal synchronization by selective visual attention. Science 291:1560–1563 Gray CM, Singer W (1989) Stimulus-speciﬁc neuronal oscillations in orientation columns of cat visual cortex. Proc Natl Acad Sci USA 86:1698–1702 Gray CM, K¨onig P, Engel AK, Singer W (1989) Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reﬂects global stimulus properties. Nature 338:334–337 Gruber T, M¨uller MM, Keil A, Elbert T (1999) Selective visual-spatial attention alters induced gamma band responses in the human EEG. Clin Neurophysiol 110:2074–2085 Hebb DO (1949) The organization of behavior: a neuropsychological theory. Wiley, New York Heller AJ (2003) Classiﬁcation and epidemiology of tinnitus. Otolaryngol Clin North Am 36:239–248 Jungh¨ofer M, Elbert T, Leiderer P, Berg P, Rockstroh B (1997) Mapping EEG-potentials on the surface of the brain: a strategy for uncovering cortical sources. Brain Topogr 9:203–217 Keil A, M¨uller MM, Ray WJ, Gruber T, Elbert T (1999) Human gamma band activity and perception of a gestalt. J Neurosci 19:7152–7161 Keil A, Gruber T, M¨uller MM (2001) Functional correlates of macroscopic high-frequency brain activity in the human visual system. Neurosci Biobehav Rev 25:527–534 Kreiter AK, Singer W (1992) Oscillatory neuronal responses in the visual cortex of the awake macaque monkey. Eur J Neurosci 4:369–375 K¨onig O, Schaette R, Kempter R, Gross M (2006) Course of hearing loss and occurrence of tinnitus. Hear Res 221:59–64 Lee Y, Bae S, Lee S, Lee J, Lee K, Kim M, Kim Y, Baik S, Woo S, Chang Y (2007) Evaluation of white matter structures in patients with tinnitus using diffusion tensor imaging. J Clin Neurosci 14:515–519 Logothetis NK (2007) The ins and outs of fMRI signals. Nat Neurosci 10:1230–1232 Melloni L, Molina C, Pena M, Torres D, Singer W, Rodriguez E (2007) Synchronization of neural activity across cortical areas correlates with conscious perception. J Neurosci 27:2858–2865 Michalski A, Kossut M, Turlejski K, Chmielowska J (1983) Responses of area 17 neurons in cats binocularly deprived by rearing in hoods. Acta Neurobiol Exp (Wars) 43:263–272 Miller R (2007) Theory of the normal waking EEG: from single neurones to waveforms in the alpha, beta and gamma frequency ranges. Int J Psychophysiol 64:18–23 M¨uller MM, Bosch J, Elbert T, Kreiter A, Sosa MV, Sosa PV, Rockstroh B (1996) Visually induced gamma-band responses in human electroencephalographic activity – a link to animal studies. Exp Brain Res 112:96–102 Munk MHJ, Nowak LG, Bullier J (1993) Spatio-temporal response properties and interactions of neurons in areas V1 and V2 of the monkey. Abstr Soc Neurosci 19:179.3 Pantev C, Eulitz C, Elbert T, Hoke M (1994) The auditory evoked sustained ﬁeld: origin and frequency dependence. Electroencephalogr Clin Neurophysiol 90:82–90 Plis SM, George JS, Jun SC, Ranken DM, Volegov PL, Schmidt DM (2007) Probabilistic forward model for electroencephalography source analysis. Phys Med Biol 52:5309–5327 Pulverm¨uller F, Keil A, Elbert T (1999) High-frequency brain activity: perception or active memory? Trends Cogn Sci 3:250–252 Saunders JC (2007) The role of central nervous system plasticity in tinnitus. J Commun Disord 40:313–334

64

T. Hartmann et al.

Schlee W, Weisz N, Dohrmann K, Hartmann T, Elbert T (2007) Unravelling the tinnitus distress network using single trial auditory steady-state responses. Int Congr Ser 1300:73–76 Siegel M, K¨onig P (2003) A functional gamma-band deﬁned by stimulus-dependent synchronization in area 18 of awake behaving cats. J Neurosci 23:4251–4260 Singer W, Gray CM (1995) Visual feature integration and the temporal correlation hypothesis. Annu Rev Neurosci 18:555–586 Supp GG, Schl¨ogl A, Trujillo-Barreto N, M¨uller MM, Gruber T (2007) Directed cortical information ﬂow during human object recognition: analyzing induced EEG gamma-band responses in brain’s source space. PLoS One 2:e684 Ts’o DY, Gilbert CD, Wiesel TN (1986) Relationships between horizontal interactions and functional architecture in cat striate cortex as revealed by cross-correlation analysis. J Neurosci 6:1160–1170 Varela F, Lachaux JP, Rodriguez E, Martinerie J (2001) The brainweb: phase synchronization and large-scale integration. Nat Rev Neurosci 2:229–239 von der Malsburg C (1981) The correlation theory of brain function. p 81 von der Malsburg C, Schneider W (1986) A neural cocktail-party processor. Biol Cybern 54:29–40 Weisz N, Moratti S, Meinzer M, Dohrmann K, Elbert T (2005) Tinnitus perception and distress is related to abnormal spontaneous brain activity as measured by magnetoencephalography. PLoS Med 2(6):e153 Weisz N, Hartmann T, Dohrmann K, Schlee W, Norena A (2006) High-frequency tinnitus without hearing loss does not mean absence of deafferentation. Hear Res 222:108–114 Weisz N, Dohrmann K, Elbert T (2007a) The relevance of spontaneous activity for the coding of the tinnitus sensation. Prog Brain Res 166:61–70 Weisz N, M¨uller S, Schlee W, Dohrmann K, Hartmann T, Elbert T (2007b) The neural code of auditory phantom perception. J Neurosci 27:1479–1484 Womelsdorf T, Schoffelen J, Oostenveld R, Singer W, Desimone R, Engel AK, Fries P (2007) Modulation of neuronal interactions through neuronal synchronization. Science 316: 1609–1612

EEG and Thinking ¨ Michael Ollinger

Abstract What goes on in the brain when we think? How can we solve a complex problem? How can we pursue an idea and ﬁnally reach a desired goal? Although a lot of neuroscience studies have extended our knowledge of the ongoing processes when our brain, for example, recalls words, discriminates coloured stimuli or detects deviants in a given display, we still know little about the ongoing dynamics when we think. What processes are necessary to compare two objects, to solve and understand a categorical syllogism, to infer the potential cause of an observed effect, or what happens when we see a Gestalt in apparently meaningless information? William James in 1890 proposed the idea that thinking is a constant ongoing stream of thoughts. In this chapter we attempt to give a brief overview of the notion of how to investigate the stream of thoughts by means of electroencephalography (EEG). First, we provide a short introduction to the technique. Then we address the notion of synchronization and describe how synchronization (binding) might help to identify the basic atoms of thinking, representing the elementary building blocks that form the stream of complex thoughts (molecules, objects). Moreover, we demonstrate how EEG can help us to understand basic thinking operations, like categorization, and make it possible to come up with new and reﬁned cognitive models. That should help us to get a clearer picture of the question: What processes and dynamics go on in our brain when we think? We hope to show that despite the existing predominance of functional magnetic resonance imaging results concerning the current debate on the cognitive architecture of our brain, EEG may provide a more appropriate and powerful tool for the understanding of the stream of thoughts.

¨ M. Ollinger Parmenides Foundation, Parmenides Center for the Study of Thinking, Kardinal Faulhaber Str. 14a, 80333 M¨unchen, Germany [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

65

66

¨ M. Ollinger

1 Introduction In the last few years the cognitive sciences have had a growing impact on daily life. There is hardly a newspaper or science magazine that does not show coloured brain images. The colours code the brain activation that occurs when people solve problems, have insights, tell lies, or simply do nothing. People ﬁnd these pictures appealing because they reduce the complexity of the brain to an understandable format. Of course brain imaging techniques have given us a number of new insights, including how the brain is organized, and possibly how it works. Sometimes we even have the impression that we can observe a person’s thinking online. Although functional magnetic resonance imaging (fMRI) has a lot of advantages, it also has some weaknesses. fMRI provides a high spatial resolution but it has only a very poor temporal resolution. During an fMRI scan, people lie in a scanner and the slightest movement of the head is detrimental to the quality of the data. Moreover, the scanner set-up is not applicable to young infants, because they need to be sedated to reduce movement. Furthermore, the purchase and running of a scanner is fairly expensive. Another much older but far-less-hyped technique (Berger 1929) is electroencephalography (EEG). EEG has contributed a great deal to our knowledge of fundamental brain mechanisms. The methodological rationale of the EEG technique is that electrodes distributed across the surface of the scalp record the electrical activation and changes of the brain. EEG has a very good temporal resolution (milliseconds) but a poor spatial resolution. Owing to increasingly reﬁned methods of analysis, to new and more easily applicable cap systems and devices, and to the growing computational capacity of modern computers, there has been a rebirth of EEG in recent years. EEG is used to answer urgent questions such as what are the ongoing processes when humans categorize, solve problems, make decisions. In other words, how do humans think? EEG can also be applied to young infants (DeHaan 2007), and is therefore the perfect psychophysiological method to investigate developmental changes in the cognitive systems of infants. This chapter is intended to give a brief overview of the latest EEG techniques and ﬁndings related to higher cognitive processes. We try to select important ﬁndings and embed them in a theoretical framework of complex thinking. Thinking, in our view, requires mental content. Mental content is activated by external events; for example a wooden plate with four legs is usually recognized as a table. At a table we can eat, work, play games, and so on. But how are the different features that make up our concept of ‘table’ grouped together? This example illustrates the famous binding problem. We will summarize a few EEG experiments that deal with this question. Strongly linked to the binding problem is the idea of categorization. That is, what are the neural mechanisms that differentiate between a table and a dog, or a dining table and a writing table? One section is dedicated to the topic of EEG and categorization.

EEG and Thinking

67

2 EEG Basics This section provides a brief introduction to the basic principles of EEG recording. Of course, we cannot provide an exhaustive overview of the EEG technique and the various algorithms and approaches to analysis. There are a number of wellwritten introductions covering this ﬁeld (Niedermayer and Silva 1993; Coles and Rugg 1997; Handy 2005; Luck 2005). However, we will introduce the basic electrochemical principles responsible for the EEG signal. We will also detail how the signal is recorded, and what technical and methodological approaches are used to analyse the EEG signal, as this is necessary for our further considerations.

2.1 The Signal When neurons are activated, electrochemical changes occur. The electrochemical changes of a single neuron in the brain are fairly small and cannot be detected from the distance of the scalp. But when a great number of neurons are simultaneously activated, their synaptic activity results in electrochemical changes that are strong enough to be detected. That is, the EEG signal, recorded at the scalp, is always the result of the activation of a sizable cell population (Niedermayer and Silva 1993; Coles and Rugg 1997). The apical dendrites of the pyramidal cells are particularly important for the signal formation – dipolar ﬁelds result from the activation of pyramidal cell populations, which are oriented parallel and orthogonal to the cortex surface. The potential differences in such dipoles produce so-called open ﬁelds, which drive currents through the tissue, the liquor, the skull, and up to the scalp (Gazzaniga 2002). Each of these biological structures has electrical properties. The tissue and liquor behave like a resistance and the skull like a capacitor; therefore, voltage changes occur that can be measured by the electrodes at the scalp. At least two electrodes are necessary to register a voltage difference at a particular site. Generally, up to 256 electrodes are placed on participants’ heads and usually one or more electrodes that are linked together act as reference electrodes (Coles and Rugg 1997). The arrangement of the electrodes on the scalp is standardized (the so-called extended 10–20 system) and at present there are convenient and handy cap systems that determine the position of the electrodes. The channels are labelled according to the head position (e.g. Fz, frontocentral; P3, left parietal; O2, right occipital; even numbers indicate the right hemisphere, odd numbers the left hemisphere). It is important to note that the signal that is recorded at a particular electrode site is not necessarily associated with brain activity generated right below the electrode (Coles and Rugg 1997). There are active and passive electrode systems. Active electrodes amplify the tiny signal (only a few microvolts) before it is fed into an ampliﬁer. Finally, the signal is recorded at a physical data carrier. The recorded signal is a pattern that varies in voltage over time. The amplitude normally ranges between −100 and 100 µV (Coles and Rugg 1997) and varies in its frequency (e.g. 4–40 Hz; see Sect. 2.2). Once the signal

68

¨ M. Ollinger

has been stored, many optimizations and analyses are possible (ﬁltering, baseline, or artefact correction, etc.; see Luck 2005 for a detailed overview). An important factor of the EEG measurement is the sampling rate. Conventional digital EEG systems do not measure continuously but with a ﬁxed sampling rate. For example, a system with a sampling rate of 250 Hz produces 250 data points at each channel per second. There is a trade-off between the quality and the resolution of the signal and the resulting data volume. Recording for 1 h from 128 channels with a sampling rate of 250 Hz results in 115,200,000 (3, 600 s × 128 × 250 s−1 ) data points. Increasing the sampling rate up to 1,000 Hz results in 460,800,000 data points – which means higher resolution but also much larger data volumes and signiﬁcantly longer times for data processing and analysis (Ward 2003). Studying brain processes via EEG requires carefully planned and conducted experiments. The design of the experiments is mostly determined by the question of interest; that is, whether we are interested in a particular event-related potential (ERP) component, or in the changes in a frequency band (e.g. alpha band), or in synchronization processes, or rather in the localization of the dipole sources that are responsible for the signal recorded at the scalp, or in all factors at the same time. The most commonly used methods are the analysis of frequencies and of ERPs, but there are a growing variety of approaches that can be applied to EEG signals (Handy 2005).

2.2 Frequency The ﬁrst ever observed EEG frequency was the frequency band between 8 and 12 Hz. Berger (1929), who ﬁrst described this EEG pattern, assigned the Greek letter alpha to the observed frequency. Moreover, the EEG frequency band is separated into a distinct set of oscillations all labelled with Greek letters: delta oscillations are between 0.5 and 3.5 Hz, theta oscillations are between 3.5 and 8 Hz, alpha oscillations are between 8 and 12 Hz, beta oscillations are between 12 and 30 Hz, and gamma oscillations are between 30 and 80 Hz (Herrmann et al. 2005), or sometimes up to more than 100 Hz. As we will see, the distinction between the different frequency bands is not arbitrary, but is rather due to the different brain functions or states that each band reﬂects. For example, it was found that alpha power is larger when the eyes are closed than with eyes open (Ward 2003). Furthermore, during nocturnal sleep clearly distinguishable EEG patterns emerge that can be linked to particular cognitive states, such as dreaming. As we already mentioned, EEG recordings can produce a vast amount of data. There are various analysis techniques dealing with this problem. The basic goal when analysing EEG data in terms of frequencies is to ﬁnd certain oscillation patterns or synchronizations of a particular wavelength (an overview is provided in Herrmann et al. 2005). The spectral power analysis, for example, allows a measurement of the extent to which the recorded EEG signal contains synchronous oscillation patterns of various frequencies. In principle, a Fourier analysis is

EEG and Thinking

69

performed on the data that decomposes the oscillating pattern into a distinct set of the simplest possible oscillations, sine and cosine waves. It is possible to compute the spectral power of a particular frequency, which represents the amount of energy in the ﬂuctuations at that frequency. This allows us to draw conclusions from the processing activity of groups of neurons in order to understand the ongoing cognitive processes (Ward 2003; Kahana 2006). The evidence indicates that oscillations perform a variety of tasks: they temporally link neurons together, they bias input selection, and they facilitate synaptic plasticity (Buzs´aki and Draguhn 2004).

2.3 Event-Related Potentials ERPs are often used to ﬁnd physiological evidence for the theoretical assumptions of cognitive psychology, like odd-ball tasks, early attention, or categorization (Coles and Rugg 1997; Luck 2005). A ﬁxed event is presented (e.g. a tone or an image) and the EEG signal related to this event is recorded at a particular electrode site. Afterwards the recorded signal is averaged. Averaging cancels out the noise and the remaining signal is the ERP. There are a number of ‘standard’ ERP signals. The most prominent ERPs have an assigned tag. For example, the N400 peaks about 400 ms after the onset of an event and has a negative deﬂection; that is, the tag codes the latency and the polarity of the component (by convention, mostly negative amplitudes are drawn upwards in EEG charts). Different experimental conditions can produce different peaks and latencies at the recorded electrode site. It is important to note that an ERP waveform is mostly a superposition of different subcomponents and that, even more important, the same observed ERP waveform can be the result of different subcomponents. Moreover, ERPs can vary considerably between subjects (Luck 2005). Owing to the fact that the changes a single event causes are very small and often masked by noise, ERP studies almost always consist of a large number of similar trials for each condition. If the number of trials is carefully chosen, this strategy cancels out random ﬂuctuation of the signal and produces a higher signal-to-noise ratio. The main problem in working with ERP waveforms is the mapping of cognitive processes onto particular components.

3 Synchronization Generally, synchronization touches the difﬁcult and unsolved problem of how the brain concerts its various parallel and distributed activities such that coherent perceptions, cognitions, thoughts, and behaviour occur (Ward 2003). This problem implicates one of the most fascinating and puzzling questions; namely how the brain is able to represent conjunctions of properties (Hummel 1999; Roskies 1999) or more generally, how is the “inner and outer world” represented, integrated,

70

¨ M. Ollinger

consciously experienced, and behaviourally manipulated? We will give a brief overview of the current debate and outline what EEG ﬁndings contributed to the answer to this question. First, we will introduce and explain the famous binding problem. Second, we will review certain theoretical assumptions concerning the neural mechanisms that are probably responsible for binding phenomena, and third, we will summarize some studies and deliberations from the ﬁeld of cognitive neurosciences that refer to consciousness, object recognition, and working memory. Finally, we will introduce a new approach that tries to model higher cognitive functions by using EEG data. “The binding problem, or constellation of problems, concerns our capacity to integrate information across time, space, attributes, and ideas.” (p. 105 in Treisman 1999). Imagine a simple rectangle that is red and vertically oriented – form, colour, orientation, and location need to ‘bind’ together. How is it possible that we can segregate objects from each other, for example segregating a cup on a table from the table? What mechanisms in the brain are responsible for our experience of coherent perceptions of the world – since we know that different and distributed brain areas contribute to encoding and processing the different properties of an object? The question is how are these brain areas dynamically linked together (Tallon-Baudry and Bertrand 1999)? In the perceptual domain it is assumed that attentional selection processes play an important role for binding, which uses either the attended location or shared features (Roskies 1999; Treisman 1999). But binding is not only restricted to perception. It might also play an important role at a conceptual level, or at higher cognitive levels (Roskies 1999; Engel and Singer 2001). Hummel (1999, p. 85) pointed out, “[b]inding is the problem of representing properties. It is a very general problem that applies to all types of KNOWLEDGE REPRESENTATION, from the most basic perceptual representations to the most complex cognitive representations.” For example, it was shown that correct binding of synaptic and semantic structures is crucial for the comprehension of language (Treisman 1999) and also for thinking (Shastri and Ajjanagadde 1993). It is plausible to assume that thinking per se is not possible without binding. A simple syllogism, for example, requires encoding the premises and inferring from the premises a valid conclusion. First the letters, words, and sentences of each premise need to ‘bind’ together in a meaningful way, then the premises have to be linked together, and ﬁnally the conclusion has to be read off (Fangmeier et al. 2006). A number of accounts deal with the binding problem (Treisman and Gelade 1980; Hummel 1999; Singer 1999; Treisman 1999; Buzs´aki 2006). A possible solution to the binding problem might be the idea of neurons coding elements that belong to the same object, or ﬁring in synchrony (Singer 1999; Treisman 1999; Engel and Singer 2001; Kaiser and Lutzenberger 2003; Buzs´aki 2006). This assumption is termed the binding-by-neural-synchrony hypothesis (Singer 1999). Buzs´aki (2006) claims that synchrony is always related to an “observer”; that is, simultaneity is always simultaneous within a deﬁned frame of reference. Synchrony requires a frame of reference that consists of a discrete and small temporal window determined by a neuron or a neuronal assembly. Synchronous activation ensures that widely distributed neurons code information

EEG and Thinking

71

about the same object, event, or proposition. Buzs´aki and Draguhn (2004) stressed that the phase-locked discharges of distributed oscillators with the same frequency may be responsible for the binding of perceptual features. Especially gamma-band activation (30–100 Hz) between such oscillating cell populations is suspected to be the ‘glue’ between neuronal cell assemblies (Engel and Singer 2001; Buzs´aki 2006; Fries et al. 2007). More generally, Sejnowski and Paulsen (2006) discussed three possible and not necessarily mutually exclusive roles of network oscillation from a computational perspective: (1) Network oscillations contribute to the representation of information – this directly refers to the binding problem. (2) Oscillations and synchrony regulate the ﬂow of information in neural circuits – it was demonstrated that attention, for instance, increases the fast brain oscillations. Therefore, this oscillation might be more involved in processing than in representing (see later). (3) Oscillations assist in the storage and retrieval of information in neural circuits – synchronization enhances obviously the synaptic plasticity and therefore the encoding of new memory content (see later). Fries et al. (2007) proposed two complementary binding mechanisms. First, distant neurons from intra-area or inter-area connections form a cell assembly by synchronization of their action potentials – this is the classical binding-by-neuralsynchrony hypothesis (Singer and Gray 1995). Second, local groups of neurons are bound together by the interplay of pyramidal cells and interneurons (Tallon-Baudry et al. 1999). The basic assumption is that excitatory input of pyramidal cells triggers networks of inhibitory interneurons (Buzs´aki 2006). These inhibitory networks oscillate with gamma frequency (Buzs´aki 2006; Sejnowski and Paulsen 2006; Fries et al. 2007). Therefore, the network of interneurons is according to the phase more or less inhibitory, and according to the network dynamics, the pyramidal input is conveyed into a temporal pattern. “Amplitude values are converted into phase values that indicate by how much a discharge precedes the peak of a gamma cycle” (Fries et al. 2007). Fries et al. demonstrated that the stronger the excitatory input of the pyramidal cells, the better the synchronization with the gamma cycle – they termed this phenomenon coincidence detection. The gamma cycle increases the signal-tonoise ratio of the signal, because small excitatory inputs during the inhibition phase of the interneurons are suppressed (Buzs´aki 2006). The argument that binding by neural synchronization is a very general process that contributes to various cognitive processes was addressed by the work of Engel and Singer (2001); see also Ward (2003). They claimed that binding plays an important role in understanding the neural correlates of consciousness. They investigated the interplay between temporal binding and sensory awareness by postulating four crucial processes: (1) (2) (3) (4)

Arousal Sensory segmentation Selection Working memory

72

¨ M. Ollinger

They experimentally demonstrated that each of these processes can be explained by temporal binding processes. They identiﬁed temporal binding as the key mechanism that is able to generate dynamically coherent internal states and to achieve perceptual selection. These may be the signiﬁcant processes for controlling the access of information to conscious awareness. In summary, Engel and Singer proposed that consciousness requires large-scale coherence between different brain areas. What can we learn about the binding phenomenon from human EEG recording (Kahana 2006)? Most of the assumptions reviewed above were empirically tested with animals (rats, ferrets, cats, and monkeys). In animals it is possible to record directly from and even stimulate the brain sites of interest. By contrast, in humans an averaged signal, which is the result of the activation of several distributed neural assemblies, is recorded (Engel and Singer 2001). Nevertheless, synchronization and oscillation can be investigated even at the human scalp, by searching for synchronization using changes of power in particular frequency bands (Kaiser and Lutzenberger 2003; Ward 2003). Tallon-Baudry and Bertrand (1999) differentiate between three types of gamma response (see also Herrmann et al. 2005): (1) A 40-Hz transient evoked response: The oscillatory response is phase-locked to an event, for example the onset of a presented stimulus (similar to the ERP method introduced above). The oscillation can be detected by averaging the recorded epochs after the event. There is evidence from multisensory domains for the existence of early 40-Hz oscillations that might be important for the precise temporal relationships between stimuli and might therefore be responsible for the binding of synchronously incoming events. (2) A 40-Hz steady-state response: A periodically oscillating stimulus results in a sinusoidal response at the driving stimulus frequency, showing maximum amplitude in the gamma range. This phenomenon can be explained by the natural resonance frequencies of the brain. It can be used to ‘tag’ stimuli with a particular frequency in order to track the stages of stimuli processing in the brain (Ding et al. 2006). (3) Induced gamma (30–80 Hz) response: This type of response consists of oscillatory bursts whose latency varied across trials. Its temporal coincidence with a given event is fairly loose. Averaging the signal across several trials cancels out the gamma response. To analyse such EEG responses, other analysis approaches are necessary. For example, time-frequency power analysis methods can deal with the different onsets of the gamma response (Tallon-Baudry and Bertrand 1999). It is assumed that induced gamma activity could be a neural correlate of higherlevel cognitive processing. Induced gamma activity is found in motor tasks and detection tasks. It is modulated by attention, and can be lateralized with reference to the cognitive task that is performed. For example, left-lateralized gamma activation is elicited during a verbal task and right-lateralized activation during a mental rotation task. The distributed localizations of gamma-band activity may be the electrical correlate of task-speciﬁc arousals in the relevant brain sites (Tallon-Baudry

EEG and Thinking

73

Fig. 1 1 coherent triangle condition (Kanizsa 1976), 2 real triangle condition, 3 control condition

and Bertrand 1999). Furthermore, it was found that stimulus-induced gamma activity occurs across different sensory modalities and tasks and it mostly appears within a time range between 200 and 400 ms. Tallon-Baudry et al. (1996) conducted an ingenious experiment to investigate the interplay between induced gamma activity and perceptual binding. They speculated that if gamma activity plays an important role in perceptual binding then coherent stimuli should induce stronger gamma activity than incoherent stimuli. They operationalized their assumption by presenting participants with either an illusory triangle (Kanizsa 1976 (Fig. 1)) or a ‘no-triangle’ stimulus under the condition that basic visual features were as similar as possible in both conditions. They found that only in the coherent triangle condition a strong burst of gamma activity (30–60 Hz) was found after 280 ms. In a second condition participants were presented with ‘real’ triangles that elicited a signature fairly similar to that in the coherent triangle condition. In a control condition they found no signiﬁcant increase in the gamma activity. This ﬁnding provides evidence that gamma activity reﬂects the neural correlate of the feature binding process in humans. Of course, one has to be careful with directly mapping the cognitive term ‘coherence’ onto a neural mechanism like the synchronous ﬁring of neural cell assemblies (Fig. 1). Ward (2003) attempted to classify different oscillation frequencies with respect to their cognitive process (see also Kaiser and Lutzenberger 2003; Buzs´aki and Draguhn 2004; Kahana 2006): • Memory: The slow theta rhythm (3.5–7 Hz) might play an important role for the encoding of memory contents. When participants navigated through a labyrinth by memory alone, there were more frequent theta oscillations than in a condition where the way through the labyrinth was indicated by arrow cues. Gamma oscillations are apparently important for successful memory formation. Ward suggested that gamma-band coherence is important for the transient coupling of functional brain areas during associative learning. Furthermore, there is greater gamma-band functional connectivity between parietal and frontal areas when people successfully recollect information from memory. It is also known that there might be a strong interaction between theta and gamma oscillations. Lisman and Idiart (1995) proposed a neurocognitive model that incorporates theta and gamma oscillations within a single framework. They argued that gamma oscillations reﬂect the neural activity of pyramidal cell assemblies storing information.

74

•

•

•

•

¨ M. Ollinger

These oscillations dissipate over time, so a refreshing mechanism is therefore necessary. Theta oscillations may provide such a mechanism. Slow theta activity refreshes the gamma synchronization of distinct pyramidal cell assemblies. In the model, theta oscillations serve as the carrier function of the superimposed fast gamma oscillations (Lisman and Idiart 1995; Ward 2003). Similar to the relationship between single letters and the whole word, gamma oscillations represent the local process (the letter), and theta activity binds this local activity together (the word). Moreover, Canolty et al. (2006) showed that high-frequency oscillations (80–150 Hz) can be modulated by theta oscillations. In ﬁve epileptic patients they placed electrodes directly at the cortex (subdural electrocorticogram; see also Kahana 2006). The EEG signals of the patients were recorded while they did various behavioural tasks. Across all tasks and participants a signiﬁcant theta/gamma coupling was found. The strength of the coupling strongly depended on the theta power as well as on the theta phase. The different tasks produced clearly distinguishable coupling patterns that varied with the degree of task differences. The more similarity there was between two tasks, the more similar was the spatial pattern of the theta/gamma coupling. From their observations the authors concluded that the coupling between different brain rhythms may facilitate the coordination of different brain areas (see also Buzs´aki and Draguhn 2004). Learning: Kaiser and Lutzenberger (2003) pointed out that fast oscillatory activity is often linked to learning and memory. The idea is that such oscillations putatively reﬂect the formation of new cell assemblies (Hebb 1949). Sejnowski and Paulsen (2006) argued that correlations in sensory input can induce synaptic plasticity. These changes enhance the likelihood that information is encoded in memory. A number of behavioural learning paradigms support this view. Moreover, this argument might also answer the question why selective attention is important for learning. Attention: Alpha activity (8–13 Hz) is often reported as suppressing cortical activity (Jung-Beeman et al. 2004). Attention may be the mechanism that guides or reinforces the activation of selected cell assemblies (Engel and Singer 2001). Increasing alpha power at particular scalp sites reﬂects the suppression of distraction that characterizes the selective attention. There is also an obvious interplay between alpha power decrease and gamma power increase. The alpha power decrease occurs at about the time when gamma power starts to increase. It is speculated that temporal feature binding by gamma oscillations starts after an alpha desynchronization (Ward 2003; Sejnowski and Paulsen 2006). Jung-Beeman et al. (2004) made an interesting suggestion. They investigated how the neural correlates of problem solutions accompanied by an “Aha!” experience differ from usual solutions. They reported two interesting phenomena in the EEG signal. First, they found a sudden burst of gamma activity around 300 ms before an “Aha!” solution occurred. Second, at an even earlier time range, they found gamma activity over parieto-occipital areas. They argued that alpha activity may suppress further perceptual input into the system, which would distract

EEG and Thinking

75

the thinking process. We think that the alpha activity provides a possible precondition for representational changes that are necessary for an insightful solution ¨ ¨ (Knoblich and Ollinger 2006; Ollinger et al. 2008). To summarize, there is growing evidence that emphasizes the importance of synchronization processes for understanding cognition. Synchronization turned out to be a fairly basic mechanism in the brain, which might be important for representing, encoding, storing, attending, and retrieving information. Synchronization can occur at small local or widely distributed neuronal networks and it varies in frequency and phase-locking of events and ongoing network oscillations. It provides a powerful tool for the suppression, ﬁltering, or reinforcement of incoming information. Although there is a lot of evidence that conﬁrms the plausibility and the importance of oscillatory processes in the brain, it still remains unclear how complex cognition can exactly be understood via synchronization. How does the brain keep track when it pursues a thought or solves a problem? Generally the question arises as to how synchronization at a simple and fast microlevel contributes to or, better say, accomplishes fairly slow, complex thoughts at the macrolevel. Fingelkurts and Fingelkurts (2005) proposed a hierarchical model – the operational architectonics framework – that tries to answer these questions. At the lowest level there are functional formations of small, local and transient cell assemblies that represent features, objects, etc. At the next level such local assemblies become integrated into more complex spatiotemporal patterns – termed ‘operational modules’. Operational modules represent complex cognitive operations and always have a more complex structure than the operations which constitute them (Fingelkurts and Fingelkurts 2005). At the highest level large-scale distributed networks bind different operational modules together in order to achieve more complex operational modules that are considered as the realization of complex cognitive macrooperations. The authors assumed a direct relationship between the ﬂuctuation in EEG signals and the process of thinking as a succession of discrete and relatively stable periods separated by rapid transitive processes – as a “stream of thoughts” (James 1890). Methodologically, Fingelkurts et al. (2004) applied a new approach to the recorded EEG data, which detects topographic sharp transition processes on a millisecond scale. It is assumed that sharp transitions in the EEG signal may reﬂect the elementary units of informational processing in the brain. After identifying statistically signiﬁcant transitions, they further analysed which EEG channels were coupled together and were therefore members of the same transient network. This method enabled them to map network activity to ongoing cognitive tasks (e.g. rest, encoding, retrieval, etc.). Of course this model is a long way from providing a closed framework for the neural correlates of thinking. Nevertheless it is an elegant approach to apprehend the dynamics of complex thinking. Fingelkurts and Fingelkurts (2005) regarded the model as a platform that bridges the gap between neural and mental dynamics. It still remains unclear whether it is justiﬁed to assume that neural assemblies are actually the basic units of cognition. Moreover, it is arguable whether observed transitions in the EEG signal reﬂect the change between cognitive acts.

76

¨ M. Ollinger

It is still a puzzling phenomenon how the brain is able to identify from the noise of active frequencies those patterns that allow us to pursue goal-directed actions, to build thoughts, or to solve problems. Furthermore, it is also an open question how emotional and motivational forces can be incorporated in the framework Do they modulate the oscillations in a manner similar to the attentional processes? Despite all these open questions, there is some progress towards a better understanding of human thinking processes. We think that EEG is the appropriate device to reﬁne our theoretical knowledge about the ongoing complex cognitive processes in the brain. EEG can help us to understand the elementary processes that represent objects in the world. Furthermore, even more complex cognitive architectures can be developed and reﬁned by the data and possibilities that EEG provides. In the next section we review some approaches that try to explain the process of categorization. In other words, the question of how the established representations are classiﬁed by the cognitive system.

4 Categorization What is thinking? According to Call and Tomasello (2005, p. 608) thinking is “characterized by mental transformation or leaps, not just direct perception or memory of particular stimuli; going ‘beyond the information given.”’ Holyoak and Morrison (2005, p. 2) deﬁned thinking as “the systematic transformation of mental representations of knowledge to characterize actual or possible states of the world, often in the service of goals.” That is, thinking is the manipulation of given and/or stored information. Thinking links, changes, or produces new mental contents and information in the service of goal attainment. Most current deﬁnitions of thinking emphasize the more or less obvious fact that there is ‘something’ that is transformed, changed, linked, composed, or manipulated. This ‘something’ refers to mental representation as concepts or categories. There are a vast number of different psychological and cognitive theories that deal with the notion of concepts and categorization (Margolis and Laurence 1999; Sternberg and Ben-Zeev 2001; Cohen and Lefebvre 2005; Holyoak and Morrison 2005). All of these theories assume a kind of ‘sameness function’ that ‘analyses’ the difference between the information at hand and provides a decision, for example, about whether an object belongs to a certain category or not (Ashby and Ell 2001; Sternberg and Ben-Zeev 2001; Ashby and Maddox 2005). By means of EEG, categorization is often studied using tasks that require visual object recognition. Participants were asked whether two consecutively presented objects belonged to the same or different categories. It was shown by EEG that categorization is quite a fast process. We are able to come up with a decision within a few hundred milliseconds (Johnson and Olshausen 2005). Johnson and Olshausen tested people with a cued-target task with different conditions. In the ﬁrst condition a word – the cue – was presented before a target. The target was a photographic

EEG and Thinking

77

object. Participants had to indicate whether the word and image belonged to the same category by pressing one button for same and another button for different categories. In a second condition participants were ﬁrst presented with an image, followed by a word stimulus. EEG measurement began with the presentation of the second stimuli (image or word). The authors found an evoked potential at the electrode Cz (central electrode at the scalp) that has similar characteristics to the classic P300 (Donchin and Coles 1988). The EEG signature showed strong target effects and varied with stimuli, semantic difﬁculty, and task difﬁculty. The signal did not vary with the target condition. Therefore, the authors concluded that ERPs in a cued-target condition reﬂect postsensory decision making processes and not sensory processes. Sim and Kiefer (2005) addressed the question whether category-related brain activity is due to modality-speciﬁc processes (visual versus functional representation systems) or due to domain-speciﬁc (e.g. natural versus artefactual) processes. In an earlier study Kiefer (2001) showed that categorizing natural and artefactual objects elicited different evoked potentials. Classifying natural objects results in a larger N170 than for artefactual objects. The N170 might be due to visual processing. The later N400 component, which is an index for semantic processing, was quite similar in both conditions. Compared with artefactual categories, natural categories showed a more positive potential over occipitotemporal and parietal areas, whereas the artefactual effects were located more frontocentral (motor cortex). Sim and Kiefer (2005) concluded that both perceptual and semantic sources contribute to category-related brain activation, and that there are different kinds of semantic knowledge. They predicted an interaction between the kind of object (natural and artefactual) and the kind of feature (visual or functional). That is, visual features become more important for the categorization of natural objects. In contrast, for artefactual objects functional features become more important. Participants did two tasks. In an exposure task they decided whether two consecutively presented objects belonged to the same category or not. In the test task participants either did a visual shape judgement task or a functional judgement task that required them to judge the functional signiﬁcance of an object. Sim and Kiefer used word pairs of common objects. Half of the pairs were artefactual objects (tools, furniture, etc.); the remaining half were natural objects (animals, plants, etc.). Word pairs were either visually (e.g. round) or functionally (e.g. used for cutting) similar. They found that natural categories showed a more positive activation for visual than for functional judgements over occipitoparietal areas in the N400 component. Furthermore, occipitoparietal activity was, in general, more positive for the visual than for the functional judgment task. That is, there is a clear linkage between the kind of category and the kind of judgement. They concluded that these results support a modality-speciﬁc account. Hauk et al. (2007) proposed another model for object categorization processes. They claim that categorization is not a sequential stage process, where perception ﬁrst establishes a visuospatial representation, then a recognition process matches the given object with stored information, and ﬁnally retrieval processes activate semantic information. In this sequential model, perceptual processes occur very early (ﬁrst 100 ms) and semantic processes appear much later (about 300–400 ms).

78

¨ M. Ollinger

Hauk et al. suggest that perceptual and conceptual processes are more intertwined than hitherto assumed. Therefore, they conducted a task to control the inﬂuence of perceptual and conceptual (semantic) aspects. They used line drawings in order to vary the two dimensions typicality and authenticity. The task was to decide whether the drawings depicted real objects or not. They introduced the following four conditions. Half of the line drawings depicted authentic objects that exist in the real world. These objects were further classiﬁed in atypical authentic objects (e.g. a camel with an extraordinary hump on its back differs from other animals, which usually do not have a hump on their backs) and typical authentic objects (e.g. a jackal). The rest of the line drawings depicted nonauthentic objects. Half of the objects were changed into more typical ones (e.g. removing the hump of a camel makes it more similar to other four-legged animals). The remaining half were changed into nonauthentic and nontypical objects (e.g. a jackal with a hump on its back). Behaviourally they found that participants showed faster responses to atypical items than to typical items. They argued that in this case attention is captured by a special feature (e.g. having a hump on the back). Because, in real-world objects atypical features are rare, the special feature constrains the search space. In the EEG signal two main effects were revealed. First, an early effect of structural typicality was found (about 116 ms after stimuli onset), which was larger for atypical than for typical items. It was shown that this effect was unaffected by authenticity, and might be a result of early visual processing. Hauk et al. explained this ﬁnding as evidence for early processes that did not reﬂect matching processes, where a stimulus is compared with a stored representation, but rather reﬂect differences in structural typicality. Second, there was a later effect of authenticity that was unaffected by typicality and that was larger for nonauthentic items (N480). This effect might result from semantic processing that differentiates authentic from nonauthentic items. It showed a bilateral temporal scalp distribution. Between the temporal course of these two main effects they found interactions between the two systems. With these results taken together, Hauk et al. could demonstrate that there are two separate systems for object categorization that have some overlaps. They postulated a fast perceptual system representing the structure of an object, and a slow semantic system; the two processes are not independent but are strongly interconnected. Wang et al. (2003) investigated the basic electrophysiological processes that occur when we compare stimuli. They found, in an S1–S2 stimulus-matching paradigm, that when the second stimulus (S2) had different attributes from the ﬁrst stimulus, an N270 ERP component can be measured. When the stimuli S1 and S2 were identical a positive P300 deﬂection was found. Zhang et al. (2003) elaborated on these ﬁndings and varied the appearance of different conﬂict types. In a modiﬁed Sternberg probe-matching paradigm they presented a memory set of three simple ﬁgures (e.g. square, triangle, and circle) and a probe set of ﬁgures. Participants had to judge whether a given probe was identical to one of the preceding memorized items. The procedure was as follows: ﬁrst the three items were sequentially presented (items 1, 2, 3), then the three probes were also sequentially presented (probes 1, 2, 3). The task was to judge whether item n and the corresponding probe n were identical. The task always required the comparison

EEG and Thinking

79

of a stored item with a presented probe. The amount of conﬂict was systematically manipulated (no-conﬂict, low- and high-conﬂict condition). In the no-conﬂict condition the probe ﬁgure was identical to the corresponding ﬁgure in the memorized set of items. In the low-conﬂict condition the probe was not an element of the item set. In the high-conﬂict condition the probe was an element of the set of memorized items, but it did not occur at the identical sequential position. Zhang et al. found that no-conﬂict probes elicited a P300. The conﬂict probes elicited an N270 that was more negative in the high-conﬂict condition than in the low-conﬂict condition. They concluded that the N270 may serve as a measure of the amount of conﬂict. They only found an N430 for the high-conﬂict condition. They speculated that the N430 may reﬂect the recruitment of higher cognitive resources to process the fact that the probe is actually a member of the stored items but in the wrong position, in contrast to the simpler case when the probe is not a member of the stored items. In our laboratory we were also interested in the processes that are at work when we have to judge whether two objects are same or different. Graf (2006) suggested that human object recognition uses computational principles that are quite similar to mathematical transformation processes. Of particular interest are topological transformations. The basic idea is that by means of gain modulation (Salinas and Sejnowski 2001) in parieto-occipital areas, a given object is changed by morphing processes until it ﬁts onto the ‘compared object’. Similarity is deﬁned in terms of the morphing distance between two objects. While recording the EEG signal from the scalp, we asked participants to judge whether two consecutively presented objects belonged to the same or to different categories. The objects we used were carefully built by a morphing algorithm (Graf 2006), and enabled us to compare objects with a small and a large morphing distance (Fig. 2). We were interested in the variations of the N270 and found that there was a fairly small N270 (recorded at Cz) when participants were presented with a pair of objects that had a small morphing distance. We found a much larger negative deﬂection when the presented pairs had a large morphing distance, and ﬁnally the N270 was even larger when the objects belonged to different categories. Behaviourally, we found that small distances between pairs of objects of the same category resulted in signiﬁcantly faster responses than for pairs of objects with a large distance or for different categories.

Small distance Large distance Fig. 2 Example of different members of the same category. The experiment always involved the presentation of pairs that have either a small distance or a large distance

80

¨ M. Ollinger

¨ We propose (M. Ollinger, K. Gramann and G. Graf, unpublished results) that the generator for the N270 might be located at the dorsal part of the anterior cingulated cortex, which is well known as the site where errors are detected (Holroyd et al. 2004). In our framework, we do not suppose a coarse error detection mechanism that merely decides between true or false, but we do assume a sensitive difference detection device that produces high neural activation when there is a large difference between two stimuli. We speculate that the detection of a large difference is accompanied by large EEG deﬂection, while this higher activity is probably functional for the acquisition of further higher-level processing that cross-checks whether there is actually a difference, and if this is true what makes the difference between the two given things.

5 Conclusion We hope that our review has demonstrated that EEG may be an appropriate and powerful technique to study higher cognition and complex thinking. Of course there are still a number of restrictions in using this technique. For example, motor artefacts are still a fair problem with regard to the data quality, and the localization of the dipoles that might be responsible for the recorded signal is still quite difﬁcult and computationally expensive, However, EEG methods have made some progress in the last decade towards dealing with these problems. There are more and more sophisticated devices and statistical methods that improve the data quality significantly. Furthermore, the increasing applicability of synchronously running fMRI and EEG allows more and more precise localization of the EEG signal sources. As we have shown, basic thinking processes are highly dynamical and fast; therefore, it is quite important to use tools that can deal with these system dynamics. EEG is such a tool. As we showed in Sect. 3, it is fairly plausible to assume that different brain areas ‘communicate’ together via synchronous frequencies. That is, having a thought or solving a problem may be a process that constantly changes those synchronization dynamics. These insights extend our understanding about brain processes dramatically; it is becoming more and more clear how the brain processes information, which brain areas interact when we make a comparison or come up with a decision, or more generally how the stream of thoughts is organized. From our perspective the next steps that are necessary for further progress in the understanding of human thinking require new neurocognitive models that are able to incorporate and extend these ﬁndings. A closed framework is necessary that can explain how elementary binding processes can be read off and manipulated by higher brain processes. How is the brain able to compose complex thoughts using such elementary binding processes? How can we understand, experience, or feel the beauty of a poem, or of a Bach cantata? Why do we reﬂect about life, love, hate, etc.? How do these elementary processes generate new insights or creativity? We think that understanding the basic principles of brain dynamics may help us to understand more complex and abstract cognitive functions.

EEG and Thinking

81

References Ashby EG, Maddox WT (2005) Human category learning. Annu Rev Psychol 56:149–178 Ashby FG, Ell SW (2001) The neurobiology of human category learning. Trends Cogn Sci 5:204–210 Berger H (1929) Ueber das Elektrenkephalogramm des Menschen. Arch Psychiatr Nervenkr 87:527–570 Buzs´aki G (2006) Rhythms of the brain. Oxford University Press, New York Buzs´aki G, Draguhn A (2004) Neuronal oscillations in cortical networks. Science 304:1926–1929 Call J, Tomasello M (2005) Reasoning and thinking in nonhuman primates. In: Holyoak KJ, Morrison RG (eds) The Cambridge handbook of thinking and reasoning. Cambridge University Press, Cambridge, pp 607–632 Canolty RT, Edwards E, Dalal SS, Soltani M, Nagarajan SS, Kirsch HE, Berger MS, Barbaro, NM, Knight RT (2006) High gamma power is phase-locked to theta oscillations in human neocortex. Science 313:1626–1628 Cohen H, Lefebvre C (2005) Handbook of categorization in cognitive science. Elsevier, Amsterdam Coles MGH, Rugg MD (1997) Electrophysiology of mind. Event-related brain potentials and cognition. Oxford University Press, Oxford DeHaan M (2007) Infant EEG and event-related potentials. Psychology, New York Ding J, Sperling G, Srinivasan R (2006) Attentional modulation of SSVEP power depends on the network tagged by the ﬂicker frequency. Cereb Cortex 16:1016–1029 Donchin E, Coles MGH (1988) Is the P300 component a manifestation of context updating? Behav Brain Sci 11:357–374 Engel AK, Singer W (2001) Temporal binding and the neural correlates of sensory awareness. Trends Cogn Sci 5:16–25 Fangmeier T, Knauff M, Ruff CC, Sloutsky V (2006) fMRI evidence for a three-stage model of deductive reasoning. J Cogn Neurosci 18:320–334 Fingelkurts AA, Fingelkurts AA (2005) Mapping of the brain operational architectonics. In: Chen FJ (ed) Focus on brain mapping research. Nova Science, New York, pp 59–98 Fingelkurts AA, Fingelkurts AA, Kivisaari R, Pekkonen E, Ilmoniemi RJ, Kahkonen S (2004) Local and remote functional connectivity of neocortex under the inhibition inﬂuence. Neuroimage 22:1390–1406 Fries P, Nikolif D, Singer W (2007) The gamma cycle. Trends Neurosci 30:309–316 Gazzaniga MS (2002) Cognitive neuroscience: The biology of the mind, 2nd edn. Norton, New York Graf M (2006) Coordinate transformation in object recognition. Psychol Bull 132:920–945 Handy TC (2005) Event-related potentials. MIT Press, Cambridge Hauk O, Patterson K, Woollams A, Cooper-Pye E, Pulvermuller F, Rogers TT (2007) How the camel lost its hump: the impact of object typicality on event-related potential signals in object decision. J Cogn Neurosci 19:1338–1353 Hebb DO (1949) The organization of behavior. A neuropsychological theory. Wiley, New York Herrmann CH, Grigutsch M, Busch NA (2005) EEG oscillations and wavelet analysis. In: Handy TC (ed) Event-related potentials. MIT Press, Cambridge, pp 229–259 Holroyd CB, Nieuwenhuis S, Yeung N, Nystrom L, Mars RB, Coles MGH, Cohen JD (2004) Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nat Neurosci 7:497–498 Holyoak KJ, Morrison RG (2005) The Cambridge handbook of thinking and reasoning. Cambridge University Press, Cambridge Hummel J (1999) Binding problem. In: Wilson RA, Keil FC (eds) The MIT encyclopedia of the cognitive sciences. MIT Press, Cambridge, pp 85–86 James W (1890) The principles of psychology, vol. I. Dover, New York

82

¨ M. Ollinger

Johnson JS, Olshausen BA (2005) The earliest EEG signatures of object recognition in a cuedtarget task are postsensory. J Vis 5:299–312 Jung-Beeman M, Bowden E, Haberman J, Frymiare J, Arambel-Liu S, Greenblatt R, Reber P (2004) Neural activity when people solve verbal problems with insight. PLoS Biol 2:500–510 Kahana MJ (2006) The cognitive correlates of human brain oscillations. J Neurosci 26:1669–1672 Kaiser J, Lutzenberger W (2003) Induced gamma-band activity and human brain function. Neuroscientist 9:475–484 Kanizsa G (1976) Subjective contours. Sci Am 234:48–52 Kiefer M (2001) Perceptual and semantic sources of category-speciﬁc effects: event-related potentials during picture and word categorization. Mem Cognit 29:100–116 ¨ Knoblich G, Ollinger M (2006) The eureka moment. Sci Am Mind 10:38–43 Lisman JE, Idiart MA (1995) Storage of 7 ± 2 short-term memories in oscillatory subcycles. Science 267:1512–1515 Luck SJ (2005) An introduction to the event-related potential technique. MIT Press, Cambridge Margolis E, Laurence S (1999) Concepts – core readings. MIT Press, Cambridge Niedermayer E, Silva LD (1993) Electroencephalography, 3rd edn. Williams & Wilkins, Baltimore ¨ Ollinger M, Jones G, Knoblich G (2008) Investigating the effect of mental set on insight problem solving. Exp Psychol 55:270–282 Roskies AL (1999) The binding problem. Neuron 24:7–9 Salinas E, Sejnowski TJ (2001) Gain modulation in the central nervous system: where behavior, neurophysiology, and computation meet. Neuroscientist 7:430–440 Sejnowski TJ, Paulsen O (2006) Network oscillations: emerging computational principles. J Neurosci 26:1673–1676 Shastri L, Ajjanagadde V (1993) From simple associations to systematic reasoning: a connectionist representation of rules, variables and dynamic bindings using temporal synchrony. Behav Brain Sci 16:417–494 Sim EJ, Kiefer M (2005) Category-related brain activity to natural categories is associated with the retrieval of visual features: evidence from repetition effects during visual and functional judgments. Cogn Brain Res 24:260–273 Singer W (1999) Binding by neural synchrony. In: Wilson RA, Keil FC (eds) The MIT encyclopedia of the cognitive sciences. MIT Press, Cambridge, pp 81–84 Singer W, Gray CM (1995) Visual feature integration and the temporal correlation hypothesis. Annu Rev Neurosci 18:555–586 Sternberg RJ, Ben-Zeev T (2001) Complex cognition. Oxford University Press, Oxford Tallon-Baudry C, Bertrand O (1999) Oscillatory gamma activity in humans and its role in object representation. Trends Cogn Sci 3:151–162 Tallon-Baudry C, Bertrand O, Pernier J (1999) A ring-shaped distribution of dipoles as a source model of induced gamma-band activity. Clin Neurophysiol 110:660–665 Treisman A (1999) Solutions to the binding problem: review progress through controversy summary and convergence. Neuron 24:105–110 Treisman A, Gelade G (1980) A feature integration theory of attention. Cogn Psychol 12:97–136 Wang YP, Tian SJ, Wang HJ, Cui LL, Zhang YY, Zhang X (2003) Event-related potentials evoked by multi-feature conﬂict under different attentive conditions. Exp Brain Res 148:451–457 Ward LM (2003) Synchronous neural oscillations and cognitive processes. Trends Cogn Sci 7:553–559 Zhang X, Wang Y, Li S, Wang L (2003) Event-related potential N270, a negative component to identiﬁcation of conﬂicting information following memory retrieval. Clin Neurophysiol 114:2461–2468

Near-Infrared Spectroscopy for Studying Higher Cognition Yoko Hoshi

Abstract Near-infrared spectroscopy (NIRS), which was originally designed for clinical monitoring of tissue oxygenation, has recently been receiving increasing attention as a useful tool for neuroimaging studies. This technique is completely noninvasive, does not require strict motion restriction, and can be used in a daily-life environment. It is expected that NIRS will provide a new direction for cognitive neuroscience research, more so than other neuroimaging techniques, although several problems with NIRS remain to be explored. This chapter describes the basic theory of NIRS, its potential and limitations, and the future prospects of this technique.

1 Introduction Near-infrared (NIR) spectroscopy (NIRS), the in vivo application of which was ﬁrst described by J¨obsis (1977) in 1977, was originally designed for clinical monitoring of tissue oxygenation. Since the early 1990s, it has also been developing as a useful tool for neuroimaging studies (functional NIRS, fNIRS) (Hoshi and Tamura 1993; Kato et al. 1993; Villringer et al. 1993). For 30 years, the technology has advanced and a wide range of NIRS instruments have been developed. Among them, the instruments for continuous-wave (CW) measurements based on the modiﬁed Beer–Lambert law (CW-type instruments) are the most readily available commercially. Instruments of this type allow us to observe dynamic changes in regional cerebral blood ﬂow (rCBF) in real time by measuring concentration changes in cerebral hemoglobin (Hb). The recent advent of multichannel CW-type instruments has greatly increased the use of NIRS in a variety of ﬁelds.

Y. Hoshi Integrated Neuroscience Research Team, Tokyo Institute of Psychiatry, 2-1-7 Kamikitazawa, Setagaya-ku, Tokyo 156-8585, Japan [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

83

84

Y. Hoshi

At the same time, however, the accuracy and reliability of NIRS have not yet been widely accepted. This is mainly attributable to incomplete knowledge of which region in the brain is sampled by NIR light, difﬁculty in selective detection of NIRS signals arising from the cerebral tissue, and the problem of quantiﬁcation. Since the detected light on the scalp carries information about not only the cerebral tissue but also the extracerebral tissue, it is necessary to separate signals originating in the cerebral tissue from those coming from the extracerebral tissue. For this purpose, a multidetector system consisting of CW-type instruments has been developed (McCormick et al. 1991). However, separation of NIRS signals was incomplete, and other methods are being explored. The major problem with NIRS is that concentration changes in Hb cannot be quantiﬁed with CW-type instruments, which has hindered NIRS from being widely employed in clinical medicine and research. Over the past 30 years, many different approaches to quantiﬁcation have been tried; however, the difﬁculty of quantiﬁcation has not yet been completely overcome. In this chapter, I will ﬁrst outline the basic theory of NIRS. Then, focusing mainly on CW measurements, I will give speciﬁc examples of the strengths and advantages of NIRS measurements over other neuroimaging modalities, and also clarify the problems. Finally, I will describe its future prospects.

2 Basic Theory of NIRS 2.1 NIRS Measurements NIR light, especially that between 700 and 900 nm, easily passes through biological tissue because the light in this region is less scattered and is absorbed by only a few biological chromophres, such as Hb, myoglobin (Mb), and cytochrome oxidase (CytOx) in mitochondria. The spectra of Hb and Mb in the NIR region vary with their oxygenation states. The spectra of CytOx also vary with its oxidation state. Thus, by measuring the light transmitted through the tissue, we can obtain information about the oxygenation states of Hb and Mb and the “redox state” of CytOx. The redox state of CytOx is very useful for monitoring tissue oxygenation. Because changes in the redox state of CytOx, however, occur only under severely hypoxic conditions (Hoshi et al. 1997), CytOx is not measured in neuroimaging studies with NIRS. Many different types of NIRS measurements, each associated with one type of instrumentation, have been developed. CW measurements, time-domain measurements (time-resolved spectroscopy, TRS), and frequency-domain measurements (phase-resolved spectroscopy, PRS) are three main categories of NIRS measurements. In CW-type instruments, the light sources emit continuously at constant amplitude, and the light intensity at a position a few centimeters away from the incident point is measured (Fig. 1a). Instruments of this type calculate relative concentration changes in chromophores according to the modiﬁed Beer–Lambert law [6]:

Near-Infrared Spectroscopy for Studying Higher Cognition

a

TPSF mean transit time

tm time (ns)

I, Φ, M

tissue

I0, Φ0, M0

I(t)

light intensity

light intensity

b

tissue

I0(t)

I

tissue

light intensity

I0

85

c

Fig. 1 Near-infrared spectroscopy (NIRS) measurements: (a) continuous wave measurements; (b) time-domain measurements (time-resolved spectroscopy); (c) frequency-domain measurements (phase-resolved spectroscopy). I light intensity, Φ phase shift, M modulation depth

A = − log(I/I0 ) = ε CL + S,

(1)

where I and I0 are the intensities of detected and illuminated light, respectively, ε is the molar absorption coefﬁcient, C is concentration of a chromophore, L is the length that light travels through the tissue (optical path length, also known as total path length, t-PL), and S denotes optical attenuation mainly due to scattering. For measurements of living tissue, such as brain tissue, multiple wavelengths are used, and concentration changes in oxygenated Hb (oxy-Hb) and deoxygenated Hb (deoxy-Hb) are calculated on the assumption that the t-PL is the same for all wavelengths and S is a constant. A sum of changes in oxy-Hb and deoxy-Hb provides the changes in total Hb (t-Hb). When the change in concentrations is not global, for example, regional brain activation, the optical path length in (1) is replaced by a partial path length (p-PL) through a region of uniform absorption change (Fig. 2). Since CW-type instruments cannot measure either t-PL or p-PL, instruments of this type do not provide absolute values of concentration changes but rather relative ones with an arbitrary unit. However, the temporal resolution of CW-type instruments is high (in general, faster than 500 ms) and they allow continuous measurement for a prolonged time. In TRS, ultrashort (picosecond-order) laser pulses are applied to the tissue, and the intensity of the emerging light is detected as a function of time (the temporal point spread function, TPSF) with picosecond resolution (Chance et al. 1988; Delpy et al. 1988; Fig. 1b). The TPSF provides information about optical properties of the head, that is, total absorption (µa ) and reduced scattering (µs ) coefﬁcients of the scalp, skull, cerebrospinal ﬂuid (CSF), and brain. Propagation of photons in the living tissue can be described by the diffusion approximation. The values of µa and µs are estimated by ﬁtting an analytical TPSF (based on the solution of the photon diffusion equation) (Patterson et al. 1991) to the observed TPSF. As with CW measurements, using multiple wavelengths and determining µa for each wavelength, one estimates absolute values of Hb concentrations in the head. The mean transit time of scattered photons is calculated from the TPSF (tm in Fig. 1b). The mean t-PL is

86

Y. Hoshi

Scalp Skull CSF

Brain

Fig. 2 NIR light through the head. The red area denotes a region of cortical activation. Determination of the path length through this region (partial path length) is required for quantiﬁcation of changes in cerebral hemoglobin concentration

determined by multiplying the speed of light in the medium by the mean transit time of scattered photons. It should be noted, however, that the mean t-PL determined by TRS measurements of the head is the summation of the p-PLs within the cerebral and extracerebral tissues. The information contained in the TPSF through TRS can also be obtained by PRS, the relation between the time and frequency information being the Fourier transform. In frequency-domain instruments, the light source is intensity-modulated at radio frequencies, and measurements are made not only of the intensity of the detected light, but also of its phase shift and modulation depth with respect to the input light (Lakowicz and Berndt 1990; Fig. 1c). It has been demonstrated that for typical tissues, and at frequencies below 200 MHz, the phase shift is linearly related to the mean optical path length.

2.2 NIR Light Propagation in the Head Light propagation is generally approximated by the diffusion equation and can be predicted by Monte Carlo simulation. Theoretical analyses of the head models have demonstrated that light propagation in the adult head is highly affected by the presence of a low-scattering CSF layer (Okada et al. 1997). This indicates that the penetration of light in the adult brain might be limited to the outer cortical gray matter. A more recent study has reported that a large source–detector spacing only broadens the sampling region on the brain surface and affects the penetration depth

Near-Infrared Spectroscopy for Studying Higher Cognition

87

in the adult head to a lesser degree, whereas the intensely sensitive region in the neonatal head is conﬁned in the gray matter; however, the deeper region of the white matter is sampled with a large source–detector spacing (Fukui et al. 2003). In theoretical analysis, optical properties in each layer of the head are critical for prediction of the light propagation, though these values in each layer used for analysis are different from study to study. This is attributable to the fact that in situ measurements of µa and µs are not feasible. Thus, further investigation is required to conﬁrm the validity of the theoretically predicted light propagation.

2.3 Interpretation of NIRS Signals Regional brain activation is accompanied by increases in rCBF and the regional cerebral oxygen metabolic rate (rCMRO2 ). It is widely accepted that the degree of the increase in rCBF exceeds that of the increase in rCMRO2 (Fox and Raichle 1986), which results in a decrease in deoxy-Hb in venous blood. Thus, increases in t-Hb and oxy-Hb with a decrease in deoxy-Hb are expected to be observed in activated areas in NIRS measurements. However, deoxy-Hb and t-Hb do not necessarily show these changes: no change in t-Hb with an increase in oxy-Hb and a reciprocal decrease in deoxy-Hb, and an increase or no change in deoxy-Hb accompanying increases in t-Hb and oxy-Hb were observed. Using a newly developed perfused rat brain model, we examined the direct effects of each change in CBF and CMRO2 on cerebral Hb oxygenation to interpret NIRS signals (Hoshi et al. 2001). We conﬁrmed that the directions of changes in oxy-Hb are always the same as those of rCBF, whereas the direction of changes in deoxy-Hb is determined by changes in venous blood oxygenation and volume. It has also been conﬁrmed that small changes in CBF are not accompanied by changes in t-Hb. Thus, oxy-Hb is the most sensitive indicator of changes in rCBF in NIRS measurements.

3 Functional NIRS 3.1 Neuroimaging Study with CW-Type Instruments With use of NIRS, various types of brain activities, such as motor and cognitive activities, have been assessed. The strengths and advantages of CW-type instruments are as follows: (1) temporal resolution is high (less than 0.5 s) and completely noninvasive, (2) measurements can be performed with less motion restriction and in natural environments, and (3) NIRS can easily be combined with any of the other modalities. These strengths and advantages of NIRS enable neuroimaging studies on subjects who have not been fully examined until now, such as children, the elderly, and patients with psychoneurological problems, as they are difﬁcult to measure by

88

Y. Hoshi

other neuroimaging techniques, such as positron emission tomography (PET) and functional magnetic resonance imaging (fMRI). In neonates and infants, NIRS has been mostly applied to investigate evoked responses to stimuli such as visual and auditory stimulation (Meek et al. 1998; Zaramella et al. 2001). More recently, however, higher-order functions such as response to language have been investigated in neonates (Peˇna et al. 2003). Although only a few NIRS studies on cognitive and socioemotional development have been reported so far (Baird et al. 2002), the importance of NIRS will soon increase in developmental psychology. In the few years following the ﬁrst application of NIRS by Okada et al. (1994) to evaluate the frontal function in chronic schizophrenics, only a few psychiatric applications were reported. Lately, however, NIRS has become an increasingly popular method in psychiatry (Fallgatter et al. 1997; Shinba et al. 2004; Suto et al. 2004). Several research groups have examined task-related hemodynamic changes in psychiatric patients and found task-dependent abnormalities in frontal hemodynamics in schizophrenia (Shinba et al. 2004) and depression (Suto et al. 2004). Such task-dependent abnormalities were also found in patients with Alzheimer’s disease (Fallgatter et al. 1997). These studies underline the usefulness of NIRS in investigating frontal lobe dysfunction and evaluating psychopathologic conditions in psychiatric patients. Using a multichannel CW-type instrument, we can examine spatiotemporal characteristics in hemodynamic changes associated with brain activity. Furthermore, multichannel NIRS instruments have the potential for imaging the sequence of brain activation (Hoshi and Tamura 1997a; Hoshi et al. 2003). In our previous study, it was observed that three brain regions (the left dorsolateral prefrontal cortex, left BA 8, and right ventrolateral prefrontal cortex) were independently activated during performance of the n-back task, in which the time course of changes in oxy-Hb was different and it appeared that these regions had worked in a complementary manner (Hoshi et al. 2003). This also implies that the results obtained in PET and fMRI studies can vary by measurement points. Examining the time course of hemodynamic changes is crucial for understanding the brain function.

3.2 Free Motion Neuroimaging Study Measurements with less motion restriction in the daily-life environment open new dimensions in neuroimaging studies. Cortical activation patterns associated with human gait could be visualized by using a 30-channel CW-type instrument (Miyai et al. 2001). This indicates that NIRS was useful for evaluating cerebral activation patterns during pathological movements and rehabilitation intervention. Furthermore, a portable NIRS instrument combined with a wireless telemetry system (the wearable NIRS system) allows subjects to move during measurements as with portable ECG and EEG instruments (Fig. 3). This NIRS system makes it possible to monitor brain activity of freely moving subjects outside of laboratories (Hoshi and Chen 2006).

Near-Infrared Spectroscopy for Studying Higher Cognition

89

Fig. 3 Measurement being performing using the wearable NIRS system

4 Problems of NIRS 4.1 Selective and Quantitative Detection of NIRS Signals Quantiﬁcation of NIRS data has been a central issue in the NIRS ﬁeld. When Hb concentration changes are global within the tissue, quantiﬁcation is possible with TRS and PRS, which can determine the t-PL. In the case of functional brain activation, where Hb concentration changes are localized, however, those changes cannot be quantiﬁed accurately. The optical path length determined by TRS and PRS is the mean t-PL but not the mean p-PL in the cerebral tissue. Since the t-PL is much longer than the p-PL (Firbank et al. 1998; Hoshi et al. 2005), Hb concentration changes are underestimated when the t-PL is substituted for the modiﬁed Beer– Lambert law. However, measurement of the p-PL is not feasible. To quantify NIRS data obtained from CW-type instruments without measuring the t-PL, the assumption that the ratio of the source–detector separation to the t-PL (differential path length factor, DPF) (Delpy et al. 1988) and that of the p-PL to the t-PL are constant has often been made. Arranging the source–detector separation for each pair to be the same, in which the t-PL can be considered a constant if the assumption is correct, multichannel CW-type instruments generate topographical images of relative concentration changes in Hb. However, this assumption is not correct. TRS measurements have revealed that the DPF varies with each position. Furthermore, the p-PL is negatively related to the t-PL at a ﬁxed source–detector spacing, and the ratio of the p-PL to the t-PL varies with each wavelength and each measurement position (M¨uhlemann et al. 2006). This means that substitution of the t-PL

90

Y. Hoshi

for the Beer–Lambert law provides not only underestimated but also inaccurate results. In addition to the issue of quantiﬁcation, amplitudes of NIRS signals vary with the source–detector position (Strangman et al. 2003). Thus, comparing amplitudes across subjects and/or regions within a subject is not valid. Diffuse optical tomography (DOT), which reconstructs images of Hb concentration changes using multiple light sources and detectors, is a potential technique for quantitative and selective detection of focal changes in cerebral hemodynamics (Arridge 1999). DOT is not based on the modiﬁed Beer–Lambert law and can be performed with TRS (Gao et al. 2002), PRS (Pogue et al. 1995), and CW-type instruments (Boas et al. 2001). TRS is also a potential tool for this purpose. It provides the TPSF, which carries information about depth-dependent attenuation even for a measurement with a single source–detector distance. While several timedomain approaches (Hielscher et al. 1996; Sato et al. 2005; Steinbrink et al. 2001) and a multidistance frequency domain approach (Choi et al. 2004) have been proposed, further investigation must be continued to apply these methods to human head measurement.

4.2 NIRS Data Analysis Unlike PET and fMRI, there are no standard methods of NIRS data analysis, and various analyses have been performed so far. This is not in itself a problem; however; the validity and reliability of each method should be conﬁrmed. Until recently, to examine whether task-related changes in NIRS signals in an individual are significant or not, comparison of NIRS signals between the resting and activation states has commonly been performed by using a paired t test. Since, however, NIRS data are time-series data, t statistics cannot be used for this aim, although they can be used for comparing means of NIRS signal changes between two states within subjects. Autoregressive models are commonly used to analyze time-series data, though it is very difﬁcult to derive an autoregressive model for NIRS data. Model-based, event-related, and both combined analyses have recently been tried in some research groups (Plichta et al. 2007; Schroeter et al. 2004). Although the model-based analysis is widely used for data analysis in fMRI and PET studies, it is unclear whether this analytical method can be applied to NIRS data, because the pattern of hemodynamic changes varies with each measurement and it is difﬁcult to derive proper hemodynamic response functions, which might be also true of PET and fMRI. In the case that the same task can be performed repeatedly without habituation, however, an event-related analysis is available. NIRS signals are not constant during the resting state, which possibly reﬂects physiological phenomena (Hoshi and Tamura 1997b; Toronov et al. 2000), and it is often observed that these signals do not return to the original levels immediately after the activation state. In such cases, baseline correction has been performed in some studies; however, it should be noted that the baseline correction could distort actual cerebral hemodynamic changes.

Near-Infrared Spectroscopy for Studying Higher Cognition

91

5 Future Prospects There remain a number of technical issues to be explored and practical difﬁculties to be solved. Nevertheless, NIRS is a tool distinct from other neuroimaging techniques for the study of brain functions and for the diagnosis, assessment, and treatment of psychoneurological diseases. Thus, a variety of novel applications of NIRS, such as NIRS-based brain-computer interface (BCI), are being tried. Most of the current BCI systems rely on the brain’s electrical activity to produce scalp EEG signals. Since the scalp EEG signals, however, are inherently noisy and nonlinear, a more accessible interface that uses a more direct measurement of brain function to control an output device is being explored. NIRS is considered as a possible alternative to electrical signals (Sitaram et al. 2007). Another optical approach to detect brain activation is also being tried. Conventional NIRS instruments detect signals corresponding to relatively slow hemodynamic responses. In contrast, a much faster signal occurring over a period of tens of milliseconds has been detected by both PRS (Wolf et al. 2002) and CW-type (Franceschini and Boas 2004) instruments. These fast signals, which are thought to be attributable to scattering changes in neurons, are much weaker than those of hemodynamic origin, and high temporal resolution is required for their detection. The instrumentation and data analysis used in the techniques have improved remarkably over the last few years, making it feasible to detect neuronal activity. This new approach is becoming a powerful clinical tool. The last few years have seen the development of the multichannel NIRS system in which user-friendliness, one of the most beneﬁcial features of NIRS, has been discarded: at the same time, efforts to miniaturize the NIRS system have been made. We have been extending the wearable system mentioned in Sect. 3.2 to the multichannel system. Wolf’s group (M¨uhlemann et al. 2006) has recently succeeded in miniaturizing NIR optical imaging and creating a wireless sensor. Such miniaturized NIRS systems will contribute not only to neuroscience research but also to monitoring tissue oxygenation, which was the original aim of NIRS development. For the last 30 years, NIRS has been making steady progress, and it has been conﬁrmed that the potential beneﬁts of NIRS are considerable. Its strengths and advantages are expected to open new dimensions in brain research that other neuroimaging techniques have not been able to. Thus, NIRS shows great promise for providing further insight into brain function and as a clinical tool.

References Arridge SR (1999) Optical tomography in medical imaging. Inverse Probl 15:R41–R93 Baird A, Kagan J, Gaudette T et al (2002) Frontal lobe activation during object performance: data from near-infrared spectroscopy. Neuroimage 16:1120–1126 Boas DA, Gaudette T, Strangman G et al (2001) The accuracy of near infrared spectroscopy and imaging during focal changes in cerebral hemodynamics. Neuroimage 13:76–90

92

Y. Hoshi

Chance B, Leigh JS, Miyake H et al (1988) Comparison of time-resolved and -unresolved measurements of deoxyhemoglobin in brain. Proc Natl Acad Sci USA 85:4971–4975 Choi J, Wolf M, Toronov V et al (2004) Noninvasive determination of the optical properties of adult brain: near-infrared spectroscopy approach. J Biomed Opt 9:221–229 Delpy DT, Cope M, van der Zee P et al (1988) Estimation of optical pathlength through tissue from direct time of ﬂight measurement. Phys Med Biol 33:1433–1442 Fallgatter AJ, Roesler M, Sitzmann L et al (1997) Loss of functional hemispheric asymmetry in Alzheimer’s dementia assessed with near-infrared spectroscopy. Cogn Brain Res 6:67–72 Firbank M, Okada E, Delpy DT (1998) A theoretical study of the signal contribution of regions of the adult head to near-infrared spectroscopy studies of visual evoked responses. Neuroimage 8:69–78 Fox P, Raichle ME (1986) Focal Physiological uncoupling of cerebral blood ﬂow and oxidative metabolism during somatosensory stimulation in human subjects. Proc Natl Acad Sci USA 83:1140–1144 Franceschini MA, Boas DA (2004) Noninvasive measurement of neuronal activity with nearinfrared optical imaging. Neuroimage 21:372–386 Fukui Y, Ajichi Y, Okada E (2003) Monte Carlo prediction of near-infrared light propagation in realistic adult and neonatal head models. Appl Opt 42:2881–2887 Gao H, Zhao H, Yamada Y (2002) Improvement of image quality in diffuse optical tomography using full time-resolved data. Appl Opt 41:778–791 Hielscher AH, Liu H, Chance B et al (1996) Time-resolved photon emission from layered turbid media. Appl Opt 35:719–728 Hoshi Y, Chen S-J (2006) New dimension of cognitive neuroscience research with near-infrared spectroscopy: free-motion neuroimaging studies In: Chen FJ (ed) Progress in brain mapping research, 1st edn. Nova Science, New York Hoshi Y, Tamura M (1993) Detection of dynamic changes in cerebral oxygenation coupled to neuronal function during mental work in man. Neurosci Lett 150:5–8 Hoshi Y, Tamura M (1997a) Near-infrared optical detection of sequential brain activation in the prefrontal cortex during mental tasks. Neuroimage 5:292–297 Hoshi Y, Tamura M (1997b) Fluctuations in the cerebral oxygenation state during the resting period in functional mapping studies of the human brain. Med Biol Eng Comput 35:328–330 Hoshi Y, Hazeki O, Kakihana Y et al (1997) Redox behavior of cytochrome oxidase in the rat brain measured by near-infrared spectroscopy. J Appl Physiol 83:1842–1848 Hoshi Y, Kobayashi N, Tamura M (2001) Interpretation of near-infrared spectroscopy signals: a study with a newly developed perfused rat brain model. J Appl Physiol 90:1657–1662 Hoshi Y, Tsou BH, Billock VA et al (2003) Spatiotemporal characteristics of hemodynamic changes in the human lateral prefrontal cortex during working memory tasks. Neuroimage 20:1493–1504 Hoshi Y, Shimada M, Sato C et al (2005) Reevaluation of near-infrared light propagation in the adult human head: implications for functional near-infrared spectroscopy. J Biomed Opt 10:064032 J¨obsis FF (1977) Noninvasive infrared monitoring of cerebral and myocardial oxygen sufﬁciency and circulatory parameters. Science 198:1264–1267 Kato T, Kamei A, Takashima S et al (1993) Human visual cortical function during photic stimulation monitoring by means of near-infrared spectroscopy. J Cereb Blood Flow Metab 13:516–520 Lakowicz JR, Berndt K (1990) Frequency domain measurement of photon migration in tissues. Chem Phys Lett 166:246–252 McCormick PW, Stewart M, Goetting MG et al (1991) Noninvasive cerebral optical spectroscopy for monitoring cerebral oxygen delivery and hemodynamics. Crit Care Med 19:89–97 Meek JK, Firbank M, Elwell CE et al (1998) Regional hemodynamic response to visual stimulation in wake infants. Pediatr Res 43:840–843 Miyai I, Tanabe HC, Sase I et al (2001) Cortical mapping of gait in humans: a near-infrared spectroscopic topography study. Neuroimage 14:1186–1192

Near-Infrared Spectroscopy for Studying Higher Cognition

93

M¨uhlemann T, Haensse D, Wolf M (2006) Ein drahtloser Sensor f¨ur die bildgebende in-vivo Nahin¨ frarotspektroskopie. Paper presented at the 3-L¨andertreffen der Deutschen, Osterreichischen und Schweizerischen Gesellschaft f¨ur Biomedizinische Technik, Zurich Okada F, Tokumitsu Y, Hoshi Y et al (1994) Impaired interhemispheric integration in brain oxygenation and hemodynamics in schizophrenia. Eur Arch Psychiatry Clin Neurosci 244:17–25 Okada M, Firbank M, Schweiger SR et al (1997) Theoretical and experimental investigation of near-infrared light propagation in a model of the adult head. Appl Opt 36:21–31 Patterson MS, Madsen SJ, Moulton JD et al (1991) Diffusion equation representation of photon migration in tissue. IEEE Microw Symp Dig 905–908 Peˇna M, Maki A, Kovaˇci´c D et al (2003) Sounds and silence: an optical topography study of language recognition at birth. Proc Natl Acad Sci USA 100:11702–11705 Plichta MM, Heinzel S, Ehlis A-C et al (2007) Mode-based analysis of rapid event-related functional near-infrared spectroscopy (NIRS) data: a parametric validation study. Neuroimage 35:625–634 Pogue BW, Patterson MS, Jiang H et al (1995) Initial assessment of a simple system for frequency domain diffuse optical tomography. Phys Med Biol 40:1709–1729 Sato C, Shimada M, Hoshi Y et al (2005) Extraction of depth-dependent signals from time-resolved reﬂectance in layered turbid media. J Biomed Opt 10:064008 Schroeter ML, B¨ucheler MM, M¨uller K et al (2004). Towards a standard analysis for functional near-infrared imaging. Neuroimage 21:283–290 Shinba T, Nagano M, Karia N et al (2004) Near-infrared spectroscopy analysis of frontal lobe dysfunction in schizophrenia. Biol Psychiatry 55:154–164 Sitaram R, Zhang H, Guan C et al (2007) Temporal classiﬁcation of multi-channel near infrared spectroscopy signals of motor imagery for developing a brain-computer interface. Neuroimage 34:1416–1427 Steinbrink J, Wabnitz H, Obrig H et al (2001) Determining changes in NIR absorption using a layered model of the human head. Phys Med Biol 46:879–896 Strangman G, Franceschini MA, Boas DA (2003) Factors affecting the accuracy of near-infrared spectroscopy concentration calculation for focal changes in oxygenation parameters. Neuroimage 18:865–879 Suto T, Fukuda M, Ito M et al (2004) Multichannel near-infrared spectroscopy in depression and schizophrenia: cognitive brain activation study. Biol Psychiatry 55:501–511 Toronov MA, Franceschini M, Filiaci S et al (2000) Near-infrared study of ﬂuctuations in cerebral hemodynamics during rest and motor stimulation: temporal analysis and spatial mapping. Med Phys 27:801–815 Villringer A, Plank J, Hock C et al (1993) Near-infrared spectroscopy (NIRS): a new tool to study hemodynamic changes during activation of brain function in human adults. Neurosci Lett 154:101–104 Wolf M, Wolf U, Choi JH et al (2002) Functional frequency-domain near-infrared spectroscopy detects fast neuronal signal in the motor cortex. Neuroimage 17:1868–1875 Zaramella P, Freato F, Amigoni A et al (2001) Brain auditory activation measured by near-infrared spectroscopy (NIRS) in neonates. Pediatr Res 49:213–219

Integration of EEG and fMRI Christoph Mulert(¬) and Ulrich Hegerl

Abstract The integration of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) has attracted a lot of interest in the last few years, offering insights into human brain function with both high temporal and high spatial information. Today, methodological problems of simultaneous measurements in terms of hardware and artifact correction are either being resolved or can be dealt with reasonably. While the combination of these two techniques had been of interest primarily in the clinical ﬁeld of epilepsy, it is now increasingly gaining importance in the ﬁeld of cognitive neuroscience, for example in offering information about mental chronometry aspects. This chapter describes relevant aspects such as the physiological principles, technical and methodological aspects, artifact correction and some interesting applications.

1 Introduction The use of functional magnetic resonance imaging (fMRI) for the investigation of human brain function has been very successful during the last decade since it is able to offer reliable “brain mapping.” Today, it looks like our understanding of several aspects of brain function could beneﬁt from a combination of fMRI with electrophysiological measurements such as electroencephalography (EEG) and eventrelated potentials (ERP). fMRI is an extremely powerful tool that is now widely used to answer research questions in different areas, including psychology (Coltheart 2006; Durston and Casey 2006), psychiatry (Callicott and Weinberger 1999; Mitterschiffthaler et al. 2006), neurology (Rocca and Filippi 2006), and even economic decisionmaking (Sanfey et al. 2003) and social interaction (Montague et al. 2002). The most C. Mulert Functional Brain Imaging Branch, Department of Psychiatry, Ludwig-Maximilian University, Nussbaumstr. 7, 80336 Munich, Germany [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

95

96

C. Mulert, U. Hegerl

important characteristic of this noninvasive method is its excellent spatial resolution. Research questions that can be answered using fMRI include “. . . and where in the brain does it happen?” Since the hemodynamic changes associated with neural activity are within the range of seconds, temporal dynamics of brain activity typically observed within the range of milliseconds cannot be sufﬁciently described with this indirect method. The patterns of limitations are completely different for the classical electrophysiological measurements such as EEG and evoked potentials, offering an excellent temporal resolution but only limited spatial resolution. On account of this, much effort was put into combining the strengths of both EEG and fMRI using simultaneous acquisition. While at the beginning of this research questions of patient safety (Lemieux et al. 1997) and feasibility (Goldman et al. 2000; Krakow et al. 2000a,b; Mulert et al. 2002; Otzenberger et al. 2005) had been paramount, additional effort has now been put into questions of artifact correction (Allen et al. 1998; Bonmassar et al. 2002; Debener et al. 2007), comparisons of independent EEG- and fMRI-based analyses (Mulert et al. 2004, 2005), and analyses of different frequency bands (Laufs et al. 2003a, 2006; Moosmann et al. 2003). Recently, it emerged that functional neuroimaging with high temporal and spatial resolution is possible using trial-by-trial coupling of EEG and fMRI (Benar et al. 2007; Debener et al. 2005; Eichele et al. 2005; Mulert et al. 2008). The striking feature of trial-by-trial coupling is its ability to separate different aspects of the blood oxygenation level dependent (BOLD) signal according to their speciﬁc relationships with distinct neural processes.

2 Physiology The physiological basis of fMRI is the BOLD contrast (Ogawa et al. 1990). The BOLD effect is related to hemodynamic changes in the brain: an increase of neural activity causes a moderate increase of the metabolic rate of oxygen consumption but a larger increase in local cerebral blood ﬂow. Neural synaptic activity has been demonstrated to be correlated with the BOLD signal (Logothetis et al. 2001). However, the mechanism of neurovascular coupling is not yet fully clear. The BOLD contrast can be measured with a high spatial resolution within the range of millimeters (Ugurbil et al. 2003) or even on the level of single cortical columns (Fukuda et al. 2006). But there are also some limitations to conventional fMRI studies: While neurons can show many different activity patterns both in terms of activation/inhibition (depolarization or hyperpolarization) and in terms of timing, fMRI shows all of them in the same way and to the degree they are related to blood ﬂow changes/energy consumption. The consequence is that it is difﬁcult to know whether inhibitory or excitatory, fast or slow, synchronized or desynchronized neuronal activity is underlying the BOLD effect in a certain brain region. The latter fact is troublesome, because changes in neuronal activity during cognitive processes can consist of changes in synchronization without a net increase in neuronal activity and would therefore not be detectable by fMRI. fMRI has a limited temporal resolution within the range of

Integration of EEG and fMRI

97

seconds and therefore cannot represent the cognition-related temporal dynamics of neuronal activity which are within the range of milliseconds. Scalp-recorded EEG represents synchronized electrical mass activity of cortical neurons. Interestingly, the EEG signal is mainly due to excitatory postsynaptic potentials and inhibitory postsynaptic potentials; on the other hand, action potentials do not generally contribute to the scalp EEG signal (Lopes da Silva and van Rotterdam 1993). The precondition of getting a measurable EEG signal is that neurons have to be spatially organized appropriately (e.g., like a palisade) and have to be synchronously active. While the EEG signal directly represents neural (synaptic) activity, it is difﬁcult to calculate the intracerebral sources of the potentials on the scalp. This difﬁculty is fundamental: It is called the “inverse problem” and was described more than 150 years ago (Helmholtz 1853). It means that different combinations of intracerebral sources can result in the same potential distribution on the scalp. The inverse problem has no unique solution; therefore, EEG-based localizations are merely reasonable estimates. Regarding the combination of EEG and fMRI, it is interesting to note that both methods tend to mainly represent synaptic activity. This is an important point since it suggests that both methods pick up similar neural processes and that combination approaches might in fact describe different aspects of the same neural process. While the exact relationship between neural activity and vascular changes is not fully understood yet, efforts are currently being made to formulate its mathematical relations (Friston et al. 2003; Lee et al. 2006). At this point, it should also be mentioned that there are of course several constellations in which a one-to-one relationship of scalp EEG information and BOLD signal changes cannot be assumed. For example, neural activity might be related to BOLD signal changes but not to scalp EEG signal changes if, owing to the spatial orientation of the electrical generators (e.g., self-canceling sources in sulci or neuronal assemblies without a strict parallel orientation), the electrical signals cancel each other out. On the other hand, highly synchronous activity of a small number of neurons might result in a detectable EEG signal change, but the associated hemodynamic changes might be small and not sufﬁciently above baseline values to survive statistical testing. Therefore, the presumed likelihood of a close relationship of EEG and fMRI data should be considered and/or independent analyses of both data sets should be performed before strategies of data fusion can be applied. Strengths and limitations of both methods are summarized in Table 1.

3 Technical and Methodological Aspects The measurement of EEG signals inside the magnetic resonance (MR) scanner is challenging both in terms of EEG quality (see later) and MR image quality (Mullinger et al. 2008). There are three different types of electromagnetic ﬁelds used in a MR scanner: the main static magnetic ﬁeld B0 , the time-varying magnetic gradient ﬁelds, and the radiofrequency (RF) pulsed ﬁeld generated by the headcoil. Changing magnetic ﬁelds induce signiﬁcant current ﬂow within the electrodes and

98

C. Mulert, U. Hegerl

Table 1 Strength and limitations of electroencephalography/event-related potentials (EEG/ERP) and functional magnetic resonance imaging (fMRI)

Time resolution Spatial resolution Which brain structure can be studied? What is measured? What is difﬁcult to measure?

EEG/ERP

fMRI

Millisecond ∼2 cm Cortex

Several seconds Millimeters Cortical and subcortical structures Changes in blood ﬂow and oxygen consumption Changes in synchronization or frequency of neuronal activity

Synchronized postsynaptic potentials Unsynchronized activity

wires. In simultaneous EEG–fMRI experiments, it is therefore important to consider patient safety issues. However, the recording of EEG signals during MR imaging can be done safely if some safety issues are considered (Lemieux et al. 1997). Large voltages can be induced in loops formed by EEG wires resulting from the RF pulses; therefore, avoiding loops as well as ferromagnetic materials is important for the safety of the subject in the scanner. We have performed more than 500 simultaneous EEG–fMRI measurements so far without any adverse incidents and several other groups also use this technique safely (Hamandi et al. 2004).

3.1 Artifacts Two major artifacts are known to disturb EEG measurements inside the scanner: the pulse artifact and the image-acquisition artifact. For investigations of highfrequency oscillations in the gamma-band range, artifacts due to helium pump activity also have to be considered.

3.1.1 The Pulse Artifact The pulse artifact – or “ballistocardiogram” artifact – is present in the main static magnetic ﬁeld and is mainly due to small electrode and body movements. The ballistocardiogram artifact is associated with the cardiac cycle, whose shape is similar in each event. This property was the basis for the ﬁrst powerful strategy for artifact correction: The averaging of the ballistocardiogram should cancel out the unrelated EEG signals – the resulting artifact template could then be used for artifact correction of the continuous EEG (Allen et al. 2000). This has to be done channelwise because the shape of this artifact clearly varies from electrode to electrode. While this “average-template subtraction” method generally works ﬁne, some problems remain. This is due to the fact that the artifact is not completely stable over time and shows a considerable trial-by-trial variability. Therefore, additional approaches have

Integration of EEG and fMRI

99

been introduced, including modiﬁcations of the template approach, using a sliding average, or methods based on topographical information by means of principal component analysis or independent component analysis (Mantini et al. 2007; Sirvastava 2005). Niazy et al. (2005) have successfully used a technique based on a temporal principal components analysis named optimal basis set and Debener et al. (2007) have recently combined optimal basis set and independent component analysis for a further improvement of artifact correction (Fig. 1). Another interesting method is the multiple source correction approach: here, artifact correction takes into account a priori knowledge of signiﬁcant brain activity and the artifact and the signiﬁcant brain activity are then separated by means of principal component analysis (Siniatchkin et al. 2007).

E34 E50 E51 E52 E53 E55 E57 E58 E59 E60

400 µV

a E34 E50 E51 E52 E53 E55 E57 E58 E59 E60

200 µV

b E34 E50 E51 E52 E53 E55 E57 E58 E59 E60

150 µV

c Fig. 1 Artifact correction of electroencephalography data acquired during functional magnetic resonance imaging (fMRI). (a) Raw data. (b) After correction of the image-acquisition artifact. (c) After correction of the pulse artifact. (Reprinted from Debener et al. 2007, with permission from Elsevier.)

100

C. Mulert, U. Hegerl

3.1.2 The Image-Acquisition Artifact While the image-acquisition artifact, which is caused by the gradient switching and RF pulses, is much larger in amplitude than the EEG signal and thus dominates the view of EEG raw data obtained in the MR scanner (Fig. 1), it is nevertheless easier to correct. One approach is based on the premise that the frequency spectrum of this artifact is much higher than the EEG rhythms usually focused on. Therefore, comparing the power spectrum information of an artifact-free EEG segment and an artifact-contaminated EEG segment allows one to deﬁne the distinct frequencies of the artifact, which then can easily be ﬁltered out (Hoffmann et al. 2000). Alternatively, it is possible to average the artifact and use an artifact average template for subtraction (Allen et al. 2000). This approach works best if high sampling rates are used (e.g., 5,000 Hz) and the MR scanner and the EEG recording are synchronized (Mandelkow et al. 2006). It has been shown that using a template subtraction strategy, the EEG signal is well preserved and, for example, visual evoked potentials can be obtained without loss of the good signal-to noise ratio (Becker et al. 2005). Recently, an interesting approach was suggested with modiﬁed echo planar imaging sequences, enabling EEG sampling every 1 ms in periods almost without any artifact (stepping stone sampling) (Anami et al. 2003).

3.1.3 Helium Pump For the investigation of high-frequency oscillations in the gamma-band range (30–100 Hz) another artifact type has to be considered. This artifact type is due the activity of the helium pump, which produces distinct high-frequency artifacts in many MR scanners. Although ﬁltering or artifact template subtraction could be applied, it is much easier to get rid of this artifact by switching off the helium pump for the time period of the experiment (Mulert et al. 2006).

4 Experimental Design and Data Analysis In some studies investigating epileptic events MR image acquisitions were manually triggered as soon as the experimenter visually noticed an epileptic spike in the ongoing EEG activity (Lemieux et al. 2001). This strategy is possible because of the hemodynamic delay of the MR image acquisition: accordingly, a few seconds after the epileptic event the respective peak of the BOLD response can be captured. While in more recent studies of epileptiform activity continuous acquisition of EEG and fMRI recordings is used (Hamandi et al. 2004), continuous acquisition is deﬁnitely necessary when cognitive experiments are performed. One possibility of interleaved simultaneous EEG–fMRI acquisition is if the respective EEG activity is phase-locked to a stimulus and occurs in a short time period (some hundreds of milliseconds), it is possible to present the stimulus and to investigate this EEG segment

Integration of EEG and fMRI

101

in-between two consecutive MR image acquisitions. In this case, the correction of image-acquisition artifacts is not necessary. This strategy has a second important advantage if auditory stimulation is used: The auditory stimuli are not disturbed by the noise of the gradient switching (Mulert et al. 2005). On the other hand, this design cannot be used if the continuous EEG is in focus (for example in the investigation of resting brain rhythms or vigilance). Here, the electroencephalogram can be analyzed only after correction of the image-acquisition artifact (Moosmann et al. 2003). Concerning the analysis, it is important to consider different strategies. One possibility is using fMRI activations to constrain inverse EEG solutions. EEG information can then be used, for example, to present source waveforms as estimates of the respective brain area. Thus, this strategy offers a description of brain activity with a high spatial and temporal resolution and can also be applied using EEG and fMRI data sets acquired in separate sessions (Bledowski et al. 2004). A problem with this method is that it is not clear a priori which brain activations are really represented in the scalp EEG. Therefore, for example, the use of dipole seeding in regions not contributing to the scalp EEG might disturb the validity of the whole inverse model. Another strategy that has been introduced more recently is single-trial coupling of EEG and fMRI. Trial-by-trial coupling predicts the BOLD signal speciﬁcally related to amplitude variations of electrophysiological components. Debener et al. (2005) were able to demonstrate speciﬁc BOLD changes following the error-related negativity. Eichele et al. (2005) showed speciﬁc BOLD signal changes for the ERP involved in pattern learning. Benar et al. (2007) demonstrated P300-related activations in the occipitotemporal junction and in medial and frontal regions. Mulert et al. (2008) found BOLD activations speciﬁc to the N1 potential in the auditory cortex and the anterior cingulate cortex (ACC). One important aspect of trial-by-trial coupling is the fact that it allows imaging of different qualities of neural activity for which conventional fMRI cannot be used: It has been shown that high-frequency oscillations in the gamma-band range of a neuronal population can be involved in aspects of processing other than slow potential changes. The potential to differentiate these aspects could be relevant for many neuroimaging investigations. In addition, different kinds of neurons may be involved in different neurophysiological processes. For instance, we know that pyramidal cells and interneurons are differentially involved in high-frequency oscillations (Traub et al. 2003) and that distinct frequency preferences of various types of neurons exist (Gloveli et al. 2005; Pike et al. 2000).

5 Applications Initially, simultaneous EEG–fMRI was used for the investigation of epileptic activity (Hamandi et al. 2004; Salek-Haddadi et al. 2002). Since then several other applications have emerged, including measuring the activity of the resting wake (Mantini et al. 2007) or sleeping brain (Czisch et al. 2004; Wehrle et al. 2007), and the investigation of several cognitive functions, including target detection (Mulert et al. 2004), error detection (Debener et al. 2005), etc.

102

C. Mulert, U. Hegerl

5.1 Brain Rhythms Concerning resting-state activity, several authors initially looked at the BOLD correlates of the alpha rhythm. Here, the typical ﬁnding was that increased alpha power was correlated with a decreased signal in the occipital cortex and with an increased signal in the thalamus (Feige et al. 2005; Goldman et al. 2002; Moosmann et al. 2003), while in another study negative correlations were found between alpha power and the BOLD signal in parietal and frontal areas (Laufs 2003 a,b).

L

R

(Y) +5 0 −5 −10

(z=29) −5

0

+5 cm (X) R (Y)

L

+5 0 −5 −10

(z=36) −5

0

+5 cm (X) R (Y)

L

+5 0 −5 −10

(z=43) −5

0

+5 cm (X)

Fig. 2 Concordant activations in the fMRI analysis and simultaneously acquired event-related potential activity localized independently with LORETA. (Reprinted from Mulert et al. 2004, with permission from Elsevier.)

Integration of EEG and fMRI

103

In a recent study, several different independent components were deﬁned on the basis of the fMRI data representing resting-state activity of the brain. The associated signal ﬂuctuations were then correlated with the EEG power of all frequency ranges. Interestingly, each functional network was characterized by a speciﬁc electrophysiological signature (Mantini et al. 2007).

5.2 Cognition/“Mental Chronometry” Simultaneous EEG–fMRI is now being used increasingly to investigate cognitive processes such as target detection (Mulert et al. 2004; Fig. 2), error processing (Debener et al. 2005) or mental arithmetic (Sammer et al. 2007). Another interesting application for cognitive neuroscience is “mental chronometry.” Here, the basic idea is that cognitive processes can be divided into sequential steps and the neural correlates of these steps can be analyzed by means of the EEG–fMRI combination (Linden 2007). A good example for the method of how this can be done is the study by Eichele et al. (2005), who demonstrated speciﬁc BOLD correlates of distinct ERP components (P2, N2, P300) during an oddball task using single-trial coupling of EEG and fMRI.

6 Outlook In the future, combining EEG and fMRI may prove to be crucial for a deeper understanding of brain activity. At the moment, both practical issues (e.g., artifact correction) and basic questions (e.g., the precise relationship between EEG and fMRI signals) are a matter of debate. Nevertheless, several studies have already shown the enormous impact of combination strategies and further improvement of technical aspects is expected to turn simultaneous EEG–fMRI into a routine method for the investigation of human brain function.

References Allen PJ, Polizzi G, Krakow K, Fish DR, Lemieux L (1998) Identiﬁcation of EEG events in the MR scanner: the problem of pulse artifact and a method for its subtraction. Neuroimage 8:229–239 Allen PJ, Josephs O, Turner R (2000) A method for removing imaging artifact from continuous EEG recorded during functional MRI. Neuroimage 12:230–239 Anami K, Mori T, Tanaka F, Kawagoe Y, Okamoto J, Yarita M, Ohnishi T, Yumoto M, Matsuda H, Saitoh O (2003) Stepping stone sampling for retrieving artifact-free electroencephalogram during functional magnetic resonance imaging. Neuroimage 19:281–295 Becker R, Ritter P, Moosmann M, Villringer A (2005) Visual evoked potentials recovered from fMRI scan periods. Hum Brain Mapp 26:221–230

104

C. Mulert, U. Hegerl

Benar CG, Schon D, Grimault S, Nazarian B, Burle B, Roth M, Badier JM, Marquis P, LiegeoisChauvel C, Anton JL (2007) Single-trial analysis of oddball event-related potentials in simultaneous EEG-fMRI. Hum Brain Mapp 28:602–613 Bledowski C, Prvulovic D, Hoechstetter K, Scherg M, Wibral M, Goebel R, Linden DE (2004) Localizing P300 generators in visual target and distractor processing: a combined event-related potential and functional magnetic resonance imaging study. J Neurosci 24:9353–9360 Bonmassar G, Purdon PL, Jaaskelainen IP, Chiappa K, Solo V, Brown EN, Belliveau JW (2002) Motion and ballistocardiogram artifact removal for interleaved recording of EEG and EPs during MRI. Neuroimage 16:1127–1141 Callicott JH, Weinberger DR (1999) Neuropsychiatric dynamics: the study of mental illness using functional magnetic resonance imaging. Eur J Radiol 30:95–104 Coltheart M (2006) What has functional neuroimaging told us about the mind (so far)? Cortex 42:323–331 Czisch M, Wehrle R, Kaufmann C, Wetter TC, Holsboer F, Pollmacher T, Auer DP (2004) Functional MRI during sleep: BOLD signal decreases and their electrophysiological correlates. Eur J Neurosci 20:566–574 Debener S, Ullsperger M, Siegel M, Fiehler K, von Cramon DY, Engel AK (2005) Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identiﬁes the dynamics of performance monitoring. J Neurosci 25:11730–11737 Debener S, Strobel A, Sorger B, Peters J, Kranczioch C, Engel AK, Goebel R (2007) Improved quality of auditory event-related potentials recorded simultaneously with 3-T fMRI: removal of the ballistocardiogram artefact. Neuroimage 34:587–597 Durston S, Casey BJ (2006) What have we learned about cognitive development from neuroimaging? Neuropsychologia 44:2149–2157 Eichele T, Specht K, Moosmann M, Jongsma ML, Quiroga RQ, Nordby H, Hugdahl K (2005) Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc Natl Acad Sci U S A 102:17798–17803 Feige B, Schefﬂer K, Esposito F, Di Salle F, Hennig J, Seifritz E (2005) Cortical and subcortical correlates of electroencephalographic alpha rhythm modulation. J Neurophysiol 93:2864–2872 Friston KJ, Harrison L, Penny W (2003) Dynamic causal modelling. Neuroimage 19:1273–1302 Fukuda M, Moon CH, Wang P, Kim SG (2006) Mapping iso-orientation columns by contrast agentenhanced functional magnetic resonance imaging: reproducibility, speciﬁcity, and evaluation by optical imaging of intrinsic signal. J Neurosci 26:11821–11832 Gloveli T, Dugladze T, Saha S, Monyer H, Heinemann U, Traub RD, Whittington MA, Buhl EH (2005) Differential involvement of oriens/pyramidale interneurones in hippocampal network oscillations in vitro. J Physiol 562:131–147 Goldman RI, Stern JM, Engel J Jr., Cohen MS (2000) Acquiring simultaneous EEG and functional MRI. Clin Neurophysiol 111:1974–1980 Goldman RI, Stern JM, Engel J Jr., Cohen MS (2002). Simultaneous EEG and fMRI of the alpha rhythm. Neuroreport 13: 2487–2492 Hamandi K, Salek-Haddadi A, Fish DR, Lemieux L (2004) EEG/functional MRI in epilepsy: the Queen Square experience. J Clin Neurophysiol 21:241–248 ¨ Helmholtz H (1853) Uber einige Gesetze der Vertheilung elektrischer Str¨ome in k¨orperlichen Leitern mit der Anwendung auf die thierisch-elektrischen Versuche. Ann Phys Chem 211–233, 289, 353–377 Hoffmann A, Jager L, Werhahn KJ, Jaschke M, Noachtar S, Reiser M (2000) Electroencephalography during functional echo-planar imaging: detection of epileptic spikes using post-processing methods. Magn Reson Med 44:791–798 Krakow K, Allen PJ, Lemieux L, Symms MR, Fish DR (2000a) Methodology: EEG-correlated fMRI. Adv Neurol 83:187–201 Krakow K, Allen PJ, Symms MR, Lemieux L, Josephs O, Fish DR (2000b) EEG recording during fMRI experiments: image quality. Hum Brain Mapp 10:10–15 Laufs H, Kleinschmidt A, Beyerle A, Eger E, Salek-Haddadi A, Preibisch C, Krakow K (2003a) EEG-correlated fMRI of human alpha activity. Neuroimage 19:1463–1476

Integration of EEG and fMRI

105

Laufs H, Krakow K, Sterzer P, Eger E, Beyerle A, Salek-Haddadi A, Kleinschmidt A (2003b) Electroencephalographic signatures of attentional and cognitive default modes in spontaneous brain activity ﬂuctuations at rest. Proc Natl Acad Sci U S A 100:11053–11058 Laufs H, Holt JL, Elfont R, Krams M, Paul JS, Krakow K, Kleinschmidt A (2006) Where the BOLD signal goes when alpha EEG leaves. Neuroimage 31:1408–1418 Lee L, Friston K, Horwitz B (2006) Large-scale neural models and dynamic causal modelling. Neuroimage 30:1243–1254 Lemieux L, Allen PJ, Franconi F, Symms MR, Fish DR (1997) Recording of EEG during fMRI experiments: patient safety. Magn Reson Med 38:943–952 Lemieux L, Krakow K, Fish DR (2001) Comparison of spike-triggered functional MRI BOLD activation and EEG dipole model localization. Neuroimage 14:1097–1104 Linden D (2007) What, when, where in the Brain? Exploring mental chronometry with brain imaging and electrophysiology. Rev Neurosci 18:159–171 Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A (2001) Neurophysiological investigation of the basis of the fMRI signal. Nature 412:150–157 Lopes da Silva F, van Rotterdam A (1993) Biophysical aspects of EEG and magnetoencephalogram generation. In: Niedermeyer E, Lopes da Silva F (eds) Electroencephalography. Williams & Wilkins, Baltimore, pp 78–91 Mandelkow H, Halder P, Boesiger P, Brandeis D (2006) Synchronization facilitates removal of MRI artefacts from concurrent EEG recordings and increases usable bandwidth. Neuroimage 32:1120–1126 Mantini D, Perrucci MG, Del Gratta C, Romani GL, Corbetta M (2007) Electrophysiological signatures of resting state networks in the human brain. Proc Natl Acad Sci U S A 104:13170–13175 Mitterschiffthaler MT, Ettinger U, Mehta MA, Mataix-Cols D, Williams SC (2006) Applications of functional magnetic resonance imaging in psychiatry. J Magn Reson Imaging 23:851–861 Montague PR, Berns GS, Cohen JD, McClure SM, Pagnoni G, Dhamala M, Wiest MC, Karpov I, King RD, Apple N, Fisher RE (2002) Hyperscanning: simultaneous fMRI during linked social interactions. Neuroimage 16:1159–1164 Moosmann M, Ritter P, Krastel I, Brink A, Thees S, Blankenburg F, Taskin B, Obrig H, Villringer A (2003) Correlates of alpha rhythm in functional magnetic resonance imaging and near infrared spectroscopy. Neuroimage 20:145–158 Mulert C, Jager L, Pogarell O, Bussfeld P, Schmitt R, Juckel G, Hegerl U (2002) Simultaneous ERP and event-related fMRI: focus on the time course of brain activity in target detection. Methods Find Exp Clin Pharmacol 24(Suppl D):17–20 Mulert C, Jager L, Schmitt R, Bussfeld P, Pogarell O, Moller HJ, Juckel G, Hegerl U (2004) Integration of fMRI and simultaneous EEG: towards a comprehensive understanding of localization and time-course of brain activity in target detection. Neuroimage 22:83–94 Mulert C, Jager L, Propp S, Karch S, Stormann S, Pogarell O, Moller HJ, Juckel G, Hegerl U (2005) Sound level dependence of the primary auditory cortex: simultaneous measurement with 61-channel EEG and fMRI. Neuroimage 28:49–58 Mulert C, Hepp P, Karch S, Leicht G, Moller HJ, Hegerl U, Pogarell O (2006) Simultaneous measurement of 40 Hz electrical activity and the corresponding BOLD-signal: methodological issues. Neuroimage 31(Suppl 1):S112 Mulert C, Seifert C, Leicht G, Kirsch V, Ertl M, Karch S, Moosmann M, Lutz J, M¨oller HJ, Hegerl U, Pogarell O, J¨ager L (2008) Single-trial coupling of EEG and fMRI reveals the involvement of early anterior cingulate cortex activation in effortful decision making. Neuroimage. Apr 29. [Epub ahead of print] Mullinger K, Debener S, Coxon R, Bowtell R (2008) Effects of simultaneous EEG recording on MRI data quality at 1.5, 3 and 7 tesla. Int J Psychophysiol 67:178–188 Niazy RK, Beckmann CF, Iannetti GD, Brady JM, Smith SM (2005) Removal of FMRI environment artifacts from EEG data using optimal basis sets. Neuroimage 28:720–737 Ogawa S, Lee TM, Kay AR, Tank DW (1990) Brain magnetic resonance imaging with contrast dependent on blood oxygenation. Proc Natl Acad Sci U S A 87:9868–9872

106

C. Mulert, U. Hegerl

Otzenberger H, Gounot D, Foucher JR (2005) P300 recordings during event-related fMRI: a feasibility study. Brain Res Cogn Brain Res 23:306–315 Pike FG, Goddard RS, Suckling JM, Ganter P, Kasthuri N, Paulsen O (2000) Distinct frequency preferences of different types of rat hippocampal neurones in response to oscillatory input currents. J Physiol 529(1):205–213 Rocca MA, Filippi M (2006) Functional MRI to study brain plasticity in clinical neurology. Neurol Sci 27(Suppl 1):S24–S26 Salek-Haddadi A, Merschhemke M, Lemieux L, Fish DR (2002) Simultaneous EEG-correlated ictal fMRI. Neuroimage 16:32–40 Sammer G, Blecker C, Gebhardt H, Bischoff M, Stark R, Morgen K, Vaitl D (2007) Relationship between regional hemodynamic activity and simultaneously recorded EEG-theta associated with mental arithmetic-induced workload. Hum Brain Mapp 28:793–803 Sanfey AG, Rilling JK, Aronson JA, Nystrom LE, Cohen JD (2003) The neural basis of economic decision-making in the ultimatum game. Science 300:1755–1758 Siniatchkin M, Moeller F, Jacobs J, Stephani U, Boor R, Wolff S, Jansen O, Siebner H, Scherg M (2007) Spatial ﬁlters and automated spike detection based on brain topographies improve sensitivity of EEG-fMRI studies in focal epilepsy. Neuroimage Traub RD, Pais I, Bibbig A, LeBeau FE, Buhl EH, Hormuzdi SG, Monyer H, Whittington MA (2003) Contrasting roles of axonal (pyramidal cell) and dendritic (interneuron) electrical coupling in the generation of neuronal network oscillations. Proc Natl Acad Sci USA 100:1370–1374 Ugurbil K, Toth L, Kim DS (2003) How accurate is magnetic resonance imaging of brain function? Trends Neurosci 26:108–114 Wehrle R, Kaufmann C, Wetter TC, Holsboer F, Auer DP, Pollmacher T, Czisch M (2007) Functional microstates within human REM sleep: ﬁrst evidence from fMRI of a thalamocortical network speciﬁc for phasic REM periods. Eur J Neurosci 25:863–871

The Neural Implementation of Working Memory Oliver Gruber

Abstract In the past, different conﬂicting models of the neural representation of working memory have been proposed depending on whether they were derived from human or from animal studies. Recent data from several behavioural and functional MRI experiments have challenged these models by suggesting an evolutionary special role of the articulatory (verbal) rehearsal mechanism. This mechanism is implemented by mainly left-hemispheric premotor brain areas that are also related to human speech. Another, probably phylogenetically older working memory system (which appears to be present in non-human primates as well) is topographically organized along parallel prefrontoparietal circuits according to different informational domains. On the basis of these ﬁndings, an evolutionary-based model of human working memory has been proposed that permits the harmonization of the conﬂicting working memory models derived from human and animal research. The evolution of premotor cortices during human phylogeny may not only have provided the neuronal basis for language functions, but may also have strongly affected working memory capacity and other higher cognitive functions that presumably underlie complex thinking processes.

1 Conﬂicting Neurocognitive Models of Working Memory in Humans Working memory has been deﬁned as a set of linked and interacting information processing components that allow temporary storage and simultaneous manipulation of information in the brain, a function critical for higher cognitive functions such as language, planning and problem-solving (Baddeley 1992). Among the cognitive models of human working memory, the one provided by Baddeley O. Gruber Department of Psychiatry, University of G¨ottingen, von-Siebold-Str. 5, 37075 G¨ottingen, Germany [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

109

110

O. Gruber

and Hitch (1974) has been the most inﬂuential during the last two decades. It divided short-term memory, which formerly had been considered as an unitary system, into three principal components: the “central” executive as a complex attention-controlling system and two subsidiary “slave” systems designed to maintain representations of information of different modalities, i.e. the visuospatial sketchpad and the phonological loop. Further, the phonological loop was subdivided into two complementary verbal short-term memory components, namely a “passive” phonological storage component and an “active” subvocal rehearsal mechanism. The visuospatial sketchpad, on the other hand, is considered to be the working memory component that is specialized for the processing and storage of visual and spatial material. Whether this component can be further subdivided into systems for spatial and visual object information (Gathercole 1994; Hecker and Mapperson 1997) and, in close analogy to the functional architecture of the phonological loop, into “passive” storage and “active” rehearsal mechanisms (Washburn and Astur 1998; Awh and Jonides 1998), is still debated. During the last about 15 years, the neural implementation of human working memory has been extensively investigated by numerous functional neuroimaging studies. Following the inﬂuential three-component model of Baddeley and Hitch (1974), most of these studies assessed merely one of the proposed working memory components, i.e. the phonological loop, the visuospatial sketchpad or the central executive, in isolation (Jonides et al. 1993; Paulesu et al. 1993; D’Esposito et al. 1995; Courtney et al. 1996, 1998; Haxby et al. 2000). Different brain systems were found to underlie verbal and visuospatial working memory. Performance of verbal working memory tasks was associated with activation of mostly left-hemispheric speech areas that included Broca’s area, lateral and medial premotor cortices and the contralateral cerebellum (Paulesu et al. 1993; Awh et al. 1996). By contrast, performance of visuospatial working memory tasks led to activation of a fairly bilateral prefrontoparietal network including the frontal eye ﬁelds (posterior superior frontal cortex) and the intraparietal cortex (Jonides et al. 1993). Finally, from initial studies it was postulated that the dorsolateral prefrontal cortex may represent the neural basis of the “central executive” (D’Esposito et al. 1995) although many subsequent studies were unable to replicate this ﬁnding (Klingberg 1998; Adcock et al. 2000; Bunge et al. 2000, 2001). It is important to note, however, that other, conﬂicting functional-neuroanatomical models of working memory have been derived from studies of non-human primates using single-cell recordings and anatomical tract-tracing techniques (see Becker and Morris 1999 for a discussion). These studies have provided evidence that working memory processes are represented by parallel prefrontoparietal and prefrontotemporal circuits which are topographically organized according to different informational domains (see Goldman-Rakic 1996 for a review). The results of these investigations suggested that the dorsolateral prefrontal cortex subserves the online maintenance of visuospatial information, whereas the ventrolateral prefrontal cortex is involved in the maintenance of information about the features of visual objects. However, only relatively few functional neuroimaging studies of working memory in humans pro-

The Neural Implementation of Working Memory

111

duced results consistent with these ﬁndings in non-human primates (Courtney et al. 1998; Haxby et al. 2000), whereas several others failed to conﬁrm the suggested organizational principle of domain-speciﬁcity of working memory (D’Esposito et al. 2000; Nystrom et al. 2000; Postle et al. 2000). Apart from differences in the methods applied, such discrepancies between ﬁndings in humans and non-human primates may also reﬂect a lack of control over the cognitive strategies chosen by human subjects to perform a given task as well as evolutionary differences between nonhuman and human brain function.

2 An Evolutionary Perspective on Working Memory Systems: Cognitive Differences Between Humans and Non-human Primates One of the most obvious differences between the cognitive capacities of humans and non-human primates is the special endowment of humans with language. Correspondingly, it has been proposed that the evolution of language may have led to an anatomical dislocation of prefrontal brain regions that underlie cognitive processes concerned with visuospatial and visual object information, respectively. The dynamic inﬂuence that the availability of language may have on working memory functions and their cerebral representation has been widely neglected so far. If one wants to obtain consistent empirical data across human and non-human species, it is crucial, however, to take into account that the evolution of language may have produced changes in the functional implementation of working memory processes in the human brain. One possible way to make the working memory task performance of human subjects more comparable with that of non-human primates is to use articulatory suppression in order to deprive subjects of strategies which are speciﬁc to the human species. The articulatory suppression effect refers to the observation that concurrent articulation impairs verbal short-term memory, presumably owing to a disruption of the verbal rehearsal mechanism (Murray 1968; Baddeley et al. 1984). As a consequence, memory performance under articulatory suppression would have to rely on alternative phonological and/or visual storage mechanisms that may be similar to working memory mechanisms in non-human primates. This assumption was recently conﬁrmed by a series of functional neuroimaging studies of working memory in humans.

2.1 The Dual Architecture of Verbal Working Memory in Humans In the ﬁrst of these studies, articulatory suppression was applied to gain further insight into the neural correlates of verbal working memory in humans. This study provided evidence for a dual architecture of human verbal working memory by demonstrating that brain regions involved in explicit verbal rehearsal can

112

O. Gruber

be dissociated from a second, prefrontoparietal working memory system which presumably underlies an alternative, non-articulatory mechanism for maintaining phonological representations, i.e. phonological storage (Gruber and von Cramon 2001). While under single-task conditions, articulatory rehearsal activated Broca’s area, the lateral premotor cortex, and parietal areas, silent articulatory suppression was found to eliminate or reduce memory-related activity in these “classical” verbal working memory areas. Instead, activation related to memory performance was observed in a different network, including the cortex along the anterior part of the intermediate frontal sulcus and the inferior parietal lobule (Fig. 1). A straightforward interpretation of these ﬁndings is that this network of prefrontal and parietal areas underlies a brain mechanism by which phonological information can be maintained across a short period of time, in particular if it is not possible to rehearse. Since articulatory suppression is thought to interfere only with the rehearsal mechanism, one may argue that the observed dissociation between the two brain systems corresponds to a dissociation of non-articulatory phonological storage from explicit verbal rehearsal. Accordingly, these results suggested that phonological storage may

Fig. 1 The dual architecture of verbal working memory in humans (Gruber 2001). Green indicates memory-related activations that occurred only during articulatory rehearsal both under single-task (ST) and non-interfering dual-task (DT) conditions. Red indicates memory-related activations that occurred only during non-articulatory maintenance of the same phonological information, i.e. under articulatory suppression (AS). Brown indicates memory-related activations that were present in all conditions investigated in this study, i.e. independent of articulatory suppression. Bars in the inserts show the mean percentage of signal changes produced by the memory tasks in relation to the respective control conditions

The Neural Implementation of Working Memory

113

be a function of a complex prefrontoparietal network (as depicted in red in Fig. 1), and not localized in only one, parietal brain region.

2.2 Similar Brain Systems for Maintenance of Phonological and Visual Information in Working Memory Are Differentially Distributed Along Human Prefrontal and Parietal Cortices The second study in this systematic re-evaluation of the functional neuroanatomy of human working memory conﬁrmed that activation of the network of anterior prefrontal cortex and inferior parietal lobule observed in the ﬁrst study did not result from a possible switch to visual memory strategies, but indeed was associated with phonological memory in a domain-speciﬁc way. Although both phonological and visual working memory processes activated roughly similar prefrontoparietal networks, they were found to be differentially distributed along the same neuroanatomical structures (Gruber and von Cramon 2001). In particular, while maintenance of phonological information again yielded strong activations along the anterior inter-

Fig. 2 Domain-speciﬁc distribution of working memory processes along human prefrontal and parietal cortices (Gruber and von Cramon 2001). The images in the upper row depict activations of frontoparietal networks that are associated with non-articulatory maintenance of (a) visual or (b) phonological information in working memory. (c) shows signiﬁcant domain-speciﬁc differences between these networks, in particular with predominant activation of the cortex along the anterior parts of the intermediate frontal sulcus (marked by black arrows) by phonological working memory (indicated in yellow and red), and of the cortex along posterior parts of the same frontal sulcus by visual working memory (indicated in blue and green)

114

O. Gruber

mediate frontal sulcus and the inferior parietal lobule, working memory for visual letter forms or colours preferentially activated more posterior prefrontal regions along the intermediate and superior frontal sulci as well as the superior parietal lobule (Fig. 2). This pattern of brain activations indicates that the prefrontal cortex along the intermediate frontal sulcus may be parcellated into anterior and posterior subdivisions that are differentially involved in phonological and visual working memory processes, respectively. Further, the fact that both phonological and visual working memory processes were distributed along identical neuroanatomical structures gave rise to the assumption that these brain structures may represent a multimodal working memory system whose subdivisions deal with different informational domains. Importantly, these ﬁndings obtained in humans that were experimentally deprived of their articulatory verbal working memory mechanism are consistent with data from numerous animal studies which suggested a similar anterior–posterior segregation of domain-speciﬁc working memory processes in monkeys. For example, Romanski et al. (1999) reported evidence that the cortex along the posterior part of the principal sulcus may be involved in visuospatial and possibly also in auditory-spatial processing, whereas the cortex along the anterior part of the principal sulcus may subserve non-spatial auditory and probably also aspects of phonetic processing.

2.3 Domain-Speciﬁcity of Verbal and Visuospatial Working Memory The functional segregation of distinct prefrontal areas along a rostrocaudal axis could also be conﬁrmed in a third functional MRI study using verbal and visuospatial working memory tasks and corresponding interference tasks (Gruber and von Cramon 2003). In this study, the ﬁnding could be replicated that distinct anterior and posterior subregions of the lateral prefrontal cortex together with other, particularly parietal association cortices form domain-speciﬁc functional networks that underlie non-articulatory phonological and visuospatial working memory performance. These bilateral prefrontoparietal networks could be differentiated from a left-hemispheric network of predominantly premotor and intraparietal areas which underlies the verbal rehearsal mechanism that probably is unique to the human species (Fig. 3).

2.4 An Evolutionary-Based Neuroanatomical Model of Human Working Memory On the basis of these empirical data and in consideration of the functionalneuroanatomical homologies observed in monkeys, an evolutionary-based functionalneuroanatomical model of working memory has been proposed according to which

The Neural Implementation of Working Memory

115

Articulatory Rehearsal

Non-articulatory Maintenance of Phonological Information

Visuospatial Working Memory

Fig. 3 Neural networks underlying different components of human working memory (Gruber and von Cramon 2003)

human working memory consists of two different and at least partially dissociable neural systems that interact with each other (Gruber and Goschke 2004). A presumably phylogenetically older, multimodal working memory system, which is also present in non-human primates, is implemented by several domain-speciﬁc prefrontoparietal and prefrontotemporal networks (depicted on the left-hand side of Fig. 4). On the other hand, a second system, which probably developed later in the context of the evolution of language, is supported by mostly left-hemispheric speech areas (depicted on the right-hand side of Fig. 4) and mediates explicit verbal rehearsal. This model appears promising in that it may not only offer new explanations for many behavioural, neuropsychological and neuroimaging ﬁndings in human subjects, but it also permits the harmonization of the conﬂicting working

116

O. Gruber

Fig. 4 An evolutionary-based neuroanatomical model of human working memory (Gruber and Goschke 2004). Verbal rehearsal is considered to be the most efﬁcient and predominant working memory mechanism in humans which can be accessed via recoding mechanisms and which operates independently of the original stimulus modality. It is neurally implemented by the mainly left-hemispheric network of premotor and parietal cortical areas, which is depicted on the righthand side (the brain is viewed from the top). These brain areas are also associated with human speech, which suggests that this brain system has developed in the context of the evolution of human language. A probably phylogenetically older working memory system which can also be found in non-human primates is topographically organized along parallel prefrontoparietal and prefrontotemporal circuits according to different informational domains, e.g. visuospatial, visual object and auditory-phonological information (see the images on the left-hand side)

memory models that were derived from human and animal research (Baddeley and Hitch 1974; Goldman-Rakic 1996).

2.5 Convergent Evidence from Studies in Patients with Focal Brain Lesions In order to further corroborate this evolutionary-based functional-neuroanatomical model of human working memory and to demonstrate the speciﬁc functional signiﬁcance of some of the brain regions that were activated during the working memory tasks, we subsequently performed investigations in patients with focal brain lesions. In particular, we were interested in two patients, one of whom showed an isolated left-sided lesion in Broca’s area, whereas the other exhibited a bilateral lesion along anterior parts of the middle frontal gyrus. According to the functionalneuroanatomical model outlined above, these patients should reveal speciﬁc deﬁcits to either the articulatory or the non-articulatory maintenance of verbal information in the sense of a double dissociation. In fact, this hypothesis could be conﬁrmed when comparing these patients with age-matched control subjects (Gruber et al.

The Neural Implementation of Working Memory

117

2005). While the patient with the lesion to Broca’s area exhibited a marked deﬁcit in articulatory rehearsal, but performed normally during non-articulatory maintenance of verbal information, the other patient suffering from the bifrontopolar lesion revealed exactly the opposite pattern, with normal performance in articulatory rehearsal, but impairment of non-articulatory maintenance of verbal information. Thus, these results corroborated the distinct and speciﬁc functional signiﬁcance of both Broca’s area and the anterior parts of the middle frontal gyrus for the integrity of the articulatory and the non-articulatory component of verbal working memory, respectively.

3 The Neural Implementation of the “Working” Component of Working Memory The evolutionary-based model of human working memory depicted in Fig. 4 merely summarizes the acquired knowledge about neural systems that underlie the pure maintenance of diverse information in working memory. Of course, further processing and manipulation of information that is kept in mind is an even more important key aspect of the working memory function, and many researchers have addressed this functional component of working memory particularly by using functional neuroimaging. It is important to note, however, that manipulation of sequential information held in working memory always requires simultaneous maintenance of parts of the same information. Thus, there is a methodological problem in properly determining brain activity related to manipulation itself since it is unclear whether the neuronal mechanisms of memory maintenance during concurrent manipulation are identical to those in the absence of additional processes. Because – as outlined above – two different brain systems may subserve verbal working memory depending on whether articulatory mechanisms are available to assist memory performance or whether they are needed for other processes such as concurrent articulations, it is conceivable that previous functional neuroimaging studies which attempted to dissociate the neural correlates of maintenance and manipulation of information may have been confounded by interference effects. Therefore, in a follow-up study we investigated the neural implementation of manipulation and maintenance processes in greater detail by systematically varying working memory load, manipulation demands (i.e. sequential reordering of the information) and the presence or absence of interference due to articulatory suppression. Direct comparisons between these different tasks permitted more precise conclusions about the neural correlates of manipulation processes in working memory. In this study, we again replicated the ﬁndings from the previous investigations showing that a bilateral network of anterior-prefrontal and inferior-parietal brain regions is involved in non-articulatory maintenance of phonological information during articulatory suppression. The same regions were also activated during the manipulation tasks, suggesting that manipulation demands may produce interference effects on brain activation associated with verbal working memory in a similar way as articulatory suppression does. Direct comparisons between the manipulation and the articulatory suppression conditions

118

O. Gruber

Maintenance of verbal information without interference

Maintenance of verbal information under articulatory suppression

Manipulation of information in verbal working memory Maintenance of visuospatial information

Fig. 5 The neural implementation of the “working” component of human working memory. Brain regions that are involved in maintenance functions of working memory in a domain-speciﬁc way together support the manipulation (i.e. sequential reordering) of (serial) verbal information in working memory, suggesting this “working” component of working memory may rely on functional interactions between (parts of) the maintenance networks

revealed additional bilateral activations along the posterior part of the superior frontal sulcus, along the intraparietal sulcus and in the precuneus, i.e. these activations could be attributed more speciﬁcally to manipulation processes involved in the sequential reordering (i.e. manipulation) of verbal information in working memory. Overall, at least three networks of brain regions were active during the manipulation tasks (Fig. 5). These three networks were strikingly similar to activation patterns that are also evoked by (1) rehearsal of verbal information (e.g. Broca’s area and left premotor cortex), (2) non-articulatory maintenance of phonological information (bilateral anterior-prefrontal and inferior-parietal brain regions) and (3) visuospatial working memory (bilateral posterior-superior-prefrontal and intraparietal cortex). Thus, this study provided evidence that executive processes such as manipulation of information in working memory may rely to a greater part on the same neuronal resources, i.e. brain systems that can also subserve pure maintenance of information in working memory. These ﬁndings are consistent with the hypothesis put forward by Goldman-Rakic (1996) that processes or phenomena that were attributed to the “central executive” by Alan Baddeley may not be implemented by a separate neurofunctional system, but may instead emerge from functional interactions between brain systems that support more elementary cognitive operations such as maintenance of information in working memory.

The Neural Implementation of Working Memory

119

4 Dynamic Interactions Between Neural Systems Underlying Different Components of Verbal Working Memory In order to gain further insight into possible dynamic functional interactions between the two complementary brain systems of human verbal working memory, we used the psychophysiological interaction approach (Friston et al. 1997) to test the hypothesis that, during inner speech, increased activity in brain regions underlying the articulatory rehearsal mechanism (i.e. left ventral premotor cortex and Broca’s area) would be linked to reduced activity of the non-articulatory (inner ear) system. This hypothesis was derived from the observation that during intensive verbal rehearsal of four letters in a delayed-recognition task this latter neural system was virtually inactive (Gruber 2001; Gruber and von Cramon 2001, 2003; Figs. 1 and 3). In fact, the left ventral premotor cortex, which was strongly activated during articulatory rehearsal (inner speech) in working memory (Fig. 6a), exhibited a negative, i.e. presumably inhibitory functional interaction, with several prefrontal and parietal cortical areas that were already known to be involved in non-articulatory maintenance of phonological information, i.e. the “inner ear” mechanism of verbal working memory (compare Fig. 6b with Figs. 1 and 3). Broca’s area, on the other hand, revealed negative coupling only with activity in one other brain region in the ventral anterior prefrontal cortex (bilaterally). Thus, these ﬁndings provided evidence for dynamic and task-related functional interactions between the two neural subsystems that together make up the dual architecture of human verbal working memory.

Ventral premotor cortex

Broca´s area

a

b

Fig. 6 Functional interactions between neural systems underlying different components of verbal working memory (Gruber et al. 2007). (a) Brain activations associated with the articulatory rehearsal component of verbal working memory and (b) negative psychophysiological interactions of the left ventral premotor cortex during articulatory rehearsal. The increasing neural activity in the left ventral premotor cortex shows signiﬁcantly enhanced negative (presumably inhibitory) coupling with activity in brain regions that are involved in the non-articulatory component of verbal working memory

120

O. Gruber

5 Summary and a Hypothesis About Complex Thinking in Humans In this chapter, recent empirical evidence from functional neuroimaging and experimental neuropsychology has been outlined which indicates that: (1) Working memory in humans is represented by (at least) two brain systems which differ from each other with respect to their functional-neuroanatomical organization and probably also with respect to their evolutionary origin. (2) There is a dual architecture of verbal working memory which is represented by two (at least partly) dissociable systems, a left-lateralized premotor-parietal network underlying articulatory rehearsal (inner speech) and a bilateral anteriorprefrontal/inferior-parietal network subserving non-articulatory maintenance of phonological information. (3) The striking similarity of the functional-neuroanatomical organization of nonarticulatory phonological and visual (both visual-object and visual-spatial) working memory along parallel prefrontoparietal and prefrontotemporal cortical networks suggests that these networks may be parts of a multimodal working memory system which presumably is phylogenetically older, because it can also be found in non-human primates. (4) More complex, “working” processes of working memory such as the sequential reordering of verbal information may not be attributable to the function of an additional brain system, but may rather emerge from functional interactions between brain systems that underlie more basic operations. Thus, from these ﬁndings one may hypothesize that dynamic functional interactions between the different neural systems that support elementary cognitive processes like verbalization, visual-spatial and visual-constructive processing, selective attention and mental imagery may also form the neural basis of complex thinking processes in humans.

References Adcock RA, Constable RT, Gore JC, Goldman-Rakic PS (2000) Functional neuroanatomy of executive processes involved in dual-task performance. Proc Natl Acad Sci U S A 97:3567–3572 Awh E, Jonides J (1998) Spatial selective attention and spatial working memory. In: Parasuraman R (ed) The attentive brain. MIT Press, Cambridge, pp 353–380 Awh E, Jonides J, Smith EE, Schumacher EH, Koeppe RA, Katz S (1996). Dissociation of storage and rehearsal in verbal working memory. Psychol Sci 7:25–31 Baddeley A (1992) Working memory. Science 255:556–559 Baddeley AD, Hitch GJ (1974) Working memory. In: Bower G (ed) Recent advances in learning and motivation, vol VIII. Academic, New York, pp 47–90 Baddeley A, Lewis V, Vallar G (1984) Exploring the articulatory loop. Q J Exp Psychol A 36:233–252 Becker JT, Morris RG (1999). Working memory(s). Brain Cogn 41:1–8

The Neural Implementation of Working Memory

121

Bunge SA, Klingberg T, Jacobsen RB, Gabrieli JD (2000) A resource model of the neural basis of executive working memory. Proc Natl Acad Sci U S A 97:3573–3578 Bunge SA, Ochsner KN, Desmond JE, Glover GH, Gabrieli JD (2001) Prefrontal regions involved in keeping information in and out of mind. Brain 124:2074–2086 Courtney SM, Ungerleider LG, Keil K, Haxby JV (1996) Object and spatial visual working memory activate separate neural systems in human cortex. Cereb Cortex 6:39–49 Courtney SM, Petit L, Maisog JM, Ungerleider LG, Haxby JV (1998) An area specialized for spatial working memory in human frontal cortex. Science 279:1347–1351 D’Esposito M, Detre JA, Alsop DC, Shin RK, Atlas S, Grossman M (1995) The neural basis of the central executive system of working memory. Nature 378:279–281 D’Esposito M, Postle BR, Rypma B (2000) Prefrontal cortical contributions to working memory: evidence from event-related fMRI studies. Exp Brain Res 133:3–11 Friston KJ, Buechel C, Fink GR, Morris J, Rolls E, Dolan RJ (1997) Psychophysiological and modulatory interactions in neuroimaging. Neuroimage 6:218–229 Gathercole SE (1994) Neuropsychology and working memory: a review. Neuropsychology 8(4):494–505 Goldman-Rakic PS (1996) The prefrontal landscape: implications of functional architecture for understanding human mentation and the central executive. Philos Trans R Soc Lond Ser B 351:1445–1453 Gruber O (2001) Effects of domain-speciﬁc interference on brain activation associated with verbal working memory task performance. Cereb Cortex 11:1047–1055 Gruber O, Goschke T (2004) Executive control emerging from dynamic interactions between brain systems mediating language, working memory and attentional processes. Acta Psychol 115:105–121 Gruber O, von Cramon DY (2001) Domain-speciﬁc distribution of working memory processes along human prefrontal and parietal cortices: a functional magnetic resonance imaging study. Neurosci Lett 297:29–32 Gruber O, von Cramon DY (2003) The functional neuroanatomy of human working memory revisited – evidence from 3T-fMRI studies using classical domain-speciﬁc interference tasks. Neuroimage 19:797–809 Gruber O, Gruber E, Falkai P (2005) Neural correlates of working memory deﬁcits in schizophrenic patients. Ways to establish neurocognitive endophenotypes of psychiatric disorders. Radiologe 45:153–160 Gruber O, M¨uller T, Falkai P (2007) Dynamic interactions between brain systems underlying different components of verbal working memory. J Neural Transm 114(8):1047–1050 Haxby JV, Petit L, Ungerleider LG, Courtney SM (2000) Distinguishing the functional roles of multiple regions in distributed neural systems for visual working memory. Neuroimage 11:380–391 Hecker R, Mapperson B (1997) Dissociation of visual and spatial processing in working memory. Neuropsychologia 35(5):599–603 Jonides J, Smith EE, Koeppe RA, Awh E, Minoshima S, Mintun MA (1993) Spatial working memory in humans as revealed by PET. Nature 363:623–625 Klingberg T (1998) Concurrent performance of two working memory tasks: potential mechanisms of interference. Cereb Cortex 8:593–601 Murray DJ (1968) Articulation and acoustic confusability in short-term memory. J Exp Psychol 78:679–684 Nystrom LE, Braver TS, Sabb FW, Delgado MR, Noll DC, Cohen JD (2000) Working memory for letters, shapes, and locations: fMRI evidence against stimulus-based regional organization in human prefrontal cortex. Neuroimage 11:424–446 Paulesu E, Frith CD, Frackowiak RSJ (1993) The neural correlates of the verbal component of working memory. Nature 362:342–344 Postle BR, Berger JS, Taich AM, D’Esposito M (2000) Activity in human frontal cortex associated with spatial working memory and saccadic behavior. J Cogn Neurosci 12:2–14

122

O. Gruber

Romanski LM, Tian B, Fritz J, Mishkin M, Goldman-Rakic PS, Rauschecker JP (1999) Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nat Neurosci 2:1131–1136 Washburn DA, Astur RS (1998) Nonverbal working memory of humans and monkeys: rehearsal in the sketchpad? Mem Cognit 26:277–286

Current Perspectives on Imaging Language Joseph T. Devlin

Abstract Functional neuroimaging has been highly successful in mapping anatomical regions involved in language processing although these regions rarely, if ever, correspond to cognitively deﬁned a priori expectations. For instance, traditional notions of orthography and phonology have no simple neuroanatomic correlates and instead may emerge from dynamic interactions across multiple brain regions. Thus, a challenge for neurolinguists is to move beyond simply mapping cognitive functions onto neuroanatomy and towards a systematic understanding of the neural information processing underlying language. Here, I illustrate this process in the domain of skilled reading and attempt to highlight a set of imaging tools which facilitate the process. The results demonstrate how cognitive theories and neurobiological investigations can be mutually informative and lead to novel explanations framed in terms of neural information processing.

1 Introduction In 1861, Pierre Paul Broca presented a short paper to the Soci´et´e Anthropologique in Paris that forever changed thinking about mental functions. He described a single patient whose ability to understand language was largely intact but who essentially could not produce speech. Despite considerable, widespread damage to this patient’s brain, Broca concluded that his speech deﬁcit resulted from pronounced damage to the posterior part of the third frontal convolution in the left hemisphere (Broca 1861). In doing so, he demonstrated for the ﬁrst time that a higher-order cognitive function, namely the faculty of articulated language, could be localised to a particular part of the brain. Although the concept of cerebral localisation was not new (Bell 1811; Legallois 1812), the idea that cognitive functions could be localised J.T. Devlin Department of Psychology, University College London, Gower Street, London, WC1E 6BT, UK [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

123

124

J.T. Devlin

Fig. 1 The classic neurological model of language. The cognitive model is shown on the left and each centre has a 1:1 correspondence with a speciﬁc left hemisphere cortical territory, shown on the right. Hearing speech involved acoustic signals (a) reaching the posterior superior temporal gyrus (red), where they were matched to auditory word forms. These, in turn, were linked to articulatory, or motor, word forms (m) located in the inferior frontal gyrus (blue). Reading words involved essentially the same system except that visual information (v) came into the brain via the occipital pole (light blue) and reached the angular gyrus (green), where they were matched to visual word forms. The representations linked to the auditory word forms in Wernicke’s area and then to motor word forms in Broca’s area. Although the cognitive model includes a centre for meaning, it had no neuroanatomical correlate when this model was proposed

(Gall and Spurzheim 1810) was widely discredited (Flourens 1842). Nevertheless, Broca’s ﬁnding was inﬂuential and similar patients bolstered his claims (Broca 1865). The identiﬁcation of patients with other types of language deﬁcits following more posterior lesions soon led to the development of the ﬁrst neurological model of language (Dejerine 1892; Lichtheim 1885; Wernicke 1874; Fig. 1). Critics of this type of “localisationism” noted that many patients with speech production deﬁcits did not have damage to Broca’s area, while in some others, damage to Broca’s area did not lead to impairments of articulated speech (Brown-Sequard 1877; Marie 1906). Even the theoretical basis of lesion-deﬁcit mapping was challenged by noting that symptoms are not the same as functions and that focal damage can have both proximal and distal consequences (Head 1920; Hughlings-Jackson 1884). These arguments were so persuasive, that 100 years later they still dominated the types of inferences being drawn from brain-damaged patients (Caramazza and McCloskey 1988; Shallice 1979). Although there were notable exceptions (Geschwind 1965; Ojemann 1979; Wada and Rasmussen 1960), most language research rejected a close correspondence between language functions and cortical territories and focused instead on the functional, or cognitive, architecture with no reference to underlying anatomy. The advent of non-invasive methods for measuring brain function (i.e. neuroimaging) in the later part of the twentieth century brought a renewed interest in the neural processes underlying language. Like the earliest neurological studies, the initial neuroimaging work focused on mapping language functions to speciﬁc cortical territories and some of this research is summarised in Fig. 2. The ﬁgure highlights two important points. First, it is apparent that language is not limited to the classic areas of Broca and Wernicke – in fact, it engages most neocortical territories in both hemispheres as well as many subcortical regions. Second, there does not appear to be even a single brain region dedicated to language – indeed, all of these

Current Perspectives on Imaging Language

125

Fig. 2 Summary of the neuroimaging studies that map language functions onto neuroanatomy. These are displayed on a single subject’s structural scan that has been inﬂated and automatically segmented in cortical regions using Freesurfer (Desikan et al. 2006). Each region is colour-coded according to the broad type of information processing attributed to it and more than one colour indicates multiple functions. For instance, the angular gyrus is coloured red–yellow, illustrating that both semantic and orthographic functions are attributed to it

regions are also involved in non-linguistic functions relating to cognitive control, memory, attention, perception, or action. These two points raise a major challenge for neurolinguistic research, namely the need to move beyond simply mapping cognitive functions onto neuroanatomy (i.e. brain mapping) and towards a systematic understanding of the neural information processing underlying language. The aim of this chapter is to demonstrate preliminary steps in this direction. Although I will focus on skilled reading as an illustrative example, hopefully it will be apparent how these same techniques can be applied more generally to investigate both linguistic and non-linguistic functions. A second aim is highlight how cognitive theories and neurobiological investigations can be mutually informative and lead to novel explanations framed in terms of neural information processing.

2 Visual Word Forms Written language is an important cultural invention that makes it possible to store otherwise ephemeral spoken signals. Its advantages are the ability to create a moreor-less permanent store of information and the potential to reach a much wider audience. Without writing, it seems unlikely that Shakespeare’s plays would still be in production or that the Harry Potter novels could ever have reached nearly a billion people. The cost of this ability, however, comes in learning to read, which unlike other aspects of language, is neither natural nor easy. Reading involves recognising visual symbols (i.e. written words) and transforming them into their corresponding linguistic information. The ﬁrst model of reading

126

J.T. Devlin

(Dejerine 1892) contained separate centres for visual, auditory, and spoken word forms as well as meaning (Fig. 1) and this basic segregation of information remains fundamental to most modern models of reading (Coltheart et al. 2001; Plaut et al. 1996). A hallmark of skilled reading is the ability to quickly and accurately recognise visual words independent of their physical characteristics (e.g. font, size, colour, retinal position, etc.) and it is this abstract representation of visual word form (typically termed “orthography”) that maps onto sounds and meaning. According to Dejerine (1891), visual word forms were stored in the left angular gyrus because damage to this region interfered with both the ability to recognise visually presented words and the ability to write them, despite otherwise intact language skills. Visual words forms were linked to auditory word forms located in the left posterior superior temporal gyrus (i.e. Wernicke’s area) and from there to articulatory motor patterns necessary for speech located in the left inferior frontal gyrus (i.e. Broca’s area) (Dejerine 1891, 1892; Wernicke 1874). In the classic neurological model, each component was identiﬁed on the basis of lesion-deﬁcit correspondences and the system was inferred from many different patients. In contrast, functional neuroimaging offers the ability to identify a system of brain areas simultaneously, although the contributions from individual areas are difﬁcult to interpret. For instance, a similar network of brain regions including Broca’s and Wernicke’s areas is revealed when reading words aloud is compared with a baseline condition such as resting or viewing a ﬁxation cross (Bookheimer et al. 1995; Petersen et al. 1989; Price et al. 1994, 1996a; Rumsey et al. 1997), but there are also important differences. Critically, in studies of reading the angular gyrus is not consistently activated, whereas a region of the posterior fusiform gyrus is (Price 2000). From these and similar ﬁndings, a number of different locations have been proposed as the location of the “visual word form area”, including the angular (Horwitz et al. 1998), supramarginal (Bookheimer et al. 1995), middle temporal (Howard et al. 1992), lingual (Petersen et al. 1989), or fusiform (Cohen et al. 2000) gyrus. Many different interpretations are possible because the comparison of reading and rest includes a number of cognitive processes, not just visual word recognition. Ideally, what is needed is a more tightly controlled comparison that subtracts activation due to the sound and meaning of the word, leaving only visual word recognition. Unfortunately, these processes often occur implicitly, making it nearly impossible to separate them by means of a simple subtraction design (Friston et al. 1996; Price et al. 1996b) and requiring alternative methods for dissecting the system. One such method is functional MRI (fMRI) adaptation, which Cohen et al. (2002) used to investigate visual word recognition speciﬁcally. Adaptation paradigms rely on the ﬁnding that neurons respond less strongly to a repeated presentation of a stimulus than to a novel stimulus (Desimone 1996). In fMRI, this manifests itself as a reduction in blood oxygen level dependent signal (Grill-Spector et al. 2006) and can be used as a tool for differentiating different levels of processing (Henson et al. 2000; Vuilleumier et al. 2002). In their experiment, Cohen et al. (2002) sought to distinguish early visual areas sensitive to the physical properties of the stimulus from higher visual areas which ignore these details and represent

Current Perspectives on Imaging Language

127

the stimulus more abstractly (e.g. a “ROSE” is a “rose”). They used a repetition priming task in which two words were either the same (radio–radio) or different (carrot–tractor) and could be in either the same or different cases (radio–RADIO). They observed a repetition suppression effect in a region of the left posterior fusiform gyrus that was case-insensitive, suggesting that this region processed visual words at an abstract (or case-invariant) level and referred to it as the “visual word form area”. Speciﬁcally, they argued that the neurons in the region encode commonly occurring letter pairs, often called “bigrams” (Grainger and Jacobs 1996), which explained why both words and pronounceable non-words (often called “pseudowords”) activate the region during normal reading (Mechelli et al. 2003). That is, neurons with receptive ﬁelds for “po”, “or”, and “rt” would collectively encode “port”, while similar neurons could encode pseudowords such as “hort” (i.e. “ho”, “or”, and “rt”). These representations would then feed into visual word detectors which would interface with non-visual properties of the stimulus such as its sound and meaning (Dehaene et al. 2005). Importantly, the use of fMRI adaptation offered a principled method for mapping cognitive accounts onto neural anatomy – in this case, mapping visual word forms as open bigrams onto the left posterior fusiform gyrus. Adaptation can also be used to test the neural validity of the cognitive account by critically examining two key claims: whether neurons in the posterior fusiform gyrus represent bigrams and whether these representations are purely orthographic. If words and pseudowords are represented by bigram detectors, then case-invariant repetition can be explained by repeated activation of the same neuronal population that recognises letter pairs. Devlin et al. (2006) argued that if true, precisely the same mechanism should apply for pseudoword repetition. Using a similar repetition priming experiment, they replicated the case-independent reduction in activation for repeated words (Cohen et al. 2002) but found that repeated pseudowords showed a slight increase in activation in the left posterior fusiform (see also Fiebach et al. 2005). These ﬁndings question the validity of bigram detectors and suggest important differences between the representations of words and pseudowords in this area. The most obvious possibility is that neuronal activity in the posterior fusiform is not limited to purely orthographic information but is also inﬂuenced by non-visual factors such as meaning. To explore this possibility, Devlin et al. (2006) again used a priming paradigm but manipulated the form and meaning relations between prime– target pairs to probe activation in the left posterior fusiform region. Like repeated words, pairs sharing visual form (e.g. corner–corn) showed a signiﬁcant neural priming effect, whereas synonyms (e.g. idea–notion) did not. In addition, there was a signiﬁcant interaction, with words that shared both form and meaning (e.g. teacher– teach) showing reduced priming relative to those which only shared form. This suggests that although visual form appears to be the primary factor driving posterior fusiform activations, it is nonetheless modulated by meaning and indicates that representations in the region cannot be purely orthographic. Another way to test this hypothesis is by use of a conjunction design (Price and Friston 1997). The basic idea is to look for activation common to a set of comparisons with the assumption that common activations must reﬂect common

128

J.T. Devlin

Fig. 3 Theory and application of a conjunction design. (a) This Venn diagram illustrates three independent contrasts (A > B, C > D, and E > F) whose intersection is shown in white. At a cognitive level, the intersection (Ω ) represents common information processing demands present in each contrast, while at a neural level this represent common activations. This is illustrated in (b), where four contrasts are shown: (1) picture naming versus saying “ok” to a scrambled image, (2) picture naming versus reading the same word aloud, (3) action relative to size decisions on common objects, and (4) action relative to size decisions on novel nonsense objects. In each case, there was signiﬁcant activation in the left posterior fusiform gyrus in a region previously described as the “visual word form area”. (Adopted from Price and Devlin 2003, copyright Elsevier, permission pending)

information processing (Fig. 3). For example, Price and Devlin (2003) found that activation in the left posterior fusiform gyrus that had been previously attributed to orthographic processes during reading was also found when subjects named pictures of objects such as a tiger. Moreover, the activation in this area was greater for naming the picture of a tiger than for reading the word “tiger”, ruling out implicit

Current Perspectives on Imaging Language

129

orthographic activation as a likely explanation (Moore and Price 1999; Wright et al. 2007). In fact, activation was also present when individuals demonstrated how to manipulate common objects (such as a broom) relative to judging the size of the image on the screen, eliminating the need to name the stimulus at all (Phillips et al. 2002). Indeed, even manipulations of novel objects which could not be named showed activation in this same region. Together these ﬁndings argue strongly against the notion that left posterior fusiform activation reﬂects activation speciﬁc to reading such as orthography and instead suggests a more general role for the region. Namely, this portion of extrastriate cortex seems likely to be involved in integrating visual form information with non-visual properties of the stimulus such as associated meanings, sounds, or actions (Devlin et al. 2006; Moore and Price 1999; Price and Friston 2005). While this type of processing is critical for reading, it is in no way speciﬁc to it and raises two important points. First, the notion of orthographic representations is essential to virtually all cognitive models of reading but may be incorrect. Instead of reading-speciﬁc visual codes, the neural evidence suggests a more general mechanism for representing visual form that is used by reading but also by other linguistic and non-linguistic functions. Second, for integration to take place requires interactions with other areas beyond the posterior fusiform gyrus.

3 Beyond Visual Word Forms If there is one clear ﬁnding from neuroanatomy it is that brain regions do not function in isolation; each region has a dense pattern of proximal and distal connections, generating a richly interconnected network (Passingham et al. 2002; Young and Scannell 2000). Consequently, neural information processing is a dynamic process that plays out over both space and time. One implication is that modular information encapsulation as proposed by Fodor (1983) does not occur anywhere in the brain, including primary sensorimotor cortices (Kosslyn et al. 2001; Pulvermuller et al. 2005). Another is that complex skills such as reading involve the coordination of many brain regions working in tandem. The signature of this interregional cooperation can be found in temporal correlations among neurophysiological events in anatomically distant regions and is often referred to as “functional connectivity” (Friston 2002). Functional connectivity studies of reading consistently demonstrate strong links between the posterior fusiform region and more anterior temporal lobe areas, the inferior parietal lobe, and distinct rostral and caudal parts of the left inferior frontal gyrus (Bitan et al. 2005; Bokde et al. 2001; Kujala et al. 2007). Mechelli et al. (2005) further demonstrated that the strength of these couplings was modulated by word type. Words with irregular spellings (e.g. “quay”) increased the coupling between the fusiform gyrus and pars orbitalis, an anterior and ventral part of Broca’s area, while pseudowords (e.g. “prant”) increased the coupling between the fusiform and pars opercularis, a posterior and dorsal region of Broca’s area. Interestingly, a similar rostrocaudal division of labour has been previously reported in the left

130

J.T. Devlin

inferior frontal gyrus, with tasks focused on the meaning of words preferentially activating pars orbitalis, while the sounds of words preferentially activate pars opercularis (Buckner et al. 1995; Demonet et al. 1992; Devlin et al. 2003; Gitelman et al. 2005; Gough et al. 2005; Poldrack et al. 1999; Price et al. 1997). Mechelli et al. (2005) argued that consistent with these ﬁndings, irregularly spelled words place a greater emphasis on semantic processing and therefore increase the coupling between the fusiform and pars orbitalis. Pseudowords rely on spelling-to-sound conversion, which places a greater emphasis on phonological processing and increases the coupling between fusiform and pars opercularis. In other words, the authors provided the ﬁrst neuroanatomical evidence for multiple reading routes – one predominantly semantic and the other predominantly phonological – consistent with modern cognitive models of reading (Coltheart et al. 2001; Plaut et al. 1996). How are these functional linkages implemented anatomically? Functional connectivity by itself cannot answer this question. Diffusion tensor imaging (DTI) tractography, however, provides a method for non-invasively tracing anatomical pathways in the brain. Using diffusion-weighted MRI, one can obtain a 3D measure of the diffusivity of water which indexes the local grey matter–white matter structure in each voxel of the brain (Le Bihan et al. 1989). From this, it is possible to infer the existence of anatomical pathways potentially linking two or more areas together (Behrens et al. 2003; Parker 2004). An important limitation is that these paths are not evidence for a physiologically active connection – only an anatomically viable pathway – and therefore provide complementary information to functional connectivity. Converging results from the two techniques, however, suggest an underlying neural circuit that is both anatomically plausible and functionally signiﬁcant. Gough et al. (2005) hypothesised that the functional double dissociation between semantic and phonological processing in left inferior frontal gyrus reﬂects distinct corticocortical association bundles linking the two parts of Broca’s area with different temporal lobe structures. Speciﬁcally, they argued that the uncinate fasciculus links pars orbitalis with the anterior temporal poles, an area associated with semantic memory processing (Hodges et al. 1995; Nobre and McCarthy, 1995; Vandenberghe et al. 1996). In contrast, the superior longitudinal fasciculus links pars opercularis and ventral premotor cortex with the temporoparietal junction, an area of auditory association cortex implicated in phonological processing (Hickok and Poeppel 2007; Scott and Wise 2004). Both of these white matter paths are easily identiﬁed using DTI tractography (Catani et al. 2005; Parker et al. 2005) so we used this technique to investigate the speciﬁcity of their terminations within Broca’s area (J.T. Devlin and K.E. Watkins, unpublished data). We found that one end of the uncinate terminated in pars orbitalis and the ventral segment of pars triangularis but not in either pars opercularis nor ventral premotor cortex. The opposite pattern was seen for the superior longitudinal pathway, with terminations in pars opercularis and ventral premotor cortex, but nothing more anterior in ventral pars triangularis nor in pars orbitalis (Fig. 4). In addition, we found a set of U-ﬁbres linking pars orbitalis with pars triangularis and another linking pars triangularis with pars opercularis, consistent with invasive tracing studies in macaques (Petrides and Pandya 2002).

Current Perspectives on Imaging Language

131

Fig. 4 Tractography results. For each path, a set of four prefrontal areas were deﬁned as target regions, including ventral premotor cortex (yellow), pars opercularis (blue), pars triangularis (red), and pars orbitalis (green). In addition, the tractography algorithm was seeded from a region outlined by the dashed orange line. (a) For the superior longitudinal fasciculus, the seed region was the tissue immediately posterior to the Sylvian ﬁssure including the supramarginal gyrus and temporoparietal junction shown in the leftmost image. The three slices to the right show voxels in the seed region colour-coded according to the prefrontal regions they linked most strongly with. The strongest paths linked the supramarginal gyrus and ventral premotor cortex but there were also paths linking the temporoparietal junction with pars opercularis and dorsal pars triangularis. No paths were found to either ventral triangularis or pars orbitalis. (b) For the uncinate fasciculus, the seed region included all of the temporal lobe anterior to Heschl’s gyrus. Here, the results demonstrate two sets of paths, one linking the anterior superior temporal gyrus and pars orbitalis and ventral pars triangularis and the other linking the ventral temporal pole with pars orbitalis and pars triangularis. No paths were found linking the anterior temporal lobe with pars opercularis or ventral premotor cortex. (c) Finally, to look for U-ﬁbre links between prefrontal regions, pars triangularis was seeded. The dorsal region demonstrated clear links with pars opercularis, while the ventral portion linked with pars orbitalis. (J.T. Devlin and K.E. Watkins, unpublished data)

Presumably this set of frontotemporal paths provide an anatomical substrate for integration of bottom-up semantic and phonological information in prefrontal cortex and also are important when exerting top-down control of these two types of temporal lobe processing.

132

J.T. Devlin

How do these frontotemporal components connect with the posterior fusiform gyrus during reading? From Mechelli et al. (2005) we know that there are at least two different routes. The connectivity of the dorsal, or non-semantic, path is unclear. Classically it was believed that the inferior longitudinal fasciculus connected the posterior fusiform to the angular gyrus (Dejerine 1892) and from there information ﬂowed through the single (dorsal) route shown in Fig. 1. Recently we designed a DTI tractography study to look for an anatomical link between the posterior fusiform gyrus and the inferior parietal lobe but failed to ﬁnd any evidence for such a path (Devlin and Price 2007). Instead, the tractography suggested that projections to both areas from an early visual region located in the middle occipital gyrus may drive the functional connectivity observed during reading (Horwitz et al. 1998; Kujala et al. 2007). If correct, then a further set of projections from the inferior parietal lobe presumably drives the functional link with pars opercularis via the superior longitudinal fasciculus. Because this functional connectivity is strongest when reading pseudowords, it suggests the path plays a role in computing spelling-to-sound correspondences, typically operationalised as translating from orthographic to phonological representations. I have argued, however, that the neurobiological evidence is inconsistent with dedicated orthographic representations and a similar argument applies to dedicated phonological representations. There is insufﬁcient space to fully justify this claim, but it sufﬁces to say that evidence for brain activity speciﬁc to phonology or speech is equivocal (Price et al. 2003, 2005). Instead, phonological processes commonly engage a system of regions including auditory association areas in the vicinity of the temporoparietal junction and articulatory motor patterns in ventral premotor regions (Fadiga et al. 2002; Watkins and Paus 2004; Wilson et al. 2004), suggesting that phonology may be an emergent property of sensorimotor interactions (Liberman et al. 1967; Liberman and Mattingly 1985). Unlike the dorsal path, the connectivity of the ventral, or semantic, pathway is fairly clear. The inferior longitudinal fasciculus links posterior parts of the fusiform gyrus with the ventral surface of the anterior temporal poles (Catani et al. 2003; Devlin and Price 2007), which in turn are linked to pars orbitalis via the uncinate. In other words, the ventral temporal pole mediates the functional link between the fusiform gyrus and pars orbitalis during reading and this is entirely consistent with claims of a semantic path. Although the speciﬁc regional contributions are unknown, it is generally accepted that large parts of the lateral and ventral temporal lobes – especially the temporal poles – contribute to semantic memory (Mummery et al. 1999; Spitsyna et al. 2006). This combination of functional and anatomical connectivity data suggests a somewhat different account of reading than either the classic neurological model (Dejerine 1892) or modern cognitive models (Coltheart et al. 2001; Plaut et al. 1996). By all accounts, there is an important link between representations of the sounds of words and their motor patterns, presumably implemented in the connections linking Wernicke’s and Broca’s areas. This system is functionally linked with visual form representations in the posterior fusiform region and together forms the non-semantic reading route. Similarly, all accounts posit a separate semantic reading

Current Perspectives on Imaging Language

133

Fig. 5 A neurocognitive framework for reading. The components of the semantic route are shown in red and those of the non-semantic route are shown in yellow. The posterior fusiform region contributes to both routes. Sensorimotor areas are shown in light blue and include visual input, motoric output, and auditory cortex engaged by hearing one’s own voice. Anatomical connections between regions are shown as white, bidirectional arrows. ILF inferior longitudinal fasciculus, SLF superior longitudinal fasciculus, UF uncinate fasciculus

route, which appears to correspond to an anatomical path linking posterior to anterior temporal lobe regions. To date there is no neurobiological evidence for a lexical route, central to the misleadingly named “dual route” reading models (which actually have three routes – a non-semantic orthography-to-phonology, a lexical, and a semantic route; Coltheart et al. 1993). Nor does the neurobiological evidence support the notion of dedicated orthographic and phonological representations. Instead, it suggest that both arise from more general visual and sensorimotor representations which are not dedicated to reading, speaking, or even language. Although this distinction may seem fussy, it has important ramiﬁcations which I will return to shortly. Figure 5 summarises the neurocognitive framework for reading laid out in this chapter. In its current form it is clearly incomplete. I have fallen into the common left-hemisphere-cortical-chauvinist trap and omitted many anatomical regions, including (but not limited to) the right hemisphere and subcortical components important for language (Fig. 2) that contribute to at least some aspects of reading (Crinion et al. 2006; Fiez and Petersen 1998). In addition, the inferior occipitofrontal fasciculus is missing even though there is evidence to suggest that it directly links the posterior fusiform gyrus with Broca’s area (Catani et al. 2002; Curran 1909; Makris et al. 2007). Ideally, this framework can be extended to incorporate these and other neurocognitive components and develop a complete model of reading. An important step in this direction would be to implement this framework as a formal (i.e. mathematical or computational) model in order to test its ability to simulate important behavioural and biological aspects of reading. In addition, it would help to ensure the stability and consistency of the framework when additional components are proposed by ensuring they did not fundamentally change the basic dynamics of the computation. The current proposal is similar in many respects to the “triangle model” of reading (Plaut et al. 1996; Seidenberg and McClelland

134

J.T. Devlin

1989), although any implementation would require at least slightly more anatomical and physiologically realistic models of cortical territories and their connectivity (O’Reilly 1998; Welbourne and Lambon Ralph 2005). One advantage would be a more ﬁne-grained representation of spatiotemporal processing that could then be evaluated against data from electroencephalography and magnetoencephalography (Dhond et al. 2007; Maurer et al. 2005; Salmelin and Kujala 2006). In addition, I have argued the need to reconsider the nature of the phonological and orthographic representations. In many ways, this may be the most theoretically important point because better representations can often overcome limitations of a model and lead to unexpected insights. For example, the original Wickelfeatures used to encode phonological and orthographic information in Seidenberg and McClelland’s (1989) model led to important inaccuracies in the model (Besner et al. 1990; Coltheart et al. 1993) which were resolved by adopting simpler but more powerful representations (Plaut et al. 1996). The basic neurocognitive framework presented here hopefully provides an initial step towards developing such a model.

4 Conclusions The goal of this chapter was to introduce a series of imaging tools to assist in the process of bringing cognitive and neurobiological investigations closer together. Ultimately, questions of cortical topography such as “Where in the brain is function Ψ?” provide only broad constraints on cognitive theories, in part because inferring a cognitive process from activation in a particular brain region is problematic (Poldrack 2006). More sophisticated methods are necessary for testing cognitive theories, on the one hand, and the nature of the neural information processing, on the other. Adaptation and conjunctions are two such methods. In the examples described here, adaptation was used to test whether a region of the posterior fusiform gyrus was sensitive to abstract visual word forms as predicted by cognitive models and then also used to further explore the speciﬁc nature of the processing in that region, with the results feeding back to the cognitive level and suggesting modiﬁcations to the nature of “orthographic” representations. While adaptation and conjunctions are primarily useful for testing functional hypotheses about a speciﬁc brain region, functional and anatomical connectivity studies provide complementary information at the systems level. Functional interactions between regions provide insights into the dynamic ﬂow of information which again constrain cognitive theories. And although the temporal resolution is poor relative to other that of techniques such as magnetoencephalography (Pammer et al. 2004) or transcranial magnetic stimulation (Pascual-Leone and Walsh 2001), functional connectivity studies offer the ability to measure both spatial and temporal information and these interactions provide a vital clue to the basic building blocks of cognition (Price and Friston 2005).

Current Perspectives on Imaging Language

135

References Behrens TE, Woolrich MW, Jenkinson M, Johansen-Berg H, Nunes RG, Clare S, Matthews PM, Brady JM, Smith SM (2003) Characterization and propagation of uncertainty in diffusionweighted MR imaging. Magn Reson Med 50(5):1077–1088 Bell C (1811) Idea of a new anatomy of the brain; submitted for the observations of his friends. Strahan and Preston, London Besner D, Twilley L, McCann R, Seergogin K (1990) On the connection between connectionism and data: Are a few words necessary? Psychol Rev 97:432–446 Bitan T, Booth JR, Choy J, Burman DD, Gitelman DR, Mesulam MM (2005) Shifts of effective connectivity within a language network during rhyming and spelling. J Neurosci 25(22): 5397–5403 Bokde AL, Tagamets MA, Friedman RB, Horwitz B (2001) Functional interactions of the inferior frontal cortex during the processing of words and word-like stimuli. Neuron 30(2):609–617 Bookheimer SY, Zefﬁro TA, Blaxton T, Gaillard W, Theodore W (1995) Regional cerebral blood ﬂow during object naming and word reading. Hum Brain Mapp 3:93–106 Broca P (1861) Remarques sur le si`ege de la facult´e du langage articul´e; suivies d’une observation d’aphemie. Bull Soc Anat Paris 6:330–357 Broca P (1865) Sure la siege de la faculte langage articule. Bull Soc Anthropol 6:377–396 Brown-Sequard CE (1877) Aphasia as an effect of brain disease. Dublin J Med Sci 63:209–225 Buckner RL, Raichle ME, Petersen SE (1995) Dissociation of human prefrontal cortical areas across different speech production tasks and gender groups. J Neurophysiol 74(5):2163–2173 Caramazza A, McCloskey M (1988) A case for single-patient studies. Cogn Neuropsychol 5:583–623 Catani M, Howard RJ, Pajevic S, Jones DK (2002) Virtual in vivo interactive dissection of white matter fasciculi in the human brain. Neuroimage 17(1):77–94 Catani M, Jones DK, Donato R, Ffytche DH (2003) Occipito-temporal connections in the human brain. Brain 126(9)2093–2107 Catani M, Jones DK, ffytche DH (2005) Perisylvian language networks of the human brain. Ann Neurol 57(1):8–16 Cohen L, Dehaene S, Naccache L, Lehericy S, Dehaene-Lambertz G, Henaff MA, Michel F (2000) The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain 123:291–307 Cohen L, Lehericy S, Chochon F, Lemer C, Rivard S, Dehaene S (2002) Language-speciﬁc tuning of visual cortex? Functional properties of the visual word form area. Brain 125:1054–1069 Coltheart M, Curtis B, Atkins P, Haller M (1993) Models of reading aloud: dual-route and paralleldistributed-processing approaches. Psychol Rev 100:589–608 Coltheart M, Rastle K, Perry C, Langdon R, Ziegler J (2001) DRC: a dual route cascaded model of visual word recognition and reading aloud. Psychol Rev 108(1):204–256 Crinion J, Turner R, Grogan A, Hanakawa T, Noppeney U, Devlin JT, Aso T, Urayama S, Fukuyama H, Stockton K, Usui K, Green DW, Price CJ (2006) Language control in the bilingual brain. Science 312(5779):1537–1540 Curran EJ (1909) A new association ﬁber tract in the cerebrum. J Comp Neurol 19(6):645–662 Dehaene S, Cohen L, Sigman M, Vinckier F (2005) The neural code for written words: a proposal. Trends Cogn Sci 9(7):335–341 Dejerine J (1891) Sur un cas de cecite verbale avec agraphie, suivi d’autopsie. Mem Soc Biol 3:197–201 Dejerine J (1892) Contribution a l’etude anatomoclinique et clinique des differentes varietes de cecite verbal. C R Hebd Sceances Mem Soc Biol 4:61–90 Demonet J-F, Chollet F, Ramsay S, Cardebat D, Nespoulous J-L, Wise R, Rascol A, Frackowiak R (1992). The anatomy of phonological and semantic processing in normal subjects. Brain 115:1753–1768

136

J.T. Devlin

Desikan RS, Segonne F, Fischl B, Quinn BT, Dickerson BC, Blacker D, Buckner RL, Dale AM, Maguire RP, Hyman BT, Albert MS, Killiany R J (2006) An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. Neuroimage 31(3):968–980 Desimone R (1996) Neural mechanisms for visual memory and their role in attention. Proc Natl Acad Sci USA 93(24):13494–13499 Devlin JT, Price CJ (2007) Anatomical connectivity of the posterior fusiform gyrus. Paper presented at the Cognitive Neuroscience Society, New York Devlin JT, Matthews PM, Rushworth MF (2003) Semantic processing in the left inferior prefrontal cortex: a combined functional magnetic resonance imaging and transcranial magnetic stimulation study. J Cogn Neurosci 15(1):71–84 Devlin JT, Jamison HL, Gonnerman LM, Matthews PM (2006) The role of the posterior fusiform in reading. J Cogn Neurosci 18(6):911–922 Dhond RP, Witzel T, Dale AM, Halgren E (2007) Spatiotemporal cortical dynamics underlying abstract and concrete word reading. Hum Brain Mapp 28(4):355–362 Fadiga L, Craighero L, Buccino G, Rizzolatti G (2002) Speech listening speciﬁcally modulates the excitability of tongue muscles: a TMS study. Eur J Neurosci 15(2):399–402 Fiebach CJ, Gruber T, Supp GG (2005) Neuronal mechanisms of repetition priming in occipitotemporal cortex: spatiotemporal evidence from functional magnetic resonance imaging and electroencephalography. J Neurosci 25(13):3414–3422 Fiez JA, Petersen SE (1998) Neuro-imaging studies of word reading. Proc Natl Acad Sciences USA 95:914–921 Flourens MJP (1842) Recherches exp´erimentales sur les propri´et´es et les fonctions du syst`eme nerveux dans les animaux vert´ebr´es, 2nd edn. Bailliard, Paris Fodor J (1983) The modularity of mind. MIT, Cambridge Friston KJ (2002) Functional integration and inference in the brain. Prog Neurobiol 68(2):113–143 Friston KJ, Price CJ, Fletcher P, Moore C, Frackowiak RSJ, Dolan RJ (1996) The trouble with cognitive subtraction. Neuroimage 4(2):97–104 Gall FJ, Spurzheim J (1810) Anatomie et physiologie du syst`eme nerveux en g´en´eral, et du cerveaux en particulier. Schoell, Paris Geschwind N (1965) Disconnection syndromes in animals and man. Brain 88:237–294 Gitelman DR, Nobre AC, Sonty S, Parrish TB, Mesulam MM (2005) Language network specializations: an analysis with parallel task designs and functional magnetic resonance imaging. Neuroimage 26(4):975–985 Gough PM, Nobre AC, Devlin JT (2005) Dissociating linguistic processes in the left inferior frontal cortex with transcranial magnetic stimulation. J Neurosci 25(35):8010–8016 Grainger J, Jacobs AM (1996) Orthographic processing in visual word recognition: a multiple read-out model. Psychol Rev 103(3):518–565 Grill-Spector K, Henson R, Martin A (2006) Repetition and the brain: neural models of stimulusspeciﬁc effects. Trends Cogn Sci 10(1):14–23 Head H (1920) Studies in neurology. Hodder and Stroughton, London Henson R, Shallice T, Dolan R (2000) Neuroimaging evidence for dissociable forms of repetition priming. Science 287(5456):1269–1272 Hickok G, Poeppel D (2007) The cortical organization of speech processing. Nat Rev Neurosci 8(5):393–402 Hodges JR, Graham N, Patterson K (1995) Charting the progression in semantic dementia: Implications for the organization of semantic memory. Memory 3(3/4):463–495 Horwitz B, Rumsey JM, Donohue BC (1998) Functional connectivity of the angular gyrus in normal reading and dyslexia. Proc Natl Acad Sci USA 95(15):8939–8944 Howard D, Patterson K, Wise R, Brown WD, Friston K, Weiller C, Frackowiak R (1992) The cortical localization of the lexicons. Positron emission tomography evidence. Brain 115 (6): 1769–1782 Hughlings-Jackson J (1884) Evolution and dissolution of the nervous system. Br Med J 1:591, 660, 703

Current Perspectives on Imaging Language

137

Kosslyn SM, Ganis G, Thompson WL (2001) Neural foundations of imagery. Nat Rev Neurosci 2:635–642 Kujala J, Pammer K, Cornelissen P, Roebroeck A, Formisano E, Salmelin R (2007) Phase coupling in a cerebro-cerebellar network at 8–13 Hz during reading. Cereb Cortex 17(6):1476–1485 Le Bihan D, Turner R, MacFall JR (1989) Effects of intravoxel incoherent motions (IVIM) in steady-state free precession (SSFP) imaging: application to molecular diffusion imaging. Magn Reson Med 10(3):324–337 Legallois J-C (1812) Exp´eriences sur la principe de la vie, notamment sur celui des movements du coeur, et sur le si`ege de ce principe. Hautel, Paris Liberman AM, Mattingly IG (1985) The motor theory of speech perception – revised. Cognition 21:1–36 Liberman AM, Cooper FS, Shankweiler D, Studdert-Kennedy M (1967) Perception of the speech code. Psychol Rev 74:431–461 Lichtheim L (1885) On aphasia. Brain 7:433–484 Makris N, Papadimitriou GM, Sorg S, Kennedy DN, Caviness VS, Pandya DN (2007) The occipitofrontal fascicle in humans: a quantitative, in vivo, DT-MRI study. Neuroimage 37(4): 1100–1111 Marie P (1906) La troisieme circonvolution frontale gauche ne joue aucun role special dans la fonction du language. Sem Med 26:241–247 Maurer U, Brandeis D, McCandliss BD (2005) Fast, visual specialization for reading in English revealed by the topography of the N170 ERP response. Behav Brain Funct 1:13 Mechelli A, Gorno-Tempini ML, Price CJ (2003) Neuroimaging studies of word and pseudoword reading: consistencies, inconsistencies, and limitations. J Cogn Neurosci 15(2):260–271 Mechelli A, Crinion JT, Long S, Friston KJ, Lambon Ralph MA, Patterson K, McClelland JL, Price CJ (2005) Dissociating reading processes on the basis of neuronal interactions. J Cogn Neurosci 17(11):1753–1765 Moore CJ, Price CJ (1999) Three distinct ventral occipitotemporal regions for reading and object naming. Neuroimage 10(2):181–192 Mummery CJ, Patterson K, Wise RJ, Vandenbergh R, Price CJ, Hodges JR (1999) Disrupted temporal lobe connections in semantic dementia. Brain 122:61–73 Nobre AC, McCarthy G (1995) Language-related ﬁeld potentials in the anterior-medial temporal lobe: II. Effects of word type and semantic priming. J Neurosci 15(2):1090–1098 Ojemann GA (1979) Individual variability in cortical localization of language. J Neurosurg 50(2):164–169 O’Reilly RC (1998) Six principles for biologically based computational models of cortical cognition. Trends Cogn Sci 2(11):455–462 Pammer K, Hansen PC, Kringelbach ML, Holliday I, Barnes G, Hillebrand A, Singh KD, Cornelissen PL (2004) Visual word recognition: the ﬁrst half second. Neuroimage 22(4): 1819–1825 Parker GJ (2004) Analysis of MR diffusion weighted images. Br J Radiol 77(Spec No 2):S176–185 Parker GJ, Luzzi S, Alexander DC, Wheeler-Kingshott CA, Ciccarelli O, Lambon Ralph MA (2005) Lateralization of ventral and dorsal auditory-language pathways in the human brain. Neuroimage 24(3):656–666 Pascual-Leone A, Walsh V (2001) Fast backprojections from the motion to the primary visual area necessary for visual awareness. Science 292:510–512 Passingham RE, Stephan KE, Kotter R (2002) The anatomical basis of functional localization in the cortex. Nat Rev Neurosci 3(8):606–616 Petersen SE, Fox PT, Posner MI, Mintun M, Raichle ME (1989) Positron emission tomographic studies of the processing of single words. J Cogn Neurosci 1(2):153–170 Petrides M, Pandya DN (2002) Comparative cytoarchitectonic analysis of the human and the macaque ventrolateral prefrontal cortex and corticocortical connection patterns in the monkey. Eur J Neurosci 16(2):291–310 Phillips JA, Humphreys GW, Noppeney U, Price CJ (2002) The neural substrates of action retrieval: an examination of semantic and visual routes to action. Vis Cognit 9(4):662–684

138

J.T. Devlin

Plaut D, McClelland J, Seidenberg M, Patterson K (1996) Understanding normal and impaired word reading: computational principles in quasi-regular domains. Psychol Rev 103:56–115 Poldrack RA (2006) Can cognitive processes be inferred from neuroimaging data? Trends Cogn Sci 10(2):59–63 Poldrack RA, Wagner AD, Prull MW, Desmond JE, Glover GH, Gabrieli JD (1999) Functional specialization for semantic and phonological processing in the left inferior prefrontal cortex. Neuroimage 10(1):15–35 Price CJ (2000) The anatomy of language: contributions from functional neuroimaging. J Anat 197:335–359 Price CJ, Devlin JT (2003) The myth of the visual word form area. Neuroimage 19(3):473–481 Price CJ, Friston KJ (1997) Cognitive conjunction: a new approach to brain activation experiments. Neuroimage 5(4 Pt 1):261–270 Price CJ, Friston KJ (2005).Functional ontologies for cognition: the systematic deﬁnition of structure and function. Cogn Neuropsychol 22(3–4):262–275 Price CJ, Wise RJ, Watson JD, Patterson K, Howard D, Frackowiak RS (1994) Brain activity during reading. The effects of exposure duration and task. Brain 117(6):1255–1269 Price CJ, Moore CJ, Frackowiak RS (1996a) The effect of varying stimulus rate and duration on brain activity during reading. Neuroimage 3(1):40–52 Price CJ, Wise RJS, Frackowiak RSJ (1996b) Demonstrating the implicit processing of visually presented words and pseudowords. Cereb Cortex 6(1):62–70 Price CJ, Moore CJ, Humphreys GW, Wise RJ (1997) Segregating semantic from phonological processes during reading. J Cogn Neurosci 9(6):727–733 Price CJ, Winterburn D, Giraud AL, Moore CJ, Noppeney U (2003) Cortical localisation of the visual and auditory word form areas: a reconsideration of the evidence. Brain Lang: 272–286 Price CJ, Thierry G, Grifﬁths T (2005) Speech-speciﬁc auditory processing: where is it? Trends Cogn Sci 9(6):271–276 Pulvermuller F, Hauk O, Nikulin VV, Ilmoniemi RJ (2005) Functional links between motor and language systems. Eur J Neurosci 21(3):793–797 Rumsey JM, Nace K, Donahue BC, Wise D, Maisog JM, Andreason P (1997) A positron emission tomographic study of impaired word recognition and phonological processing in dyslexic men. Archiv Neurol 54:562–573 Salmelin R, Kujala J (2006) Neural representation of language: activation versus long-range connectivity. Trends Cogn Sci 10(11):519–525 Scott SK, Wise RJ (2004) The functional neuroanatomy of prelexical processing in speech perception. Cognition 92(1–2):13–45 Seidenberg M, McClelland J (1989) A distributed, developmental model of word recognition and naming. Psychol Rev 96:523–568 Shallice T (1979) Case study approach in neuropsychological research. J Clin Neuropsychol 1(3):183–211 Spitsyna G, Warren JE, Scott SK, Turkheimer FE, Wise RJ (2006) Converging language streams in the human temporal lobe. J Neurosci 26(28):7328–7336 Vandenberghe R, Price CJ, Wise R, Josephs O, Frackowiak RSJ (1996) Functional anatomy of a common semantic system for words and pictures. Nature 383:254–256 Vuilleumier P, Henson RN, Driver J, Dolan RJ (2002) Multiple levels of visual object constancy revealed by event related fMRI of repetition priming. Nat Neurosci 5(5):491–499 Wada J, Rasmussen T (1960) Intracarotid injection of sodium amytal for the lateralization of cerebral speech dominance: experimental and clinical observations. J Neurosurg 17:266–282 Watkins KE, Paus T (2004) Modulation of motor excitability during speech perception: the role of Broca’s area. J Cogn Neurosci 16(6):978–987 Welbourne SR, Lambon Ralph MA (2005) Exploring the impact of plasticity-related recovery after brain damage in a connectionist model of single-word reading. Cogn Affect Behav Neurosci 5(1):77–92 Wernicke C (1874) Der aphasiche Symptomenkomplex. Cohen and Weigert, Wrocław

Current Perspectives on Imaging Language

139

Wilson SM, Saygin AP, Sereno MI, Iacoboni M (2004) Listening to speech activates motor areas involved in speech production. Nat Neurosci 7(7):701–702 Wright ND, Mechelli A, Noppeney U, Veltman DJ, Rombouts SA, Glensman J, Haynes JD, Price CJ (2007) Selective activation around the left occipito-temporal sulcus for words relative to pictures: individual variability or false positives? Hum Brain Mapp Young MP, Scannell JW (2000) Brain structure-function relationships: advances from neuroinformatics. Philos Trans R Soc Lond B Biol Sci 355(1393):3–6

Functional Neuroimaging and the Logic of Conscious and Unconscious Mental Processes Bal´azs Guly´as

Abstract With the advent of positron emission tomography, or PET, cognitive neuroscience research has been equipped with a unique research tool. At present, PET is the most versatile neuroimaging research tool, as it is capable of the quantitative mapping of physiological and biochemical parameters of the living human brain. With the help of PET one can map the active neuronal populations during cognitive processes and thereby decipher the logic of “brain activation”. This deciphering has, however, its stipulations due to the spatial and temporal constraints of the technique. During sensory, motor or higher cognitive processes, networks of cortical neuronal populations are active in the human brain. These networks consist of a modality-dependent “core network” which is complemented by “recruited ﬁelds”, the latter being task- or stimulus-dependent. The very same cortical populations may participate in the processing and analysis of various information content (convergence), whereas the processing and analysis of different contents may engage different cortical ﬁelds, cortical macronetworks (divergence). Visual paradigms have widely been used to explore the neuronal correlates of conscious and non-conscious visual information processing. A wide variety of paradigms have been tested using PET and the underlying cortical macronetworks have been identiﬁed. Whereas the neuronal correlates of conscious and non-conscious visual processes can be identiﬁed at the macronetwork level with PET, the “neuronal correlates of consciousness” still remain a conundrum.

1 Introduction Functional neuroimaging techniques entered the research battery of neurosciences over two decades ago. With the use of positron emission tomography (PET), functional magnetic resonance imaging (fMRI) or magnetoencephalography the B. Guly´as Karolinska Institutet, Department of Clinical Neuroscience, Psychiatry Section, 171 76 Stockholm, Sweden [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

141

142

B. Guly´as

active neuronal populations, responsible for sensory, motor, cognitive or emotional processes, are supposed to be identiﬁed, localised and visualised in the living human brain. The seemingly uncontested theory of brain imaging claims that with use of these techniques, the functional architecture of the human brain has been mapped during various sensory, motor or cognitive tasks. Can we indeed explore, interpret and understand the neurobiological basis of the human brain’s mental processes with imaging? Are these techniques helping us reveal the neurobiological underpinnings of cognitive processes? With the help of them, can we exploit the differences between conscious and unconscious brain processes? Is functional neuroimaging a royal way to understand brain function or do we build a new phrenology without understanding what we measure? This question was raised by us a few years ago (K´eri and Guly´as 2003). After more than two decades of the imaging revolution and years after our investigation into the core of the conundrum regarding what neuroimaging “sees”, more and more authors are asking the same question. What is the essence of the conundrum? One basic problem arises from the fact that brain activities are multidimensional and can be approached form various points of view, using different methods. For the “direct” measurements of brain activation blood ﬂow and metabolic changes, changes in the electrical activity of cells and cell populations, neurotransmitter dynamics, as well as other consequent biochemical, physiological and/or physical parameters (e.g. neuromagnetic changes) can be utilised (Fig. 1). When we use imaging techniques mainly based on blood ﬂow and metabolic measures, we must take into consideration that even during the simplest task all of these processes operate in a closely interacting manner. Therefore, before drawing ﬁnal conclusions about brain function from pure imaging data, we must clarify the exact relationship among the levels of neuronal organisation and function (Fig. 2).

Behaviour

Activity of neuronal assemblies

Cerebral blood flow and metabolism

Neurotransmitter dynamics

Fig. 1 The four facets of the functioning human brain: neuronal activity of cortical micronetworks (action potentials, ﬁeld potentials); neurotransmitter dynamics (transmitter release, binding, uptake, reuptake, modulatory effects, etc.); regional cerebral blood ﬂow and metabolism; behaviour (K´eri and Guly´as 2003)

Functional Neuroimaging and the Logic of Mental Processes

143

Action potentials, field potentials

Energy supply

Increased metabolism, Increased oxygen and Glucose consumption Increased blood flow

Functional neuroimaging

Neuronal communication

Neurotransmitter events

Sensory, motor, cognitive, emotional brain functions

Correlation with behaviour Fig. 2 The classical logic of brain activation. Neuronal information transmission processes are initiated by release of neurotransmitters, resulting in neuronal communication and action potentials. These all are energy-requiring processes. The outcome is brain activation in the sensory, motor, cognitive or emotional systems of the brain. The measurable output is a behavioural event. Functional neuroimaging can cover neurotransmitter events, metabolism-related and blood-ﬂow-related changes (K´eri and Guly´as 2003)

With the help of a wide variety of neuroimaging techniques, the spatial-temporal domain of the human brain is covered almost entirely (Fig. 3). The most widely used functional neuroimaging techniques, PET and fMRI, are based on their usefulness in the localisation of the active neuronal processes by using at least two approaches: (1) As active neuronal populations, participating in information processing in the brain, require more energy than those populations not directly involved in such processes at a given time, their glucose and oxygen consumption will be increased. This entails increased regional cerebral blood ﬂow in the active brain regions, which, in turn, can be localised by using PET or fMRI (Roland 1993). (2) The communication between neurons is primarily due to neurotransmission. The various neurotransmitter–neuroreceptor systems can be explored with PET by using radiolabelled ligands. This way, the “status” of the various communication systems between active neuronal populations can be identiﬁed and quantiﬁed (Halldin et al. 2001a,b; Guly´as et al. 2008). Despite the more than 10,000 publications to date on the localisation of cortical functions with functional neuroimaging techniques, a few caveats should be noted here. The origin of the imaging signal is not fully clariﬁed in either technique. It is accepted without question that neuronal activation, increased ﬁring at neuronal level, the essence of all brain processes, is an energy-requiring process that entails

144

B. Guly´as 2

1 CT MRI SPECT

PET

0

-1

-2

-3

-4 Lifetime

Lesions Autoradiography

Year

Light microscope Iontoforesis, microstimulation

Day EM

Hour

Videomicroscopy fMRI EEG ERP MEEG Brain Area

Second Voltage sensitive dyes Microelectrodes

Millisecond

Column Cell layer Cell Synaps

Fig. 3 The place of functional imaging techniques in the battery of neuroscience methods, covering the spatial-temporal domain of the human brain

increased brain metabolism, oxygen and glucose consumption and blood ﬂow. The increased neuronal activation and increased spiking frequency during information processing in the brain should, however, be compared with the spontaneous ﬁring rate and/or baseline activities of the neurons. Does ordered neuronal ﬁring necessarily require more energy, and consequently higher blood ﬂow and metabolism, than spontaneous ﬁring (Singer 1999)? The detailed contribution of, and the balance between, excitatory and inhibitory neuronal activities in pattern generation is not fully understood. The number of inhibitory interneurons signiﬁcantly exceeds the number of excitatory neurons, whereas the body size and axon length (along which the action potential should proceed) of the excitatory neurons (mainly pyramidal cells) signiﬁcantly surpass those of the interneurons. Consequently, the energy requirement of an excitatory neuron is higher than that of an inhibitory interneuron. What happens in neuronal pattern generation, though? What are the realistic proportions of activated interneurons and excitatory neurons, and what is the price of activation and inhibition? What is the real price of neuronal pattern generation during information processing in the brain? (Lennie 2003; Buzs´aki et al. 2007). These questions remain unanswered for the time being. Nevertheless, despite the shortcomings of our knowledge in these ﬁelds, we accept that functional neuroimaging is – if not the royal way – still a useful way towards a better understanding of brain functions, including higher-level non-conscious and conscious processes.

2 Functional Neuroimaging with PET The principle of PET can be traced back to Georg de Hevesy’s discovery that radioisotopes can be used as tracers of biological processes. In PET positron-emitting radionuclides such as 11 C, 13 N, 15 O or 18 F are incorporated into a biologically inert

Functional Neuroimaging and the Logic of Mental Processes

145

11C Scintillation crystal Photomultiplier

Detector channel β+ e−

Detector

Coincidence circuit (if t < x nsec, then)

Fig. 4 The physical basis of positron emission, positron–electron annihilation and coincidence detection with the help of a functionally coupled detector system

or biologically active molecule and administered to a living human or animal. Owing to the decay of the radionuclides, positrons are continuously generated. The positrons, when encountering an electron, are annihilated and along one axis in two directions two 511-keV γ photons leave the place of positron–electron annihilation (Guly´as and Sj¨oholm 2007). With an appropriate detector system, the generated γ photons can be detected. If we arrange the detectors on a ring or multiple rings and use some “coincidence logic” for detecting the γ photons, we can determine within which detector channel a positron–electron annihilation took place. If two photons are detected within, say, 5 ns in two geometrically appropriately conﬁgured detectors, then we accept one annihilation event within the “channel” connecting the surface of the two detectors (Fig. 4). The detectors contain a scintillation crystal in which a γ photon is transformed into a photon pertaining to the visible range, and a photomultiplier tube which ampliﬁes the incoming photon and emits an electric signal as a photon enters it from its crystal surface. The detectors are built into rings and the rings are housed in the scanner. In state-of-the-art scanners, each ring can contain several hundred, in special scanner types (e.g. HRRT) several thousand, detectors. The inner diameter of the rings determines the size of the gantry, which, in turn, determines whether the scanner can only be used for small animal studies, head-only studies in humans, or whole-body studies. The annihilation data, collected during PET data acquisition, are used for image reconstruction. Most commonly, a back-projection technique is applied to recover the original “annihilation maps”, representing the radioactivity distribution inside the detector ring of the scanner. However, other techniques (e.g. iterative) are also gaining popularity in more recent scanner types. The reconstructed images correspond to the original annihilation maps, which can, in turn, be transformed into

146

a

B. Guly´as

b

c

d

Fig. 5 Horizontal slices of the human brain representing regional cerebral blood ﬂow (a) (tracer, 15 O-butanol) and neuroreceptor or transporter density distributions: (b) dopamine D2 (ligand, 11 C-raclopride), (c) serotonin 5HT 11 1A (ligand, C-WAY100635) and (d) serotonin transporter (ligand, 11 C-MADAM) (Halldin et al. 2001a)

radioactivity values (e.g. nanocuries per milligram tissue). These values, with appropriate kinetic models and complementary measurements (e.g. arterial input function), can give rise to quantitative parametric biological maps corresponding to, for instance, receptor distribution values or regional blood ﬂow or glucose metabolism values (Fig. 5). Biologically inert tracers (e.g. 15 O-water or 15 O-butanol) can be used for tracing, for instance, blood ﬂow. Biologically active tracers can be used for measuring, for instance, glucose metabolism (ﬂuorodeoxyglucose) or mapping the human brain’s neuroreceptor composition and its changes (radioligands, e.g. 11 C-racliprode for the dopamine D2 receptors). The radiotracers are synthesised in a dedicated radiochemistry laboratory. A prerequisite of a radiochemistry laboratory with a large range of various tracers is a cyclotron, which produces the radionuclides. The most commonly used PET radionuclides are 11 C, 13 N, 15 O and 18 F, with half-lives of 20, 10, 2 and 110 min, respectively. These radionuclides or radioisotopes are often called bioisotopes, as they can rather easily be incorporated into biologically active molecules and their relatively low biological dose equivalents and short half-lives make their routine use in humans possible. PET radiochemistry is a special branch of chemistry. Owing to the rapid decay of the isotopes, a radiochemical synthesis must be quick and effective: it usually cannot exceed two to three half-lives of the isotope and the end product must contain at least 20% of the original radioactivity. The radiochemists should work under strict safety conditions. The products should be radiochemically and chemically clean, and should be sterile and pyrogen-free for human parenteral use. This whole process requires a complex laboratory setup and the concerted activities of different experts, ranging from physicists to physicians (Fig. 6). From the point of view of exploring the human brain’s higher functions, two tracer categories are of considerable importance: blood ﬂow tracers and CNS radioligands. Blood ﬂow tracers, for instance 15 O-water or 15 O-butanol, are inert chemicals which enter the bloodstream and circulate with the blood. The global

Functional Neuroimaging and the Logic of Mental Processes cyclotron

isotope production

radiochemistry quality control

tracer synthesis

administration

147 PET scanner

data sampling

image analysis

evaluation

Fig. 6 Major constituents of a positron emission tomography (PET) centre

cerebral blood ﬂow in primates, with special regard to humans, is relatively high: it is approximately 20% of the total circulation. Regional cerebral blood ﬂow is in direct proportion to neuronal activity. Cortical neuronal populations with increased neuronal activity require more oxygen, more glucose and, consequently, more blood. Increased regional cerebral blood ﬂow is therefore a direct indicator of increased neuronal activity in the brain. For this reason, regional cerebral blood ﬂow measurements with PET can be used to localise task-speciﬁc neuronal activity in the human brain. The technique is widely used and until the introduction of the fMRI blood oxygen level dependent technique it was the main methodological approach of functional localisation in neuroimaging.

3 PET and Neurotransmitter Studies Changes in the human brain’s neurotransmitter–neuroreceptor system can also be an important correlate of neuronal processes. As neuronal transmission is an essential part of information processing in the brain, the delicate balance between various neurotransmitter systems changes during brain activation, such as perception or cognition. Imaging studies of neuroreceptors in cognitive neuroscience can be especially useful in the exploration of, for instance, the human brain’s reward system or attentional system. Since the introduction of PET, one of the main applications of the technique has proved to be the detailed mapping of central neuroreceptor systems. Several hundred novel PET radioligands for various neuroreceptor systems have been developed, tested and validated during the past two decades. PET has also become an indispensable research tool in the development of novel CNS drugs, acting on neuroreceptors (Farde 1996; Halldin et al. 2001a,b). These studies have also contributed to our better understanding of the human brain receptor structure and the alterations in the various neuroreceptor systems. Parallel with the build-up of our PET tracer repertoire and the development of new models to characterise ligand–receptor interactions, there is an increasing need to develop theories for understanding the correlative behaviour of various receptor

148

B. Guly´as

systems too. The analysis of the normal status and maturation of the brain’s receptor systems provides us with crucial information on understanding normal brain functions. Alterations in the ﬁne-tuning between the various systems may result in neurological and psychiatric diseases, which could be better understood by a more detailed exploration of the receptor systems. The study of the correlative behaviour of neuroreceptor systems can also be the starting point for creating advanced diagnostic classiﬁcations and designing novel CNS drugs and therapeutic approaches.

4 The Human Brain’s Receptor Fingerprint Similarly to the facts that each and every person can be identiﬁed solely by the pattern of ridges and furrows on the surface on their ﬁngertips, and each and every person can be identiﬁed uniquely by the sequence of the base pairs in his or her DNA (DNA ﬁngerprint), the human brain has a highly individual receptor ﬁngerprint. Each and every brain’s receptor composition is somewhat different, despite the fact that, naturally, the basis of the human brain’s receptor composition is determined genetically and there are no differences in the fundamental arrangement of neurotransmitter and neuroreceptor systems between individuals. What differs is the ﬁne-tuning of the delicate balance between the various receptor and transmitter systems, and this balance (1) is highly individual and (2) changes continuously during life. Through use of post-mortem brain imaging techniques, mainly autoradiographic studies with receptor ligands, a burgeoning new research ﬁeld has been initiated to study the detailed receptor ﬁngerprints of individual brain regions, originally classiﬁed by Brodmann on the basis of histological differences. These studies have by now resulted in a novel receptor architectural classiﬁcation of the human brain and contribute to our better understanding of the brain’s structure–function relationship (Geyer et al. 1997, 1998; Zilles and Palomero-Gallagher 2001; Zilles et al. 2002, 2004; Bozkurt et al. 2005; Scheperjans et al. 2005a,b; Morosan et al. 2005; Hurlemann et al. 2005; Eickhoff et al. 2007). The study of the development of receptor systems across evolution may provide us with fundamental information about the mechanisms of brain development (Pilpel et al. 1998; Ekstrom et al. 2003; Ache and Young 2005). In the study of the vertebrate brain, and more speciﬁcally the primate brain, the receptor ﬁngerprint in different species may contribute to a more versatile and detailed picture of the evolution of the brain. The stability and variability of cortical areal receptor patterns across species can explain important facts about the human brain. The intraindividual analysis of the brain’s receptor architecture’s temporal changes during morphogenesis, maturation and ageing may reveal fundamental information about normal brain development and shed light on the various main “paths” along which individual brains change during life under physiological conditions. These complex temporal development patterns can then be compared with patterns of pathological conditions, including the neuropsychiatric diseases,

Functional Neuroimaging and the Logic of Mental Processes

149

long-term drug treatment with CNS medicines, drug addiction and addictive behaviour. Short-term challenges, including physiological and pharmacological ones, can also modify the brain’s receptor balance. The understanding of the relationship between short-term challenges and the resulting receptor balance alterations is a vital question in brain research. The interindividual analysis of brain receptor ﬁngerprints can shed light on differences in, among others, personality proﬁles, proneness to disease conditions, cognitive styles and behavioural differences between individuals. It can identify those brain regions which show more stability across a larger population and identify other brain regions responsible to a larger extent for the individual variability of brains. The impact of environmental inﬂuences (interpersonal, social or physical) on the brain’s receptor ﬁngerprint also needs a detailed interindividual analysis of receptor systems. Finally, a meticulous comparison between diseased brains (neurological or psychiatric diseases, addictive traits, etc.) and healthy controls can provide us with a vista into the nature of the diseases of the human brain.

5 Receptor Fingerprints, Personality and Cognition Since the introduction of Robert Cloninger’s pioneering model (Cloninger 1986, 1987, 2000; Cloninger et al. 1993), the correlation between the human brain’s individual receptor systems and various dimensions of human personality, temperament and character has been intensively studied. It has been clearly demonstrated, for instance, that the personality trait of “sociability” or detachment, appearing in extrovert and introvert social behaviour, closely correlates with the density of striatal dopamine D2 receptors (Farde et al. 1997; Breier et al. 1998; Jonsson et al. 2003) as well as reward seeking and novelty seeking (Ekelund et al. 1999; Okuyama et al. 2000; Suhara et al. 2001; Van Gestel et al. 2002; Rogers et al. 2004; Cohen et al. 2005). But the correlation of personality traits and other monoamine systems has also been explored recently. For instance, the serotonin system is known to be involved in challenge and sensation seeking behaviour (Netter et al. 1996; Gerra et al. 1999; Peirson et al. 1999; Hennig et al. 2000), fear conditioning (Garpenstrand et al. 2001), suicidal tendencies (Baud 2005; Bondy et al. 2006) and harm avoidance (Van Gestel et al. 2002; Park et al. 2004; Sz´ekely et al. 2004), whereas the norepinephrine (noradrenaline) system may be included in personality traits such as reward dependence and novelty seeking (Sara et al. 1995; Garvey et al. 1996; Samochowiec et al. 2002). More and more recent studies have aimed at the understanding of the human brain’s neurotransmitter systems, with special regard to the balance between the different neurotransmitters, and their correlation with personality traits and dimensions. The correlation between the brain’s receptor ﬁngerprint and cognitive functions or “cognitive styles” is now a subject of research which hopefully will bear fruit in the near future.

150

B. Guly´as

6 PET in Cognitive Neuroscience Research Before the advent of fMRI, PET was the most sophisticated imaging technique used for the mapping of sensory, motor or cognitive processes in the human brain. The “classical” paradigm used the early period of neuroimaging was the subtraction paradigm. The idea can be traced back to the Dutch physiologist and ophthalmologist Cornelius Franciscus Donders in the nineteenth century when he applied “sequential” stimulation paradigms to “decompose” the neurophysiological basis of complex brain functions with superimposed components. In modern subtraction paradigms two experimental situations, say A and B, are presented and the undergoing neuronal activities are measured with a neuroimaging technique. The two paradigms differ from each other in only one stimulus dimension, one stimulus feature. The hypothesis behind the subtraction paradigm states that the difference in brain activation measurements, displayed on the PET or fMRI images, highlights those neuronal populations which are responsible for the processing and analysis of the given stimulus feature. For instance, in the case of colour vision the presentation of the same image in full colour and in monochromatic grey results in different regional cerebral blood patterns, the difference between which may indicate those neuronal populations which are highly dedicated to the analysis of colour information (Fig. 7). The human brain is never in a real resting condition and each sensory, voluntary or cognitive process causes perturbation on an already existing activation state. The consideration of this fact is built into other, more complex, multifactorial paradigm designs, used nowadays in functional neuroimaging studies. The analysis of imaging data can be based upon different approaches. With use of a data-driven approach, fundamental statistical tests (t test, analysis of variance, analysis of covariance, etc.)

a

b c

Fig. 7 The subtraction paradigm: visual scenes (a, b) differing from each other in only one stimulus dimension (colour) result in PET images of regional cerebral blood ﬂow distribution, the subtraction of which would result in an image displaying the neuronal populations engaged by the processing and analysis of colour information (c) (Guly´as and Roland 1991, 1994a)

Functional Neuroimaging and the Logic of Mental Processes

151

are applied to the image data sets, usually comprising a large number of volume elements, or voxels. This approach is not biased with respect to preliminary working hypothesis or observer expectations. Hypothesis-driven approaches can also have legitimacy under various conditions, especially when expectations from certain anatomical regions are evident. The most commonly used analysis of PET images, similarly to that of other neuroimaging modalities, is based upon the general linear model, one of the most important and widely used statistical models applicable to biological and social datasets. The most commonly used version of the general linear model is statistical parametric mapping (Friston et al. 2007).

7 The Logic of Brain Operations 7.1 Convergence and Divergence One of the leading questions at the beginning of the functional neuroimaging epoch was the involvement of one or more cortical neuronal populations in the processing and analysis of simple brain tasks, such as a perceptual task. Theoretically, the status quaestionis of this issue can be traced back to the age-old localisation problem in neurology: one cortical region, one brain function. The extreme version of the hypothesis suggests that one brain cell in higher cortical regions is specialised for one highly deﬁned function, e.g. the recognition of the observer’s grandmother (“the grandmother cell”; Gross 1992). Of course, as complex brain functions cannot be bound to one single neuron, the moderate version of the hypothesis is inclined to state that certain well-deﬁned brain functions and the function of wellcircumscribed, anatomically coherent, neuronal populations correlate. A classic example is the “colour area of man”: a well-deﬁned region in the human fusiform gyrus which, according to the original protagonists of the hypothesis, speciﬁcally underlies colour perception (Lueck et al. 1989; Zeki et al. 1991). However, detailed neuroimaging studies on simple perceptual or motor functions indicate that even the simplest perceptual or motor tasks engage a number of cortical neuronal populations (Guly´as and Roland 1991, 1994a, b; Claeys et al. 2004). For instance, in the case of visual perception, the incoming visual information is distributed in the visual cortex and a number of cortical regions are engaged in the processing and analysis of the various visual submodalities of the visual image before a uniﬁed and integrated visual percept is generated. A body of literature indicates clearly that for the processing of even the simplest perceptual, motor or cognitive tasks a network of neuronal populations is activated, i.e. functional networks – and not single cortical areas – are activated. This is even true for the processing and analysis of the simplest visual submodalities, such as colour, form or disparity (Figs. 8 and 9). The analysis of a large number of imaging (and other) studies in the sensory ﬁelds indicates that the incoming information’s processing diverges: the information is reaching, either by parallel or by serial channels, a number of cortical neuronal

152

B. Guly´as

C D E F

a

B

b

c

d

e

f

Fig. 8 Divergence. Cortical ﬁelds activated by the processing and analysis of colour information. Meta-analysis of four colour detection related PET experiments. The horizontal image slices in panels (b–f) intersect the image plans indicated in (a). (Guly´as and Roland, unpublished data)

Functional Neuroimaging and the Logic of Mental Processes

153

Fig. 9 Divergence: Cortical areas participating in colour perception, measured with PET and functional magnetic resonance imaging (fMRI) (Claeys et al. 2004)

populations which participate in the processing and analysis of the given sensory information. This can be referred to as the divergence principle (Guly´as 2001a). The question then is the following: Can the same neuronal populations which form a part of a functional network X form a part of another functional network Y? Can and do the same cortical regions participate in different functional networks, underlying different perceptual, motor or cognitive processes in the brain? Or, in other terms, are the very same neuronal populations multifunctional and can they participate in the processing and analysis of various tasks? Again, comparative analysis of several functional neuroimaging studies with sensory, motor or cognitive functions indicates that different functional macronetworks may include the very same cortical regions. For instance, we have shown that cortical areas involved in colour and disparity processing or the analysis of spatial frequency and orientation information may be congruent at the macronetwork level. This fact, however, does not necessarily mean that at the cellular level the very same neuronal populations are active as, for instance, in blobs and/or interblobs of the primary visual cortex (V1 or Brodmann area 17) neurons specialised for various tasks (orientation, disparity, colour, etc.) may overlap with each other and it is possible that during the processing of one or another visual submodality only a fraction of the neurons in a cortical region are active predominantly (Guly´as and Roland 1994a, b, 1995; Fig. 10). The fact that during the processing and analysis of a sensory information the same “hubs” in various cortical macronetworks can be involved indicates that convergence is also a key phenomenon in cortical information processing. The very same cortical ﬁelds may participate in the processing and analysis of various types of information, be they sensory, motor, cognitive, etc. This can be referred to as the convergence principle. In short, the basic logic of information processing in the human brain can be traced back to two elementary principles: divergence and convergence. By divergence we mean that the very same information reaches various cortical neuronal

154

B. Guly´as

Fig. 10 Convergence. Cortical ﬁelds activated by the processing and analysis of colour (a) and disparity (b) information. The area of overlap of the neuronal populations participating in both tasks in the occipital cortex is indicated by a dotted ellipse. (From Guly´as et al. 1994)

populations and it is processed and analysed by divergent cortical neuronal populations. By convergence we mean that the very same cortical neuronal population may participate in the processing and analysis of various cortical information processes, both bottom-up and top-down.

7.2 Core Networks and Recruited Fields Cognitive tasks may vary according to their complexities. Some tasks are simple and therefore require relatively little brain work, whereas complex tasks may extensively use the human brain’s processing capacity. It sounds quite natural that during the processing of, say, a simple visual task, the lateral geniculate nucleus, the primary visual cortex and a few other visual cortical areas should be involved, independently of the submodalities of the visual information. Indeed, during the processing of various visual tasks a number of visual cortical ﬁelds are present in the various cortical macronetworks underlying the different tasks. These networks can contain a central core, a “core network”, which is, in the case of sensory information processing, sensory modality dependent. In addition to this, additional cortical neuronal populations may are recruited to this core network, which are essential for the task performance. These ﬁelds can be termed “recruited ﬁelds” or “recruited neuronal populations”. For instance, in the case of a visual task on feature uncertainty analysis using gratings with different orientations and spatial frequencies, we could identify a core network of cortical neuronal populations present in all task performances, whereas certain cortical ﬁelds were only present when the decision had to be made along one speciﬁc stimulus modality, for instance orientation or spatial frequency (Guly´as and Roland 1995; Fig. 11). We have tried to explore the task, stimulus and input modality of the cortical networks, with special regard to the core and the recruited ﬁelds. In one experiments,

Functional Neuroimaging and the Logic of Mental Processes

155

Spatial frequency and orientation

U

Fig. 11 Core networks and recruited ﬁelds. Cortical ﬁelds activated by spatial frequency (ﬁrst column), orientation (second column) or conjoint spatial frequency–orientation (thirds column) tasks, using identical stimuli (gratings with different spatial frequencies and orientations). The cortical ﬁelds activated by all tasks (core network) are shown in the fourth column. The position of the horizontal slices is shown at the right (Guly´as and Roland 1995)

for instance, we used the same stimuli under different task conditions (Vidny´anszky et al. 2000). The subjects visually inspected identical stimuli: gratings in the centre or placed out of centre in a rectangle. The task was either to make a discrimination of the form (regular or irregular grating) or the position (centred or not). That is, whereas the stimuli were identical, the tasks were different. The resulting cortical networks contained a core network and recruited ﬁelds which were taskdependent, indicating that recruited cortical neuronal populations may indeed be task-dependent (Fig. 12). We have also investigated the other way around: to what extent are recruited cortical ﬁelds stimulus-dependent when the tasks are identical. The subjects were asked to inspect little objects, so-called parallelopipeda, in three dimensions (i.e. as real-life objects) or in two dimensions (as photographs). The objects were presented as consecutive pairs (ﬁrst object followed by the second object) and the experimental subjects in both cases had to say whether the two objects were identical or not. Again, in both cases there was a core network present in the brain, to which stimulus-dependent cortical neuronal populations were recruited (Kov´acs et al. 1998; Fig. 13). How about higher-level sensory tasks, for instance the recognition of the same visual form on the basis of various visual input cues? It is a common fact that visual

156

B. Guly´as

Fig. 12 Task dependence of recruited ﬁelds. Using identical stimuli (a) in a form or position discrimination task, the form (b) and position (c) tasks activate partially overlapping networks of cortical ﬁelds. Some constituents of the congruent part of these networks (core network) are shown in (d), whereas recruited ﬁelds present in the form discrimination task alone are shown in (e) (Vidny´anszky et al. 2000)

a 2D vs. control

b

3D vs. control

c

Fig. 13 Stimulus dependence of recruited ﬁelds. In an identical task condition (matching to sample discrimination task) with real three-dimensional objects, parallelopipeda or their two-dimensional photos (a), in addition to the core network present in both discrimination tasks (b, c), certain ﬁelds were only recruited in the three-dimensional task (c) (Kov´acs et al. 1998)

Functional Neuroimaging and the Logic of Mental Processes

157

luminance colour disparity texture motion Fig. 14 Input cue dependence of cortical networks generating identical form percepts on the basis of different input cues. The odd-one-out task is shown in the upper-left panel. The central rectangle was the reference ﬁgure, the one to the left or the right of it was different from the reference ﬁgure and the subjects had to identify this odd ﬁgure. The contours of the shape of the rectangles was made up by luminance, colour, disparity, texture or motion cues. In the colour-coded image, the input cue dependent cortical ﬁelds are shown in a schematic brain image. They represent, in fact, the recruited ﬁelds dependent upon the input cues (Guly´as et al. 1994,b, 1998)

contours can be obtained on the basis of various visual information: luminance, colour, disparity, texture and motion can each create visual contours. Consequently, the same form stimuli can be generated by each of these visual cues. In an experimental series using an “odd one out” paradigm we explored whether the generation of identical form stimuli (rectangles) is dependent on input cues or not. The ﬁndings clearly indicated that despite the fact that the resulting form percepts were identical, the cortical networks contributing to the generation of the percept were input cue dependent, demonstrating the importance of recruiting various cortical neuronal populations for the various submodalities (Guly´as et al. 1994, 1998; Fig. 14). How about the core networks and recruited cortical ﬁelds in higher cognitive functions? Take, for instance, speech or word generation on the basis of various input cues. We can repeat words or generate a said text by listening to another speaker (input cue – hearing), by reading a text (input cue – reading) or by using our touch and analyse a meaningful surface texture such as Braille letters (input cue – somatosensory). Should our aforementioned logic be valid for such a higher process, as well, the input cue (vision, audition, somatosensation) would complement the very same core network of word understanding and word generation in the human brain. This question has been explored not as a uniﬁed experiment but by different groups focusing on the various input modalities. What can we expect? The best established model on speech generation can be traced back to Paul Broca’s discovery of the prefrontal motor speech area (the “Broca area”), followed by Karl

158

B. Guly´as

Wernicke’s discovery of the “sensory speech area”. A series of observations based predominantly on lesion studies have been crystallised in the Wernicke–Geschwind model of speech generation. In this model the three key “players” of the cortical basis of speech understanding and speech generation are the motor speech area, or Broca area, the sensory speech area, or Wernicke area, and the ﬁbre tract connecting these two cortical regions, the fasciculus arcuatus (Fig. 15a). Depending upon the

Broca’s area (45) Primary motor area (4) Wernicke’ sarea (22) Parieto-temporo-occipital association region (39) Striate and extrastriate visual areas (17,18,19)

A

a

b

c

B Fig. 15 (A) The Wernicke–Geschwind model of language understanding and generation. The various cortical regions, related to visual input based reading, text understanding and speech generation are shown in colours. The arrows indicate the information ﬂow between the regions. (B) Information ﬂow and the activated core network and input cue speciﬁc recruited ﬁelds in various forms of language understanding and reproduction tasks. In (a) the input is visual (reading a text). In addition to the core network, including the parieto-occipital association regions, the Wernicke and Broca areas, and the sector of the primary motor area responsible for motor speech generation, visual cortical areas responsible for the analysis and processing of visual information are active. In (b) the activation is auditory (hearing a text). In addition to the core network the primary auditory region is active. In (c) the input cue is somatosensory (Braille reading). In addition to the core network, the somatosensory cortex is activated

Functional Neuroimaging and the Logic of Mental Processes

A

C A

C

B

D

159

a

−52

−46

−44

b

c Fig. 16 Examples from the functional imaging literature for the three ways of activating the language recognition and speech production system in the brain. (a) Speech production on the basis of reading a text (Palmer et al. 2001). (b) Auditory input based language generation (Papathanassiou et al. 2000). (c) Speech production on the basis of somatosensory input cues (Burton et al. 2004)

input modality, other sensory modality speciﬁc regions (visual, auditory, somatosensory) may be recruited to this core network (Fig. 15b). Indeed, imaging studies on speech generation based upon different sensory inputs indicate that the core network in each case is the same, whereas the other cortical ﬁelds recruited to this core network are sensory modality dependent and include, respectively, visual, auditory or somatosensory cortical areas (Fig. 16). Similar ﬁndings have been described in other modalities, as well. For instance, motor tasks with identical core components but different performance variations (e.g. handwriting – fast, precise, normal, slow, etc.) activate a core network of cortical motor regions and a number of additional ﬁelds depending upon the performance variant (Seitz et al. 1997) or during olfactory memory and discrimination tasks certain neuronal populations are consistently present, whereas others are taskdependent (Savic et al. 2000; Fig. 17). These observations support the wide validity of the model that certain cortical core networks are the basis of cortical activation patterns, whereas these networks are complemented with additional “recruited” ﬁelds.

160

B. Guly´as

Fig. 17 Core networks in olfaction. The neuronal populations activated by three various olfactory tasks (olfactory memory, white; quality discrimination, green; intensity discrimination, yellow) show congruence in the prefrontal cortex and the cerebellum (Savic et al. 2000)

8 The Physiological Limits of Cortical Neuronal Activity Neuronal activities entail increased cellular metabolism and, consequently, increased glucose and oxygen utilisation. This can be measured by PET by using either direct glucose metabolic measurements or cerebral blood ﬂow measurements. A main lesson of these types of measurements is that the brain’s energy and blood ﬂow resources cannot be “boosted” without a limit; only rather limited global increases are allowed. The reason behind the limitation of the global cerebral blood ﬂow is mainly anatomical: the brain is inside the cranium, which restricts any signiﬁcant increase in global cerebral blood ﬂow and blood volume. In the case of the intracranial circulation a careful balance should be kept between the incoming and the outgoing blood volumes, without resulting in brain oedema and brain death. The brain can, however, efﬁciently reorganise its regional circulation pattern if needed. When the regional activity of certain cortical neuronal population requires an increased energy supply and, consequently, an increased regional blood ﬂow, the existing blood supplies in the brain are redistributed. Brain regions with higher energy demand due to actual neuronal activation and, consequently, more glucose and oxygen request will have increased regional blood ﬂow, and this surplus is compensated by somewhat decreased regional blood ﬂow in cortical regions not participating in the actual information processing. This redistribution principle is a key component of our understanding of how cerebral information ﬂow may be organised. And to keep this principle in mind when we dwell upon the neurobiological basis of conscious or non-conscious brain processes is especially important: it should be understood that brain activities cannot be “boosted” indeﬁnitely. The physiological limits of brain activation are set by the limitations of cerebral blood ﬂow and metabolism. Consequently, the number of activated brain units (i.e. neurons, neuronal assemblies) cannot be increased without a limit when, for instance, a non-conscious neurobiological process becomes a conscious brain activation.

Functional Neuroimaging and the Logic of Mental Processes

161

This important observation is also echoed by the question often asked by lay people: “Is it true that we use only 5 or 10% of our brain’s capacity?” Not quite! We need our whole brain, but indeed, at any time owing to the limitations of the brain’s energy supply provided by its blood supply, not all neuronal populations can be active. The activations are “economised” and the number of active neuronal assemblies has an upper limit. The brain’s limited blood supply is redistributed according to the actual processing requirements in the brain: increased regional blood ﬂow occurs in cortical regions with activated neuronal populations, at the expense of decreased blood ﬂow in cortical regions whose activation is not essential for the performance of the given task (Kawashima et al. 1995).

9 A Step Away from Conscious Sensory Perception: Visual Illusions, Illusory Figures If one would like to investigate the border areas of conscious and non-conscious mental processes, an excellent test case is offered by visual illusory contours and visual illusions. In the case of visual illusions, the human brain sees something which is “not out there”, i.e. the human visual system completes a given piece of visual information so that it becomes complete, with respect to, for instance, a visual contour. Illusory contours and illusory ﬁgures do not necessarily “mobilise” conscious efforts during perceptual processes. However, the neuronal mechanisms behind the perception of illusory contours and ﬁgures may provide us with an insight into the neuronal basis of completing insufﬁcient visual information to form an interpretable percept and reveal the activation pattern related to the “completion” of missing information in the visual images. In an imaging experiment with PET we have studied this question and compared the visual activation patterns obtained during the perception of real line contours and illusory contours (Larsson et al. 1999). The difference images revealed activations in the early visual cortical areas (V1, V2, fusiform gyrus) and indicated that top-down inﬂuences impinge upon the early retinotopically organised visual areas (Fig. 18). These observations may be surprising in the light of the often extreme complexity of illusory ﬁgures. However, other studies using fMRI have resulted in similar ﬁndings, supporting the hierarchically “low-level” solution in the brain (Mendola et al. 1999).

10 Bimodal Figures Bimodal or instable ﬁgures also provide us with a useful test case for understanding the role of conscious “top-down” inﬂuence on visual perception. Bimodal ﬁgures are ﬁgures with more than one possible interpretation (Fig. 19a). If one looks at them without consciously choosing one of the two possible interpretations, the

162

B. Guly´as

a

b

c Fig. 18 Subtraction images related to illusory contour detection, using 15 O-butanol as a blood ﬂow tracer in PET measurements. (a) Illusory contours – real contours. (b) Illusory contours – control task. (c) Real contours – control task (Larsson et al. 1999)

perceived ﬁgure(s) may ﬂuctuate between the two states, two percepts with different interpretations. The corresponding neuronal activation patterns related to a “free ﬂoating state” (or unattended state) and a consciously attended state (when we decidedly prefer one possible interpretation) may reveal for us the neuronal machinery behind conscious top-down inﬂuence. Such imaging experiments were performed using conﬁgurations and revealed that the highest activations are found at higher-level visual cortical areas in the occipitoparietal and occipitotemporal pathways (Kleinschmidt et al. 1998; Fig. 19b), whereas in lower levels of the visual pathways, including the primary visual cortex and the pulvinar, deactivation was present (Fig. 19c). That is, the brain tries to solve such a conundrum as close as possible to the sensory input side but already at an advanced level of complexity, in this way applying a kind of a “subsidiarity” principle: tackle sensory questions at the lowest possible level in the sensory processing pathways. This fact may serve as an important lesson for consciousness studies: The human brain is an “Ockham machine” and applies the principle of parsimony in its activities, with special regard to the non-conscious and conscious border area. The question then arises: Can it be visualised with functional imaging techniques that the brain can exercise a conscious effort to actively choose one of the possible

Functional Neuroimaging and the Logic of Mental Processes

163

Fig. 19 (a) Examples of ambiguous ﬁgures. (b) Coronal and horizontal images with signiﬁcant increases of neuronal activity during the detection of ambiguous ﬁgures. The peak activations are in the occipitoparietal and occipitotemporal visual pathways. (c) Signiﬁcant decreases of neuronal activations during the same tasks (Kleinschmidt et al. 1998)

interpretations and thereby decide over the percept to be perceived? Can we ﬁnd neuronal populations in the human brain responsible for the brain’s conscious efforts to determine the content of a percept? If so, there should be a difference in cortical networks underlying a passive exploration of bimodal ﬁgures and cortical networks underlying the perceiving of one of the possible contents of the percept.

11 On the Border of Conscious and Non-conscious: Visual Imagery Visual imagery can be a test case for understanding “passive visual perception”. In this case we visualise in front of the “mind’s eye” already-seen visual scenes or we create by our consciousness never-seen visual scenes. An age-old question is whether the underlying neuronal networks in visual perception and in visual imagery are identical or not. In other words, whether during visual imagery the very same cortical macronetwork is activated from the primary visual cortex onwards as in visual perception, when the information reaches the visual cortex from the retina through the lateral geniculate nucleus (Fig. 20). On the basis of some evidence, some claim that the cortical regions involved in visual imagery are identical in visual imagery and visual perception (Kosslyn and Thompson 2003). Others claim that in visual imagery only higher-level visual

164

B. Guly´as

Fig. 20 Information ﬂow during visual perception and two possible ways of information ﬂow during visual imagery. (a) During visual perception, visual information is processed from the retina through the geniculate nucleus to the primary visual cortex and thereon to parietal and temporal visual cortical regions. (b) According to one model, during visual imagery the process is initiated in the prefrontal cortex and the primary visual cortex is activated by internal cues. From thereon the higher visual cortical areas are activated in the same manner as during visual perception. (c) According to other models, the activation of the primary visual cortex during visual imagery is not necessary; only the higher-level visual cortical areas are engaged in the process (Roland and Guly´as 1994)

Emotions (?)

Text (?) Recall (?)

Fig. 21 Cortical ﬁelds activated during the visualisation of the text of the Hungarian national anthem with closed eyes. The parieto-occipital ﬁelds are most probably related to the recall of stored visual information related to the text, the prefrontal activation may be related to the memory component of the task, whereas the limbic activation may be due to the strong emotional component of the text of the national anthem (Guly´as 2001b)

cortical regions are active (Roland and Guly´as 1994). These regions are usually engaged in the analysis, storage and recall of higher-level visual information, as demonstrated by other studies (Roland and Guly´as 1995). In our studies we found no activation in the primary visual cortex when subjects had to visualise letters and texts with closed eyes; the activations engaged higher-level visual cortical areas (Guly´as 2001b; Fig. 21). It was demonstrated that the human brain can voluntarily consciously recall visual contents from its visual memory and recreate visual sceneries in an active manner. As these activities require prefrontal and higher-level

Functional Neuroimaging and the Logic of Mental Processes

165

visual cortical activations, one can draw the conclusion from these observations that the conscious generation of visual percepts and visual contents requires the activation of neuronal populations at higher levels in the visual pathway, whereas the lower-level visual cortical areas are not included in these conscious processes. This model, however, has been challenged by other observations, including the study of blindsight.

12 Blindsight and Its Companions The neurological condition of blindsight and its twin conditions (death hearing, numb touch) provide us with an excellent testable hypothesis on the “brain requirements” of consciousness (Weiskrantz 1990a). In the case of blindsight a part or the whole of the primary visual cortex is missing (Weiskrantz 1990b, 1996, 2004; Cowey and Stoerig 1991; Stoerig and Cowey 2007). The damage is usually onesided and rarely respects “histological borders”, i.e. Brodmann area borders. The medical reason behind it can be an accident resulting in loss of brain tissue, tumour or stroke (Fig. 22a). Owing to the lack of the primary visual cortex, the subjects do not perceive what they see in the blind visual ﬁeld. More precisely, they claim that they do not really see anything, especially when visual stimulus conditions are weak, for instance when the blind visual ﬁeld is stimulated with low-contrast stimuli or with slow-moving stimuli. However, subjects can make a proper discrimination of the stimulus, even if they claim they do not see it “consciously”. By increasing the stimulus contrast or the stimulus velocity, the subject is more and more “conscious” of the perception of a stimulus in the blind visual ﬁeld; however, “full consciousness of the stimulus” is not reachable. Under usual everyday visual circumstances, the

Fig. 22 (a) Three-dimensional reconstruction of the magnetic resonance image of a patient with loss of his right primary visual cortex (shown on the left side, owing to radiological convention) (Morris et al. 2001). (b) Activations (in colour) in the PET images during the stimulation of the blind visual hemiﬁeld of the subject (Barbur et al. 1993)

166

B. Guly´as

subject can be regarded as a functionally blind subject, at least as far as the affected hemisphere and, consequently, the affected visual ﬁeld are concerned (Weiskrantz et al. 1995). A further consequence of blindsight is that these subjects cannot recognise objects in the blind visual ﬁeld, but if a frightening emotional visual stimulus is presented, the appropriate physiological reactions (changes in heart rate, pupil diameter, skin conductance) can be registered, indicating that the brain’s emotional system registers the frightening image. Indeed, visual information reaches the amygdala through neuronal tracts avoiding the geniculate nucleus and the visual cortex and relays direct visual information about the frightening stimulus. The activation of the amygdala can be demonstrated by neuroimaging techniques (Morris et al. 2001). Correlative magnetic resonance imaging and PET studies on blind sight subjects show the primary visual cortex lesion but also show that other visual cortical areas are active and, logically, they process visual information (Fig. 22b). This observation, together with the psychophysical ﬁndings, may indicate that the primary visual cortex plays an essential role in the conscious visual perception (Stoerig 2006). Similar observations have been made in the case of damage to the primary auditory and somatosensory cortex (Engelien et al. 2000; Garde and Cowey 2000).

13 The Effect of Unconscious Perception on Conscious Decisions Based upon Visual Input It is widely know from the psychophysical literature that unconscious perceptual information may signiﬁcantly inﬂuence the human brain’s conscious decisions. An eloquent example of this phenomenon is the inﬂuence of the pheromone sense on decision making. Pheromones are airborne compounds, produced by the body. Human pheromones are most commonly hormones or their metabolites produced predominantly by the genital and other glands (armpit, etc.) of the human body. Most human pheromones are not odorous and are not perceived as smells by the olfactory sense. There is a dedicated sensory system, the pheromone sense, which is responsible for the perception of pheromones. The peripheral part of this system is not identical with the two other systems perceiving and relaying smell information: the olfactory system (cranial nerve I) and the trigeminal system (cranial nerve V). In contrast to them, its peripheral sensory receptors are in the vomeronasal organ in the nasal septum and its afferent ﬁbres run parallel to the olfactory nerve. The terminations are, however, not identical to those of the olfactory nerve (piriform cortex), but reach neurons in the fusiform gyrus (Sobel and Brown 2001; Savic 2002). In earlier publications we showed that pheromones elicit clear-cut activations in the human brain (Savic et al. 2001). These activations are gender-speciﬁc in a double sense: the neuronal activation patters are dependent upon the subject’s gender (male, female) as well as the “gender” of the pheromone (male or female). The exposure

Functional Neuroimaging and the Logic of Mental Processes

167

to pheromones is usually not accompanied by a conscious olfactory experience, as under experimental conditions most subjects, predominantly males, are not aware of the fact that they were exposed to pheromones and claim that they did not smell anything unusual. Pheromones can signiﬁcantly modify our behavioural responses. It is a classic observation that girls living together in dormitories synchronise their menstrual cycles (Wilson 1992; Stern and McClintock 1998). Pheromones can affect attraction–detraction between humans. It has been demonstrated, for instance, that in the waiting room of a medical ofﬁce seats treated with the non-odorous male pheromone androstadienone are more frequently occupied by women than by men, the difference being signiﬁcant. In an experimental series on face classiﬁcation, unconscious pheromone effects signiﬁcantly modiﬁed the subjects’ decision on the masculinity or femininity of the faces seen. The subjects were asked to look at faces which were morphed from a female face into a male face or vice versa. When the subjects, without being aware of it, were exposed to pheromones, their decisions were signiﬁcantly modiﬁed, indicating that conscious decision making on the basis of a sensory input can be signiﬁcantly modiﬁed by the unconscious perception of other input information reaching the human brain (Kov´acs et al. 2004; Fig. 23).

Fig. 23 The morphing series of faces containing 100% female (upper left) and 100% male (lower right) components, or varying amounts of female and male components (in-between the two puresex faces). Experimental subjects had to determine the ranges for which the faces can be regarded as female (from the pure female face until the continuous green line) or male (from the pure male face until the red line). When the subjects were exposed to androstendion, a male pheromone, the female range was shortened, i.e. some morphed faces were not regarded anymore as feminine (threshold at the dotted green line) (Kov´acs et al. 2004)

168

B. Guly´as

Fig. 24 The cortical activations in the female brain elicited by androstadienone activation (Guly´as et al. 2003) appear to be within cortical regions identiﬁed as fundamental parts of the “social brain” (in coloured dotted lines: inferior parietal cortex/blue/, dorsolateral prefrontal cortex/green/, orbito-frontal cortex/yellow/) (Decety and Jackson, 2004)

In another study we mapped the cortical regions in the female brain participating in the perception of male pheromones as contrasted with the perception of pleasant, neutral or unpleasant odorous airborne chemicals (Guly´as et al. 2004). The cortical activations during pheromone perception were present inside those cortical regions which in other functional imaging studies using social cognitive paradigms (reviewed in Decety and Jackson 2004) were identiﬁed as essential parts of the “social brain” (Fig. 24). These are the cortical regions which participate in decisions on ﬁnding someone sympathetic or antipathic, feeling empathy, deciding on a just or unjust situation, reacting in a socially engaged manner, etc. It is highly possible that unconscious pheromone effects may markedly modify our social decisions in close personal encounters by way of activating neuronal populations in the “social brain” so that the subject remains unconscious of the nature of these effects. This ﬁnding, with a body of other observations, supports the contribution of the unconscious perceptual information to conscious decisions and calls our attention to the complexity of “conscious” decisions.

14 Conclusions The search for the neuronal correlates of the conscious components of brain processes has recently become a main objective of functional neuroimaging research. The available neuroimaging techniques can reveal activated cortical neuronal networks, consisting of active neuronal populations. These are the basis of the neuronal correlates of both conscious and non-conscious brain processes, and as such are the legitimate subjects of functional neuroimaging studies in consciousness

Functional Neuroimaging and the Logic of Mental Processes

169

research. The examples in this chapter, taken mainly from the ﬁeld of visual consciousness research, demonstrate the applicability of the functional neuroimaging approach in this search. The search for the neuronal and neuroimaging correlates of consciousness itself, however, is most probably an ill-posed question and requires further theoretical and methodological clariﬁcations.

References Ache BW, Young JM (2005) Olfaction: diverse species, conserved principles. Neuron 48:417–430 Barbur JL, Watson JD, Frackowiak RS, Zeki S (1993) Conscious visual perception without V1. Brain 116:1293–1302 Baud P (2005) Personality traits as intermediary phenotypes in suicidal behavior: genetic issues. Am J Med Genet C Semin Med Genet 133:34–42 Breier A, Kestler L, Adler C, Elman I, Wiesenfeld N, Malhotra A, Pickar D (1998) Dopamine D2 receptor density and personal detachment in healthy subjects. Am J Psychiatry 155:1440–1442 Bondy B, Buettner A, Zill P. (2006) Genetics of suicide. Mol Psychiatry 11:336–351 Bozkurt A, Zilles K, Schleicher A, Kamper L, Arigita ES, Uylings HB, Kotter R (2005) Distributions of transmitter receptors in the macaque cingulate cortex. Neuroimage 25:219–229 Burton H, Snyder AZ, Raichle ME (2004) Default brain functionality in blind people. Proc Natl Acad Sci U S A 101:15500–15505 Buzs´aki G, Kaila K, Raichle M (2007) Inhibition and brain work. Neuron 56:771–783 Claeys KG, Dupont P, Cornette L, Sunaert S, Van Hecke P, De Schutter E, Orban GA (2004) Color discrimination involves ventral and dorsal stream visual areas. Cereb Cortex 14:803–822 Cloninger R (1986) A uniﬁed biosocial theory of personality and its role in the development of anxiety states. Psychiatr Dev 3:167–226 Cloninger CR (1987) A systematic method for clinical description and classiﬁcation of personality variants. A proposal. Arch Gen Psychiatry 44:573–588 Cloninger CR (2000) Biology of personality dimensions. Curr Opin Psychiatry 13:611–616 Cloninger CR, Svrakic DM, Przybeck TR (1993) A psychobiological model of temperament and character. Arch Gen Psychiatry 50:975–990 Cohen MX, Young J, Baek JM, Kessler C, Ranganath C (2005) Individual differences in extraversion and dopamine genetics predict neural reward responses. Brain Res Cogn Brain Res 25:851–861 Cowey A, Stoerig P (1991) The neurobiology of blindsight. Trends Neurosci 14:140–145 Decety J, Jackson PL (2004) The functional architecture of human empathy. Behav Cogn Neurosci Rev 3:71–100 Eickhoff SB, Schleicher A, Scheperjans F, Palomero-Gallagher N, Zilles K (2007) Analysis of neurotransmitter receptor distribution patterns in the cerebral cortex. Neuroimage 34:1317–1330 Ekelund J, Lichtermann D, Jarvelin MR, Peltonen L (1999) Association between novelty seeking and the type 4 dopamine receptor gene in a large Finnish cohort sample. Am J Psychiatry 156:1453–1455 Ekstrom P, Meissl H (2003) Evolution of photosensory pineal organs in new light: the fate of neuroendocrine photoreceptors. Philos Trans R Soc Lond B Biol Sci 358:1679–1700 Engelien A, Huber W, Silbersweig D, Stern E, Frith CD, D¨oring W, Thron A, Frackowiak RS (2000) The neural correlates of ‘deaf-hearing’ in man: conscious sensory awareness enabled by attentional modulation. Brain 123:532–545 Farde L (1996) The advantage of using positron emission tomography in drug research. Trends Neurosci 19:211–214 Farde L, Gustavsson JP, Jonsson E (1997) D2 dopamine receptors and personality traits. Nature 385:590

170

B. Guly´as

Friston KJ, Ashburner JT, Kiebel SJ, Nichols TE, Penny WD (eds) (2007) Statistical parametric mapping. Academic, Amsterdam Garde MM, Cowey A (2000) “Deaf hearing”: unacknowledged detection of auditory stimuli in a patient with cerebral deafness. Cortex 36:71–80 Garpenstrand H, Annas P, Ekblom J, Oreland L, Fredrikson M (2001) Human fear conditioning is related to dopaminergic and serotonergic biological markers. Behav Neurosci 115:358–364 Garvey MJ, Noyes R Jr, Cook B, Blum N (1996) Preliminary conﬁrmation of the proposed link between reward-dependence traits and norepinephrine. Psychiatry Res 65:61–64 Gerra G, Avanzini P, Zaimovic A, Sartori R, Bocchi C, Timpano M, Zambelli U, Delsignore R, Gardini F, Talarico E, Brambilla F (1999) Neurotransmitters, neuroendocrine correlates of sensation-seeking temperament in normal humans. Neuropsychobiology 39:207–213 Geyer S, Schleicher A, Zilles K (1997) The somatosensory cortex of human: cytoarchitecture and regional distributions of receptor-binding sites. Neuroimage 6:27–45 Geyer S, Matelli M, Luppino G, Schleicher A, Jansen Y, Palomero-Gallagher N, Zilles K (1998) Receptor autoradiographic mapping of the mesial motor and premotor cortex of the macaque monkey. J Comp Neurol 397:231–250 Gross CG (1992) Representation of visual stimuli in inferior temporal cortex. Philos Trans R Soc Lond B Biol Sci 335:3–10 Guly´as B, Roland PE (1991)Cortical ﬁelds participating in form and colour discrimination in the human brain. Neuroreport 2:585–588 Guly´as B, Heywood CA, Popplewell DB, Cowey A, Roland PE (1994) Visual form discrimination from colour or motion cues: functional anatomy by positron emission tomography. Proc Natl Acad Sci U S A 91:9965–9969 Guly´as B, Roland PE (1994a) Processing and analysis of form, colour and binocular disparity in the human brain: functional anatomy by positron emission tomography. Eur J Neurosci 6:1811–1828 Guly´as B, Roland PE (1994b) Binocular disparity detection in human visual cortex: functional anatomy by positron emission tomography. Proc Natl Acad Sci U S A 91:1239–1243 Guly´as B, Roland PE (1995) Visual cortical ﬁelds participating in spatial frequency and orientation discrimination: functional anatomy by positron emission tomography. Hum Brain Mapp 3:133–152 Guly´as B, Roland PE, Cowey A, Heywood CA, Popplewell D (1998) Visual form discrimination from texture cues: a PET study. Hum Brain Mapp 6:115–127 Guly´as B (2001a) The dynamics of cortical macronetworks in the human brain. Brain Res Bull 53:251–253 Guly´as B (2001b) Neural networks for internal reading and visual imagery of reading: a PET study. Brain Res Bull 53:319–328 Guly´as B, K´eri S, O’Sullivan BT, Decety J, Roland PE (2004) The putative pheromone androstadienone activates cortical ﬁelds in the human brain related to social cognition. Neurochem Int. 44:595–600 Guly´as B, Sj¨oholm N (2007) Principles of positron emission tomography. In: Hillary FG, DeLuca J (eds) Functional neuroimaging in clinical populations. Guildford, New York, pp. 3–30 Guly´as B, Halldin C, Mazi`ere B (2008) In vivo imaging of neurotransmitter systems. In: Handbook of Neurochemistry. (Eds.: E. Sylvester Vizi and Michel Hamon). Springer. pp. 75–100 Halldin C, Guly´as B, Farde L (2001a) PET studies with carbon-11 radioligands in neuropsychopharmacological drug development. Curr Pharm Des 7:1907–1929 Halldin C, Guly´as B, Langer O, Farde L (2001b) Brain radioligands – state of the art and new trends. Q J Nucl Med 45:139–152 Hennig J, Toll C, Schonlau P, Rohrmann S, Netter P (2000) Endocrine responses after d-fenﬂuramine and ipsapirone challenge: further support for Cloninger’s tridimensional model of personality. Neuropsychobiology 41:38–47 Hurlemann R, Matusch A, Eickhoff SB, Palomero-Gallagher N, Meyer PT, Boy C, Maier W, Zilles K, Amunts K, Bauer A (2005) Analysis of neuroreceptor PET-data based on cytoarchitectonic maximum probability maps: a feasibility study. Anat Embryol (Berl) 210:447–453

Functional Neuroimaging and the Logic of Mental Processes

171

Jonsson EG, Cichon S, Gustavsson JP, Grunhage F, Forslund K, Mattila-Evenden M, Rylander G, Asberg M, Farde L, Propping P, Nothen MM (2003) Association between a promoter dopamine D2 receptor gene variant and the personality trait detachment. Biol Psychiatry 53:577–584 Kawashima R, O’Sullivan BT, Roland PE (1995) Positron-emission tomography studies of crossmodality inhibition in selective attentional tasks: closing the “mind’s eye”. Proc Natl Acad Sci U S A 92:5969–5972 K´eri S, Guly´as B (2003) Four facets of a single brain: behaviour, cerebral blood ﬂow/metabolism, neuronal activity and neurotransmitter dynamics. Neuroreport 14:1097–1106 Kleinschmidt A, B¨uchel C, Zeki S, Frackowiak RS (1998) Human brain activity during spontaneously reversing perception of ambiguous ﬁgures. Proc Biol Sci 265:2427–2433 Kosslyn SM, Thompson WL (2003) When is early visual cortex activated during visual mental imagery? Psychol Bull 129:723–746 Kov´acs G, Guly´as B, Roland PE (1998) Processing of 2D and 3D shapes in the visual association cortex. Neuroimage 7:S335 Kov´acs G, Guly´as B, Savic I, Perrett DI, Cornwell RE, Little AC, Jones BC, Burt DM, G´al V, Vidny´anszky Z (2004) Smelling human sex hormone-like compounds affects face gender judgment of men. Neuroreport 15:1275–1277 Larsson J, Amunts K, Guly´as B, Malikovic A, Zilles K, Roland PE (1999) Neuronal correlates of real and illusory contour perception: functional anatomy with PET. Eur J Neurosci 11: 4024–4036 Lennie P (2003) The cost of cortical computation. Curr Biol 13:493–497 Lueck CJ, Zeki S, Friston KJ, Deiber MP, Cope P, Cunningham VJ, Lammertsma AA, Kennard C, Frackowiak RS (1989) The colour centre in the cerebral cortex of man. Nature 340:386–389 Mendola JD, Dale AM, Fischl B, Liu AK, Tootell RB (1999) The representation of illusory and real contours in human cortical visual areas revealed by functional magnetic resonance imaging. J Neurosci 19:8560–8572 Morosan P, Schleicher A, Amunts K, Zilles K (2005) Multimodal architectonic mapping of human superior temporal gyrus. Anat Embryol (Berl) 210:401–406 Morris JS, DeGelder B, Weiskrantz L, Dolan RJ (2001) Differential extrageniculostriate and amygdala responses to presentation of emotional faces in a cortically blind ﬁeld. Brain 124:1241–1252 Netter P, Hennig J, Roed IS (1996) Serotonin and dopamine as mediators of sensation seeking behavior. Neuropsychobiology 34:155–165 Okuyama Y, Ishiguro H, Nankai M, Shibuya H, Watanabe A, Arinami T (2000) Identiﬁcation of a polymorphism in the promoter region of DRD4 associated with the human novelty seeking personality trait. Mol Psychiatry 5:64–69 Palmer ED, Rosen HJ, Ojemann JG, Buckner RL, Kelley WM, Petersen SE (2001) An eventrelated fMRI study of overt and covert word stem completion. Neuroimage 14:182–193 Papathanassiou D, Etard O, Mellet E, Zago L, Mazoyer B, Tzourio-Mazoyer N (2000) A common language network for comprehension and production: a contribution to the deﬁnition of language epicenters with PET. Neuroimage 11:347–357 Park JW, Kim JS, Lee HK, Kim YI, Lee KS (2004) Serotonin transporter polymorphism and harm avoidance personality in chronic tension-type headache. Headache 44:1005–1009 Peirson AR, Heuchert JW, Thomala L, Berk M, Plein H, Cloninger CR (1999) Relationship between serotonin and the temperament and character inventory. Psychiatry Res 89:29–37 Pilpel Y, Sosinsky A, Lancet D (1998) Molecular biology of olfactory receptors. Essays Biochem 33:93–104 Rogers G, Joyce P, Mulder R, Sellman D, Miller A, Allington M, Olds R, Wells E, Kennedy M (2004) Association of a duplicated repeat polymorphism in the 5 -untranslated region of the DRD4 gene with novelty seeking. Am J Med Genet B Neuropsychiatr Genet 126:95–98 Roland P (1993) Brain activation. Wiley-Liss, New York Roland PE, Guly´as B (1994) Visual imagery and visual representation. Trends Neurosci 17: 281–287

172

B. Guly´as

Roland PE, Guly´as B (1995) Visual memory, visual imagery, and visual recognition of large ﬁeld patterns by the human brain. Functional anatomy by positron emission tomography. Cereb Cortex 5:79–93 Samochowiec J, Kucharska-Mazur J, Rybakowski F, Ostapowicz A, Horodnicki J, Rozpara M, Kaminski R, Skrzypinska A, Arentowicz G, Rommelspacher H, Schmidt LG (2002) Norepinephrine transporter polymorphism and personality trait of reward dependence in male alcoholics. Pharmacopsychiatry 35:195–196 Sara SJ, Dyon-Laurent C, Herve A (1995) Novelty seeking behavior in the rat is dependent upon the integrity of the noradrenergic system. Brain Res Cogn Brain Res 2:181–187 Savic I (2002) Imaging of brain activation by odorants in humans. Curr Opin Neurobiol 12:455–461 Savic I, Guly´as B, Larson M, Roland PE (2000) Olfactory functions are mediated by parallel and hierarchical processing. Neuron 26:735–745 Savic I, Berglund H, Gulyas B, Roland P (2001) Smelling of odorous sex hormone-like compounds causes sex-differentiated hypothalamic activations in humans. Neuron 31:661–668 Scheperjans F, Grefkes C, Palomero-Gallagher N, Schleicher A, Zilles K (2005a) Subdivisions of human parietal area 5 revealed by quantitative receptor autoradiography: a parietal region between motor, somatosensory, and cingulate cortical areas. Neuroimage 25:975–992 Scheperjans F, Palomero-Gallagher N, Grefkes C, Schleicher A, Zilles K (2005b) Transmitter receptors reveal segregation of cortical areas in the human superior parietal cortex: relations to visual and somatosensory regions. Neuroimage 28:362–379 Seitz RJ, Canavan AG, Y´ag¨uez L, Herzog H, Tellmann L, Knorr U, Huang Y, H¨omberg V (1997) Representations of graphomotor trajectories in the human parietal cortex: evidence for controlled processing and automatic performance. Eur J Neurosci 9:378–389 Singer W (1999) Neuronal synchrony: a versatile code for the deﬁnition of relations? Neuron 24:49–65, 111–125 Sobel N, Brown WM (2001) The scented brain: pheromonal responses in humans. Neuron 31: 512–514 Stern K, McClintock MK (1998) Regulation of ovulation by human pheromones. Nature 392: 177–179 Stoerig P (2006) Blindsight, conscious vision, and the role of primary visual cortex. Prog Brain Res 155:217–234 Stoerig P, Cowey A (2007) Blindsight. Curr Biol 17:R822–R824 Suhara T, Yasuno F, Sudo Y, Yamamoto M, Inoue M, Okubo Y, Suzuki K (2001) Dopamine D2 receptors in the insular cortex and the personality trait of novelty seeking. Neuroimage 13: 891–895 Sz´ekely A, Ronai Z, Nemoda Z, Kolmann G, Gervai J, Sasvari-Sz´ekely M (2004) Human personality dimensions of persistence and harm avoidance associated with DRD4 and 5-HTTLPR polymorphisms. Am J Med Genet B Neuropsychiatr Genet 126:106–110 Van Gestel S, Forsgren T, Claes S, Del-Favero J, Van Duijn CM, Sluijs S, Nilsson LG, Adolfsson R, Van Broeckhoven C (2002) Epistatic effect of genes from the dopamine and serotonin systems on the temperament traits of novelty seeking and harm avoidance. Mol Psychiatry 7:448–450 Vidny´anszky Z, Guly´as B, Roland PE (2000) Visual exploration of form and position with identical stimuli: functional anatomy with PET. Hum Brain Mapp 11:104–116 Weiskrantz L (1990a) Blindsight: a case study and its implications. Oxford University Press, Oxford Weiskrantz L (1990b) The Ferrier lecture, 1989. Outlooks for blindsight: explicit methodologies for implicit processes. Proc R Soc Lond B Biol Sci 239:247–278 Weiskrantz L (1996) Blindsight revisited. Curr Opin Neurobiol 6:215–220 Weiskrantz L (2004) Roots of blindsight. Prog Brain Res 144:229–241 Weiskrantz L, Barbur JL, Sahraie A (1995) Parameters affecting conscious versus unconscious visual discrimination with damage to the visual cortex (V1). Proc Natl Acad Sci U S A 92: 6122–6126

Functional Neuroimaging and the Logic of Mental Processes

173

Wilson HC (1992) A critical review of menstrual synchrony research. Psychoneuroendocrinology 17(1992):565–591 Zeki S, Watson JD, Lueck CJ, Friston KJ, Kennard C, Frackowiak RS (1991) A direct demonstration of functional specialization in human visual cortex. J Neurosci 11:641–649 Zilles K, Palomero-Gallagher N (2001) Cyto-, myelo-, and receptor architectonics of the human parietal cortex. Neuroimage 14(1 Pt 2):S8–S20 Zilles K, Palomero-Gallagher N, Grefkes C, Scheperjans F, Boy C, Amunts K, Schleicher A (2002) Architectonics of the human cerebral cortex and transmitter receptor ﬁngerprints: reconciling functional neuroanatomy and neurochemistry. Eur Neuropsychopharmacol 12:587–599 Zilles K, Palomero-Gallagher N, Schleicher A (2004) Transmitter receptors and functional anatomy of the cerebral cortex. J Anat 205:417–432

Knowledge Systems of the Brain Josef Ilmberger

Abstract Thinking is operating on knowledge representations. But how can knowledge be deﬁned? Research on the nature of kinds of knowledge (or mental content, or mental representations etc.) has created a plethora of classiﬁcations and open questions that have occupied philosophers and psychologists alike over centuries. Despite all efforts no common framework has been established and controversies are bound to continue. Many of the problems are rooted in terminology varying over research disciplines and personalities. In this chapter, a taxonomy of “knowledges” is delineated that might help to clear the picture and thus facilitate interdisciplinary collaboration.

1 Background “Thinking is the systematic transformation of mental representations of knowledge to characterize actual or possible states of the world, often in the service of goals. Obviously, our deﬁnition introduces a plethora of terms with meanings that beg to be unpacked. . .” (p. 2 in Holyoak and Morrison 2005). The goal of this chapter is to do some unpacking, and in focusing on the term “knowledge” it can be stated more broadly that human and nonhuman animals as well as robots need knowledge in order to evaluate sensory data (from the outside world and from within the body), to operate on these data and existing knowledge, and to act accordingly (searching for food, having sex, and collecting even more information in action–perception cycles). But what is knowledge? How is it acquired and how is it stored? Are there fundamentally different kinds of knowledge ? What kinds of knowledge do nonhuman animals or robots have? For those interested in such questions, percepts, concepts J. Ilmberger Department of Physical Medicine and Rehabilitation, Ludwig-Maximilian University, Neuropsychology Unit, Marchioninistr.15, 81337 Munich, Germany [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

175

176

J. Ilmberger

(feature-based, theory-based, prototype-based, atomistic?), categories, perceptual symbols, simulators, vehicles, cognits, image schemas, and redescribed representations form a dense jungle in which it is hard to orientate, and worse, communicate. No common framework has been established and controversies are bound to continue. I do not think that traditional classiﬁcations like procedural versus declarative or implicit versus explicit knowledge will help, as they are aimed at human knowledge only. I believe that the problem is rooted in terminology varying over research disciplines and personalities. In the present chapter, I will delineate a taxonomy of “knowledges” that might help to clear the picture and stabilize the ground for interdisciplinary collaboration. This taxonomy can be applied equally well to human and nonhuman living as to artiﬁcial systems. Deacon (1998) described three types of reference which he called iconic, indexical, and symbolic; he follows in this terminology the classiﬁcation of representational relationships by Charles Sanders Peirce. In my opinion these terms are excellent labels for the three layers of the taxonomy of knowledge described here. Iconic and indexical knowledges are nonverbal and exist also in nonhuman animals; symbolic knowledge is, perhaps with a few exceptions, only found in humans in the form of several language systems.

2 Icons: Same and Different Substances The surrounding world consists of various substances (stuff like milk, kinds of things like mouse, individual entities like mama as described by Millikan 1999 following Aristotle’s categories) which may be perceived, classiﬁed, and sometimes identiﬁed. Iconic knowledge, on which all other knowledge is built, is gained by comparison processes consisting of same/different discriminations. In its simplest form, these comparisons are performed in one sensory channel on one-dimensional data and may be performed by thresholding. Consider a robot which has the ability to move slowly on a dark surface while continuously scanning the ground by means of a light sensor. This sensor gives numerical values between 0 and 100; each of these values may be seen as a sensory event corresponding to stuff. As long as this value is below a certain threshold (say 50), the robot continues moving (i.e., consecutive values would be 33 21 9 17. . ., classiﬁcation would be same same same same). If now the robot approaches a bright area, these values would increase (23 45 12 89), the classiﬁcation would be same same same different, and the robot would stop, turn, and move in another direction. It thus can be said that the robot distinguishes two classes or categories of sensory events/substances (dark ones and bright ones) and acts accordingly. These classes are collections of patterns; percepts are representations of invariants of these collections – in this case invariance is given with respect to the inbuilt threshold (all sensory events are either smaller or larger than the threshold, regardless of individual ﬂuctuations). Classiﬁcation is simple in the sense that a speciﬁc sensory event is classiﬁed as belonging exactly to one of two classes – same

Knowledge Systems of the Brain

177

(as another event) or different (from another event) – and entails one of two types of behavior: continue behavior (which may be doing nothing in other cases), or change behavior. A deﬁnition by Hayek makes clear what is meant by “classiﬁcation”: “By ‘classiﬁcation’ we shall mean a process in which on each occasion on which a certain recurrent event happens it produces the same speciﬁc effect, and where the effects produced by any one kind of such events may either be the same or different from those which any other kind of event produces in a similar manner. All the different events which whenever they occur produce the same effect will be said to be events of the same class, and the fact that every one of them produces the same effect will be the sole criterion which makes them members of the same class.” (p. 48 in Hayek 1976). Harnad calls the very same process categorization: “Categorization is any systematic differential interaction between . . . a sensorimotor system and its world.” But: “Categorization (is) . . . not about exactly the same output occurring whenever there is exactly the same input. Categories are kinds, and categorization occurs when the same output occurs with the same kind of input . . .” (pp. 21–22 in Harnad 2005). An important note must be made at this point with regard to the term “detection.” We could also say that the robot is able to detect two classes or categories. We have to keep in mind, however, that the classes or categories are not “hidden” in the robot’s world and then “detected,” but that the discrimination process is “ . . . a process which creates the distinctions in question.” (p. 48 in Hayek 1976). By using several thresholds, the robot would be able to detect more categories (but classiﬁcation still would be simple in that a speciﬁc sensory event belongs to one and only one class) – if we had only two types of retinal cone cells, our vision would be dichromatic. The perceptual abilities of this robot may be called amorphous, similar to those available in bacteria for chemotaxis (Adler and Tso 1974; but see Vergassola et al. 2007 for a recent elaboration on this topic) or to our olfactory experience; no objects (e.g., forms distinct from the background) may be perceived with these mechanisms. More complex sensorimotor procedures are needed for the detection of object-like substances like mama and mouse, and invariant representations, i.e., percepts are formed during recurrent confrontation with those substances. Percepts are clearly different from retinal input, as such phenomena like ﬁlling-in (“. . . in which visual features, such as colour, brightness, texture and motion, of the surrounding area are perceived in a certain part of the visual ﬁeld even though these features are not physically present.”; p. 220 in Komatsu 2006) demonstrate. The formation of percepts enables “distal qualities to appear as the same through wide variation in proximal manifestations.” (p. 539 in Millikan 1999). In human infants, sensory patterns of events are built on the principles of cohesion, boundaries, substance, and spatial–temporal continuity (Spelke 1988). This is by no means a passive process: objects have to be actively tracked by eye and head movements and interactions like grasping, pushing, and pulling, that is, percepts are nearly always linked to motor action forming sensory–motor or perception–action cycles (Fuster 2006). These early motor actions consist of reﬂexes (like the palmargrasp reﬂex) and motor primitives (like tracking a moving object with the eyes); for a more detailed discussion on the distinction between reﬂexes and motor primitives

178

J. Ilmberger

and its use in robotics see Konczak (2005). In later stages, motor gestalts may be assembled from distinct collections of motor patterns; these motor gestalts would be invariants of movement patterns in analogy to percepts which are invariants of sensory patterns. A speciﬁc class or collection of patterns is deﬁned by the fact that its elements can be transformed into each other. For a very precise and helpful discussion on the distinction between gestalts and patterns, the reader is referred to Breidbach and Jost (2006). Recurrent objects are perceived as being the same by tracking, “the perception of sameness bridging, for example, over motions of perceived and perceiver, over changes in properties of the object, and over temporary disappearances of the objects behind other objects. The mechanisms responsible for the ability to track and for perceptual ‘identity-’ or ‘existence-constancy’ may well be largely endogenous . . .” (p. 538 in Millikan 1999). It is an interesting question which premises the identiﬁcation of individuals (like “mama”) is built upon on this knowledge level. The assumption of uniqueness works only if no similar object is ever present at the same time. If “mama” had a twin sister but only one of them was present at a time, uniqueness could be assumed by the child (or any other observer, for that matter). The feeling of familiarity we have towards individual persons we know (who we normally of course consider to be unique) may be dramatically disturbed: patients with Capgras syndrome suffer from the delusion that people they are highly familiar with are doubles (see also the beautiful novel The Echo Maker by Richard Powers 2006 on this fascinating subject). Leaving psychiatric syndromes aside, having a percept of a recurrent thing does not mean knowing about that thing (“to have at least a minitheory”; p. 117 in Mandler 1988). It is in principle just the ability to discriminate mama shapes from mouse shapes, to reidenitfy them, and to respond with certain motor patterns. Learning is based on stimulus discrimination and generalization; with knowledge restricted to the iconic level, “the baby could generalize from the family cat to the neighbor’s cat, but whatever behavior it had learned to display in the presence of cats would probably not transfer to dogs, and even less likely to ﬁsh or birds.” (p. 20 in Mandler 2004a). Iconic units of knowledge, i.e., percepts and motor gestalts, are unimodal and probably localized in a highly modular fashion in biological brains. I am aware of the fact that percepts are often considered to be multimodal, integrating information from various sensory systems (Ernst and B¨ulthoff 2004), and that there is evidence for cross-modal interactions in areas considered to be unisensory up to now (Kayser and Logothetis 2007), but I think that representations consisting of multimodal associations belong to the next higher level of knowledge, i.e., the indexical level. Also, even if a visual percept such as a face is inﬂuenced by crossmodal interactions, it still is a visual percept. In concluding, it should be pointed out that making iconic statements of the type same same same is “something that we don’t do. It is, so to speak, the act of not making a distinction.” (p. 74 in Deacon 1998). Registering no change is the default, and this blocks out most of the information that is irrelevant in given situations. “Whether because of boredom or limitations of a minimal nervous system, there are times when almost anything can be iconic of anything else (stuff, stuff, stuff . . .).”

Knowledge Systems of the Brain

179

Fig. 1 A moth hoping that potential predators make iconic classiﬁcations “same same same”

(p. 76 in Deacon 1998). Deacon in this context gives camouﬂage as an example: “A moth on a tree whose wings resemble the graininess and color of the bark, though not perfectly, can still escape being eaten by a bird if the bird is inattentive and interprets the moth’s wings as just more tree.” (p. 75 in Deacon 1988; my italics). I think Fig. 1 shows what Deacon had in mind.

3 Indexes: Associations and Higher-Order Classes Although on the iconic level nothing is known about the substances which are represented by percepts – the brain only knows “there is this thing again” or “there is one of those things again” (the computational requirements necessary for this kind of knowledge should not be underestimated, however) – there is the potential of learning more because there are recurrent combinations and sequences of these percepts enabling the formation of concepts. Millikan (who, if I understand her right, sees percepts as “early substance concepts”) describes this most pertinently: “From the standpoint of an organism that wishes to learn, the most useful and accessible subjects of knowledge are things that retain many of their properties, hence potentials for theoretical or practical use, over numerous encounters with them. This makes it possible for the organism to store knowledge about the thing collected on earlier encounters for use on later occasions, the knowledge retaining its validity over time. Substances are (by deﬁnition) what afford this sort of opportunity to a learner.” (p. 531 in Millikan 1999). Indexical knowledge is gained by repeated correlation/association of iconic units or the formation of higher-order classes. As a ﬁrst approximation, consider again a simple robot in a world with red stationary things, that resist (stop the robot) if touched, and green and blue moving things giving way (do not stop the robot)

180

J. Ilmberger

if touched. The three associations based on color (red – resisting, green – giving way, blue – giving way) could be learned individually during repeated encounters, and if the correlations are strong enough, the robot could stop to approach resisting objects, thus saving energy. If, however, the robot was programmed to detect commonalities between percepts, it could make use of categorical categorization and form a category of moving things that may be approached. Forming higher classes instead of storing single percept-action elements is much more economic if the world gets more complex. This formation of superordinates is abstraction, of course, and if a yellow triangular moving thing were to appear in this simple world, the robot could infer that it can be approached (of course it could be wrong, and other and ﬁner classiﬁcations would be needed). This concept formation through abstraction is also found rather early in human infants as can be shown by so-called generalized imitation tests. Mandler summarizes these experiments as follows: “A series of studies showed that after seeing modeled behavior with one animal, 9- to 14-month-olds would use any other animal to imitate animal-speciﬁc behaviors, such as drinking or sleeping, but not a non-animal. . . . By contrast, they typically refused to imitate modeled actions that were inappropriate to a kind, such as putting a vehicle to bed or giving it a drink.” (p. 511 in Mandler 2004b). Clearly, the child not only is able to discriminate dogs from cars on an iconic/percept level, but children at this age know something about these objects, they know about commonalities between classes of percepts, and these commonalities are used to form higher classes such as living things versus nonliving things. In the case of the robot this identiﬁcation of a commonality may seem rather trivial as “motion” very obviously “is there” as a feature, but in a more complex or natural surrounding it is a very difﬁcult task to abstract regularities, since there is a potentially indeﬁnite number of commonalities between objects, both at a given time and even more so over time. The question is which of these communalities are chosen, and how might this be performed by nervous systems. As a basic example, I again cite Mandler (2004a), who gives a vivid description of how a baby could acquire the concept “animate object”: “Consider again the baby who has not yet developed expectations about the world, whose foveal acuity is still poor, but whose attention is attracted to moving objects . . . What might this infant notice about events like the following? She sees an object nearby, she cries, the object begins to move, approaches, looms, and she is picked up. . . . she might not be able to analyze much more than that an object began to move independently of the rest of the surround, moved on a somewhat irregular path, did so contingently on her cries, and ended up by interacting with her.” (p. 72). Over repeated observations, three discriminations can be made: “biological versus nonbiological motion, contingent motion, and self- versus other-instigated motion” (p. 71), which can be redescribed into the concept “animate.” Mandler calls this process of concept formation perceptual meaning analysis. For more details on conceptual development see Mandler (2004a) and Karmiloff-Smith (1996). The other question is how brains might be able to detect commonalities on a computational basis. Commonalities can be seen as “hidden environmental variables

Knowledge Systems of the Brain

181

that are involved in regularities but are reﬂected only implicitly in the known variables . . . That is, we should look for ‘something’ in one subset of the known variables that correlates with ‘something’ in another such subset.” (p. 192 in Favorov and Ryder 2004). These authors present a physiological computational model using synaptic learning in dendrites of pyramidal cells as a possible underlying mechanism. The basic idea is this: “A pyramidal cell’s dendrites discover correlated functions of environmental variables, and the cell thereby tunes to the source of those correlations. Thus a new representation is created, in a way that solves the problem with Hume’s abstraction by ensuring that newly created representations are predictively related to others. Such new representations serve as inputs to cells elsewhere in the network, allowing them to discover more deeply hidden sources of correlation.” (p. 191 in Ryder and Favorov 2001). Computational details and simulation data are given in the papers cited. The term “concept” has been used in philosophy and cognitive sciences with a large variety of meanings and connotations over the centuries (a compilation of original papers is provided in Margolis and Laurence 1999, together with an extensive overview). Within the taxonomy described here, concepts are associated iconic units as well as higher, abstracted classes of iconic units. Terms used in a similar meaning are “simulators” (Barsalou 1999, 2005), “vehicles” (Prinz 2005), and “cognits” (Fuster 2006). No symbolic language, either verbal or other, is involved in these processes in robots or infants of early age. Knowledge on the indexical level may be independent from language. Concepts are abilities to interact with the world on the basis of internal representations, i.e., they almost always have motor components. “Simulators1 do not arise in a vacuum but develop to track meaningful units in the world. As a result, knowledge can accumulate for each unit over time and support optimal interactions with it. . . . Meaningful units include important individuals . . . and categories, where a category is a set of individuals in the environment or introspection. Once a simulator becomes established in memory for a category, it helps identify members of the category on subsequent occasions, and it provides categorical inference about them . . .” (p. 587 in Barsalou 1999). A child will at a very early stage form a multimodal concept of “mama” (face, smell, feel, movement characteristics) including motor gestalts (smile, reach out), and the presence of one conceptual element will be an index of other elements, enabling inference and prediction. During ongoing experience recurrent correlated features and new combinations lead to a consolidation and enrichment of concepts, supplemented by the differentiation of motor gestalts (e.g., ) representing appropriate actions. It is very likely that in living systems the perceptual and motor components share neuronal resources and coding (as described in the common-coding approach by Prinz 1997 and Knoblich and Prinz 2005), thus enabling fast and economic action planning and execution. The activation of conceptual knowledge depends on given situations; not all available knowledge may be needed within a certain background. If I go to a supermarket to buy bread and milk, I do not need to remember information on how 1

Barsalou’s equivalent to concepts.

182

J. Ilmberger

these items are manufactured or the stock market prices of the supermarket’s parent company, but I do need to remember where to ﬁnd them. Only parts of conceptual knowledge are needed at a time. Barsalou (2005) calls this partial activation of concepts “situated conceptualization.” He deﬁnes it as “. . . a multimodal simulation that supports one speciﬁc course of situated action with a particular category instance” (p. 620 in Barsalou 2005). He then speciﬁes: “A given concept produces many different situated conceptualizations, each tailored to different instances in different settings. A situated conceptualization creates the experience of ‘being there’ with a category instance in a setting via integrated simulations of objects, settings, actions, and introspections. On recognizing a familiar type of instance, an entrenched situated conceptualization associated with it becomes active, which provides relevant inferences via pattern completion.” (p. 620 in Barsalou 2005). For a wealth of empirical data on the situated nature of concepts see Yeh and Barsalou (2006). As already stated, no language (linguistic or otherwise) has to be involved on the knowledge level of indexical/conceptual content. Some robots and most nonhuman animals have at least parts of this kind of knowledge. On the indexical level, single gestures or utterances may be used for communication, but such utterances are just another element of the concept. Even if these utterances seem sophisticated (e.g., appearing as names of things as in vervet monkeys’ different alarm calls for snakes, leopards, or eagles as reported by Seyfarth et al. 1980), they are just indexes for different classes of predators. The advantages of this kind of animal communication for survival are evident, but it does not involve using language in the sense that humans use symbol-based languages as described in the following section.

4 Symbols: Multiple Language Systems Single aspects of indexical knowledge may be communicated by vocal or motor gestures; however, this form of communication is rather inﬂexible and limited as the respective signs are just narrow indexes. The potential richness of concepts cannot be expressed. Contrary to that, symbols are signs that get their meaning from objects and from other symbols of a language; symbols may mean different things in different language contexts. Sets of symbols with grammars will be called languages in this taxonomy. Examples of symbols used by humans are linguistic, musical, mathematical, or bodily symbols. My personal notion of humans having several languages available for communication is based on Gardner’s theory of multiple intelligences (Gardner 1983). I am aware of the long list of taxonomies of cognitive processes or mental faculties developed over the centuries (for an excellent and critical discussion of this topic see Uttal 2001, especially Chap. 3), but with respect to the symbolic level discussed in this chapter Gardner gives two hard criteria that help to delineate the various content systems (in my terms “knowledges”): a knowledge domain may be severed selectively by a neurological condition (e.g., aphasia, amusia) or it may be extremely rich in human experts (mathematicians, dancers) – see Gardner (1983) for details. Exploring the knowledge structures available in those various language systems would go far beyond the scope of this chapter; also, this

Knowledge Systems of the Brain

183

is a wide area of future research. In what follows, I will concentrate on natural language as it is the most extensively used system in human communication. However, it should be kept in mind that although natural (linguistic) language is of central importance in humans, (natural) “language is only a tiny domain of the application of the notion of mental meaning . . . it is much more common for there to be meaning without language than meaning with language.” (p. 1039 in Eliasmith 2005). In initial natural language learning, words are directly coupled to indexical content, i.e., the word “cat” is coupled with the infant’s concept . At this stage of language development, words are just indexes pointing to a concept. Correspondence is not a one-to-one relation, as the phenomenon of overextension shows. Early words are used in a broader way than in adult language (e.g., using the word “dog” to include foxes; see Mandler 2004b). Nevertheless, early words may be said to be grounded in the conceptual (and thus perceptual) system (for an extensive discussion of cognitive symbol grounding, see Harnad 1990 and Cangelosi 2005). During further learning, words become interconnected to form a separate layer of symbolic knowledge. Symbols thus may be grounded in concepts, but also their meaning becomes more and more deﬁned by their relation to other symbols. Deacon has pointed this out: “It is by virtue of this sort of dual reference, to objects and to other words . . ., that a word conveys the information necessary to pick out objects of reference. This duality of reference is captured in the classic distinction between sense and reference. Words point to objects (reference) and words point to other words (sense), but we use the sense to pick out the reference, not vice versa. . . . This referential relationship between the words – words systematically indicating other words – forms a system of higher-order relationships that allows words to be about indexical relationships, and not just indices in themselves.” (p. 82f in Deacon 1998). Knowledge coded in words may be communicated to others. Linguistic symbols (as compared with other symbol systems) are especially effective in the transfer of knowledge between individuals (were it otherwise, we all would dance or paint in communication). Knowledge transfer is the most important value of language; an example is knowledge of categories: “Language allows us to acquire new categories indirectly, through ‘hearsay’, without having to go through the time-consuming and risky process of direct trial-and-error learning. Someone who already knows can just tell me the features of an ‘X’ that will allow me to recognize it as an ‘X’.” (p. 37 in Harnad 2005; italics by Harnad). If, in addition to hearsay, there is a notational system available (written languages), the external ﬁxation of knowledge becomes possible. Transfer and accumulation of knowledge over generations in this case is even more effective and provides an enormous ratchet effect (Tomasello 2000) in the development of a community. The use of languages with symbols thus may be seen as the basis of human culture (the difference from nonhuman cognition possibly being not dramatic otherwise, but see Premack 2007 for more dissimilarities between animal und human abilities): “The vast gulf that seems to separate what humans and other primates can do cognitively – in the domain of mathematics, as just one instance – in many, if not most, cases is the result of fairly small differences of individual psychology that enable humans to accumulate knowledge across generations and to use collective artefacts such as linguistic and mathematical symbols” (p. 626 in Call and Tomasello 2005).

184

J. Ilmberger

5 Interactions Between Knowledge Layers Symbolic knowledge builds on indexical knowledge, and indexical knowledge builds on iconic knowledge. Surely there are dynamic interactions between the different levels, such as the interplay between indexical and symbolic contents during learning a language. These interactions raise important questions, such as the relation between thought and language (e.g., What is the relation between knowledge coded in languages and conceptually based knowledge? How does a language disturbance (e.g., aphasia, amusia) relate to other knowledge? What are the knowledges of artiﬁcial and nonhuman biological systems? As appetizers, see Bachmann and Cannon 2005; Call and Tomasello 2005; Papafragou 2005). A discussion of just the basic topics would require a separate chapter, but I think that on the basis of the taxonomy of knowledges described here, research questions such as these can be formulated more easily and clearly.

6 Conclusion The taxonomy of knowledges described in this chapter, with iconic, indexical, and symbolic layers (Fig. 2), is applicable in cognitive sciences such as neurophilosophy, neuropsychology, or robotics. The main goal in writing this chapter was the

Fig. 2 Taxonomy of “knowledges”

Knowledge Systems of the Brain

185

reduction of terminological confusion; a clear terminology will be helpful both in the communication between researchers in the various ﬁelds as well as for the identiﬁcation of research topics. A major focus of knowledge research has been natural language; one major ﬁnding with respect to the taxonomy proposed in this chapter is that lesion and imaging studies by Damasio et al. (2004) have shown, by contrasting processes of recognition versus naming of objects, that in human brains linguistic (symbolic) and conceptual (indexical) knowledges differ at least partly in their neural substrates. There are massive interactions between these two levels, however; it recently could be shown that brain activation patterns observed while subjects were thinking of properties of speciﬁc objects (conceptual knowledge) can be predicted by analysis of the linguistic-semantic features associated with the objects name (Mitchell et al. 2008). In parallel, interest in other languages is increasing, as books like The Languages of the Brain (Galaburda et al. 2002) or, as one example, functional imaging studies in dancers (Calvo-Merino et al. 2005) show. Future research on those other language domains will be a very exciting enterprise. On the basis of a unifying taxonomical frame, it will be much easier to integrate data from machine learning, animal research, and investigations with humans (behavioral, imaging, and brain-lesion studies) and to explore the various forms of embodiment of these knowledge structures in artiﬁcial and biological brains.

References Adler J, Tso W-W (1974) “Decision”-making in bacteria: chemotactic response of Escherichia coli to conﬂicting stimuli. Science 184:1292–1294 Bachmann P, Cannon TD (2005) Cognitive and neuroscience aspects of thought disorder. In: Holyoak KJ, Morrison RG (eds) The Cambridge handbook of thinking and reasoning. Cambridge University Press, Cambridge, pp 493–526 Barsalou LW (1999) Perceptual symbol systems. Behav Brain Sci 22:577–660 Barsalou LW (2005) Situated conceptualization. In: Cohen H, Lefebvre C (eds) Handbook of categorization in cognitive science. Elsevier, Amsterdam, pp 619–650 Breidbach O, Jost J (2006) On the gestalt concept. Theory Biosci 125:19–36 Call J, Tomasello M (2005) Reasoning and thinking in nohuman primates. In: Holyoak KJ, Morrison RG (eds) The Cambridge handbook of thinking and reasoning. Cambridge University Press, Cambridge, pp 607–632 Calvo-Merino B, Glaser DE, Gr`ezes J, Passingham RE, Haggard P (2005) Action observation and acquired motor skills: an fMRI study with expert dancers. Cereb Cortex 15:1243–1249 Cangelosi A (2005) Approaches to grounding symbols in perceptual and sensorimotor categories. In: Cohen H, Lefebvre C (eds) Handbook of categorization in cognitive science. Elsevier, Amsterdam, pp 719–737 Damasio H, Tranel D, Grabowski T, Adolphs R, Damasio A (2004) Neural systems behind word and concept retrieval. Cognition 92:179–229 Deacon T (1998) The symbolic species. Penguin Books, London Eliasmith C (2005) Neurosemantics and categories. In: Cohen H, Lefebvre C (eds) Handbook of categorization in cognitive science. Elsevier, Amsterdam, pp 1035–1054 Ernst MO, B¨ulthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8: 162–169 Favorov OV, Ryder D (2004) SINBAD: a neocortical mechanism for discovering environmental variables and regularities hidden in sensory input. Biol Cybern 90:191–202

186

J. Ilmberger

Fuster JM (2006) The cognit: a network model of cortical representation. Int J Psychophysiol 60:125–132 Galaburda AM, Kosslyn SM, Christen Y (eds) (2002) The languages of the brain. Harvard University Press, Cambridge Gardner H (1983) Frames of mind. The theory of multiple intelligences. Basic Books, New York Harnad S (1990) The symbol grounding problem. Physica D 42:335–346 Harnad S (2005) To cognize is to categorize: cognition is categorization. In: Cohen H, Lefebvre C (eds) Handbook of categorization in cognitive science. Elsevier, Amsterdam, pp 20–43 Hayek FA (1976) The sensory order. University of Chicago Press, Chicago Holyoak KJ, Morrison RG (2005) Thinking and reasoning: a reader’s guide. In: Holyoak KJ, Morrison RG (eds) The Cambridge handbook of thinking and reasoning. Cambridge University Press, Cambridge, pp 1–9 Karmiloff-Smith A (1996) Beyond modularity. MIT Press, Cambridge Kayser C, Logothetis NK (2007) Do early sensory cortices integrate cross-modal information? Brain Struct Funct 212:121–132 Knoblich G, Prinz W (2005) Linking perception and action: an ideomotor approach. In: Freund HJ, Jeannerod M, Hallett M, Leiguarda R (eds) Higher-order motor disorders. Oxford University Press, Oxford, pp 79–104 Komatsu H (2006) The neural mechanisms of perceptual ﬁlling-in. Nat Neurosci 7:220–231 Konczak J (2005) On the notion of motor primitives in humans and robots. In: Berthouze L, Kaplan F, Kouzima H, Yano H, Konczak J, Metta G et al. (eds) Proceedings of the ﬁfth international workshop on epigenetic robotics: modeling cognitive development in robotic systems. Lund University cognitive studies, vol 123. Lund University, Lund, pp 47–53 Mandler JM (1988) How to build a baby: on the development of an accessible representational system. Cogn Dev 3:113–136 Mandler JM (2004a) The foundations of mind. Oxford University Press, Oxford Mandler JM (2004b) Thought before language. Trends Cogn Sci 8:508–513 Margolis E, Laurence S (eds) (1999) Concepts. MIT Press, Cambridge Millikan RG (1999) A common structure for concepts of individuals, stuffs, and real kinds: more mama, more milk, and more mouse. In: Margolis E, Laurence S (eds) Concepts. MIT Press, Cambridge, pp 525–547 Mitchell TM, Shinkareva SV, Carlson A, Chang K-M, Malave VL, Mason RA, Just MA (2008) Predicting human brain activity associated with the meanings of nouns. Science 320:1191–1195 Papafragou A (2005) Relations between language and thought: individuation and the count/mass distinction. In: Cohen H, Lefebvre C (eds) Handbook of categorization in cognitive science. Elsevier, Amsterdam, pp 255–275 Powers R (2006) The echo maker. Farrar, Straus and Giroux, New York Premack D (2007) Human and animal cognition: continuity and discontinuity. Proc Natl Acad Sci U S A 104:13861–13867 Prinz W (1997) Perception and action planning. Eur J Cogn Psychol 9:129–154 Prinz JJ (2005) The return of concept empiricism. In: Cohen H, Lefebvre C (eds) Handbook of categorization in cognitive science. Elsevier, Amsterdam, pp 679–695 Ryder D, Favorov OV (2001) The new associationism: a neural explanation for the predictive powers of cerebral cortex. Brain Mind 2:161–194 Seyfarth RM, Cheney DL, Marler P (1980) Monkey responses to three different alarm calls: evidence of predator classiﬁcation and semantic communication. Science 210:801–803 Spelke E (1988) The origins of physical knowledge. In: Weiskrantz L (ed) Thought without language. Clarendon, Oxford, pp 168–184 Tomasello M (2000) The cultural origins of human cognition. Harvard University Press, Cambridge Uttal WR (2001) The new phrenology: the limits of localizing cognitive processes in the brain. MIT Press, Cambridge Vergassola M, Villermaux E, Shraiman BI (2007) “Infotaxis” as a strategy for searching without gradients. Nature 445:406–409 Yeh W, Barsalou LW (2006) The situated nature of concepts. Am J Psychol 119:349–384

Neural Representation of Time and Timing Processes Elsbieta Szelag(¬), Joanna Dreszer, Monika Lewandowska, and Aneta Szymaszek

Abstract This chapter reviews existing studies on neural representation of time and timing processes. New ﬁndings in clinical neuropsychology, functional magnetic resonance imaging, electrophysiology, and psychophysics are presented to explain how the temporal information is processed within our brains. The literature data are illustrated with results of our ﬁndings. We outline the taxonomy of time perception to provide a background for discussing existing experimental studies. Evidence has indicated that similar brain structures are involved in both subsecond and suprasecond timing, implicating that temporal processing in these two ranges is probably mediated by common neural networks.

1 Introduction Temporal information processing (TIP) is a matter of interest and a broad subject in cognitive psychology. All aspects of human cognition, like language, attention, memory, motor control or decision making, can be characterized by the speciﬁc temporal structure and have a temporal dimension. Thus, patterning in time seems an essential feature of our ‘working brains’. Psychology of time, therefore, has focused studies on human capacities in keeping time, estimating, perceiving and conceiving time. To follow these activities, as well as to understand our cognitive functioning, one should distinguish between content-related functions (e.g. language, attention, memory, etc.) and temporal machinery, which can be considered as logistic-related functions. Existing empirical studies usually concentrate on the former functions.

E. Szelag Laboratory of Neuropsychology, Nencki Institute of Experimental Biology, 3 Pasteur Street, 02-093 Warsaw, Poland [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

187

188

E. Szelag et al.

However, to provide a better insight into our mental activity, it is necessary to focus also on the logistic-related functions which provide a temporal structure for our every action, or every perceived stimulus on a presemantic level, thus prior to any semantic evaluation. Temporal control of perceptual and motor acts has been neglected for a long time in experimental studies. Over the last few decades, psychology of time has ﬂourished, as evidenced by the exponential increase in the number of publications and scientiﬁc meetings. It has become clear that without the knowledge of temporal mechanisms, human cognition can hardly be understood. Two main aspects of the expansion of timing studies should be mentioned. First, new research avenues have opened because of multidisciplinary approaches to this area, including modern experimental technologies, like neuroimaging. Second, clinical applications of theoretical concepts have unveiled new horizons in rehabilitation. Application of knowledge from experimental studies on time and timing to neurorehabilitation seems very promising for many patients (both children and adults) suffering from different language deﬁcits, e.g. language-learning impairment, dyslexia, aphasia or cognitive impairments. Accordingly, better understanding of neural representation of TIP as well as of the underlying neuronal machinery seem of great importance on both theoretical and practical levels. Despite increased interest in this topic, the cortical representation of TIP has still been difﬁcult to characterize in detail and seems underrepresented in the existing literature.

2 Taxonomies of Time Perception Multidisciplinary approaches to TIP require a taxonomy system in order to prepare a background for empirical studies. The discussion of cortical representation of TIP should be, therefore, followed by a description of time taxonomy systems. Several fundamental questions in this ﬁeld were formulated during the 1960s by, e.g., Karl Ernst von Baer, Ernst Mach and Karl Donders. However, a pioneer in time taxonomies was Paul Fraisse (1984), who identiﬁed two elementary temporal experiences: perception of succession (limited up to about 100 ms) and perception of duration (up to 2–5 s). Durations that exceeded the latter time limit are processed by other means and are considered as estimation of duration. Ernst P¨oppel is unquestionably an expert in the area of psychology of time. His experimental and theoretical work has gained worldwide recognition, being referred to in many studies. P¨oppel (1994, 1997) suggested that human beings obtain access to experienced time from elementary time experiences. He developed a hierarchical system of time perception in which different elementary time experiences are incorporated at different processing levels. Accordingly, at least four such elementary experiences can be identiﬁed: simultaneity (or nonsimultaneity), succession (or temporal order, TO, of events), subjective present (or “now”) and duration. This hierarchical taxonomy relates particular experiences to one another; they may appear

Neural Representation of Time and Timing Processes

189

as independent phenomena of subjective reality. These temporally discrete system states are observed at different levels of information processing, i.e. milliseconds, seconds, minutes or more. They may serve as platforms to integrate the incoming temporal information, giving rise to the creation of personal identity of our percepts and motor acts. Converging evidence from different areas of experimental psychology supports the existence of time-processing platforms, also called time windows, which have been used as a reference in attempts to integrate data and concepts from current experimental studies. The idea of different processing platforms, related to distinct timing mechanisms, has also been reﬂected in recent psychophysical, neuropsychological, clinical, neuroimaging, pharmacological and neurochemical studies (Lewis and Miall 2003a, 2006; Rammsayer 1999; Wittmann 1999). These authors argued for the possibility of different neural systems, processing intervals in subsecond or suprasecond time ranges. Accordingly, subsecond timing was supposed to be more automatic and reﬂects sensory mechanisms beyond cognitive control. This processing platform has often been related to processes associated with skilled movements of subsecond durations or coordination of muscles during motor actions. On the other hand, the suprasecond processing platform seems to be more cognitively dependent; therefore, timing mechanisms interact in a complex way with cognitive processes, associated with attention, working or reference memory, spatial abilities or context task speciﬁcation. Since such evidence indicated a clear dissociation between millisecond and multisecond temporal ranges, different brain structures were expected to be involved during experience of sub- and suprasecond timing. Despite these pieces of evidence, new neuroimaging comparisons of these two processing platforms have frequently indicated disparate relationships, pointing to some areas consistently activated in both millisecond and multisecond ranges (Fig. 1). In the next sections we provide an overview of the existing data published in the literature, illustrating them with our recent ﬁndings on neural representation of the two time levels. This search for neural substrates of these processing platforms will focus on neuroimaging, clinical and psychophysical data.

3 Evidence from Neuroimaging Studies Discussion of modern neuroimaging techniques constitutes an important part of this chapter. During the few last years there has been increased interest in brain mapping studies, using several neuroimaging methods which have been helpful also in studying TIP. However, some important limitations of these approaches should be mentioned. For example, electrophysiological methods are characterized by relatively good temporal resolution; thus, they may explain time-related dynamic changes during TIP. These methods, however, have rather poor spatial resolution: thus, it is difﬁcult to draw conclusions with respect to time ‘geography’ within a brain. In contrast, functional magnetic resonance imaging (fMRI) offers good spatial reso-

190

E. Szelag et al.

Fig. 1 Cut-off between automatic and cognitive timing systems

lution, thus helping identify particular brain structures during the performance of a given timing task. Hence, overlapping evidence from fMRI and electrophysiological studies may provide more complex insight into neural representation of time experience.

3.1 Electrophysiological Studies There are some indications that electrophysiological brain activity is designed to process the temporal information on a single-neuron level. Referring to taxonomies of time perception described above, it is related to TIP in the some tens of milliseconds range, speciﬁcally to mechanisms controlling the perception of succession or TO of events (Efron 1963; Fink et al. 2005, 2006; Hirsh and Sherrick 1961; reviewed in Szelag et al. 2004c).

Neural Representation of Time and Timing Processes

191

The evidence supporting this assumption came from studies employing electrical (electroencephalography) or magnetic (magnetoencephalography) recordings of the human brain. They revealed spontaneous coherent oscillatory activity near 40 Hz (Bas¸ar-Eroglu et al. 1996; Joliot et al. 1994; P¨oppel 1971). Each oscillation period of about 30 ms may constitute an elementary neuronal basis for temporal binding of incoming events, aimed probably at reducing ‘perceptual noise’ or the complexity of incoming information. Each period of oscillatory activity may represent a simple atemporal processing unit within which the information is treated as co-temporal. Because of this temporally imprecise processing, the TO can be detected only if two events are processed within two successive oscillation periods, e.g. if they are separated by a time interval of some tens of milliseconds. In contrast, sensory information coming in during one oscillation period has been assumed to be simultaneous. This central oscillatory activity is probably implemented within neuronal assemblies by corticothalamic pathways and represents a neurophysiological background for sensory processing. This approach to TIP found support in several behavioural studies performed during the last 50 years in which subjects reported the TO of two auditory, visual or tactile sensory stimuli presented in a rapid succession (Szelag et al. 2004c; Szymaszek et al. 2008; Wittmann and Fink 2004). These studies revealed that the TO of successive stimuli can be correctly identiﬁed if they are separated by an interval of at least 30 ms, which is considered as a TO threshold (see also later). Auditory evoked potentials provide another approach to studying electrophysiological correlates of TIP. In our study (Lewandowska et al. 2008), cortical activations associated with TO detection of two successively presented tones were compared in two conditions: difﬁcult (below the threshold) and easy (above the threshold). We argued that the amplitude of the P2 component, observed at Cz and Fz sites, constituted an electrophysiological correlate of TO perception with significantly higher amplitude for the difﬁcult than for the easy condition. Moreover, in the difﬁcult condition a negative correlation between P2 amplitude and the number of correct responses was observed. Speciﬁcally, a higher correctness level was recorded for lower P2 amplitudes. This may suggest the allocation of attentional resources to more difﬁcult TO detection. Such a conclusion is in agreement with a hypothesis postulating the involvement of cognitive processes (i.e. attention) in subsecond timing (compare above). In more difﬁcult stimulus condition (shorter pause between the ﬁrst and second stimulus) more attentional resources (e.g. alertness, vigilance, switching or concentration of attention) have to be engaged to follow correctly the identiﬁcation of temporal order. Further evidence on timing in the multisecond range came from contingent negative variation (CNV), i.e. a slow negative wave that develops during temporal reproduction. Elbert et al. (1991) found CNV in reproduction of intervals shorter than 3 s, whereas for longer standards it was reduced or disappeared. To conclude, CNV revealed differences between reproduction of shorter and longer intervals with a cut point at a 3-s interval. This ﬁnding provides electrophysiological evidence for the speciﬁc temporal integration mechanism limited in time up to above 3 s which

192

E. Szelag et al.

was postulated in P¨oppel’s taxonomy of time perception (P¨oppel 2004; also compare above). Accordingly, two distinct temporal processing may be postulated. The former is related to binding of particular incoming events which is limited in time up to approximately a few seconds. The later one is active beyond such integration time window, thus, concerns longer intervals. It is controlled by the other nontemporal mechanisms, like memory or attention. To conclude, studies on CNS provided an electrophysiological correlate of temporal processing at multisecond range. To summarize, electrophysiological evidence may suggest a broad activation in neuronal assemblies underlying TIP in both sub- and suprasecond ranges and, moreover, attentional involvement in millisecond timing.

3.2 fMRI Studies Over the last few decades, several fMRI studies using a wide range of timing tasks (e.g. simple motor tapping tasks or more complex duration judgement tasks) have attempted to investigate the neural basis of human timing experience. Converging evidence from recent fMRI studies supports the involvement of different brain systems in millisecond or multisecond time ranges (Lewis and Miall 2003b, 2006). Overall, recent studies have shown that although some brain areas may be activated in particular temporal ranges, many regions are activated in both sub- and suprasecond ranges (Rubia and Smith 2004). With respect to subsecond durations, the activation of some parts of the motor system was postulated, speciﬁcally that of cerebellum, basal ganglia, premotor and motor cortex. Greater activity of some parts of the motor system during millisecond rather than multisecond intervals was conﬁrmed by Lewis and Miall (2003b, 2006) in a meta-analysis review of 30 neuroimaging studies published in the existing literature. A possible mechanism underlying such activation would be based on the neural architecture of these structures which could feasibly measure millisecond durations. For example, Matsuzaka et al. (1992) hypothesized that temporally predictable behaviours could result in the build-up of cells sensitive to movement preparations which may be specialized in timing processes. Furthermore, TIP in the millisecond range may also be represented in the auditory cortex, speciﬁcally in the middle and superior temporal gyri (Lewis and Miall, 2003a). The involvement of these gyri was also reported in experiments involving no auditory cues. The auditory activity was interpreted somewhat inconsistently in recent neuroimaging literature and was based on the possible use of auditory imagery of visual or tactile standards. Thus, it should be regarded with caution, because the explanation of auditory cortex activity in the millisecond time range is rather underrepresented in these studies. Below we discuss clinical data which may explain this involvement in more detail. On the other hand, cognitive modules of the prefrontal and parietal cortices are more involved in measurement of longer (suprasecond) rather than shorter

Neural Representation of Time and Timing Processes

193

(subsecond) interval timing. These regions comprise inferior parietal cortex, probably owing to greater involvement of explicit attention during longer intervals, supplementary motor area, dorsolateral and inferior prefrontal cortices. It should be noted that the experimental paradigms used in millisecond and multisecond ranges are often quite different, making it difﬁcult to determine whether different neural systems are used for a particular time range, since disparities in results may be linked either to the duration of measured intervals or to the other procedural factors. To clarify this issue, a direct comparison of brain regions involved in sub- and suprasecond timing was provided by Lewis and Miall (2003a), who used a temporal discrimination paradigm (often regarded as the purest measure of TIP) for standard durations of either 0.6 or 3 s. Overall, they found a common network of activation in both ranges, which comprised general regions involved in temporal discrimination tasks, regardless of experienced duration. In addition they also found speciﬁc activation areas, characteristic selectively for either subsecond or suprasecond timing ranges. The common activation network for both durations comprised bilateral insula and dorsolateral prefrontal cortex, right-hemispheric presupplementary motor area, frontal pole and inferior parietal cortex. In contrast, for the subsecond range, the contribution of motor areas to the non-motor timing task reﬂected activity in the cerebellum and frontal operculum, as well as in the temporal gyri. It suggests specialization of these structures in broad aspects of the timing task. In suprasecond timing, the contribution of attentional processes was reﬂected by stronger parietal activity (associated with longer standards), while the involvement of spatial orientation and memory was reﬂected by the activity in the posterior cingulate area. To sum up, these results support the notion of possible artiﬁcial dichotomy between sub- and suprasecond timing or between motor and perceptive timing (Fig. 1). Both ranges and both functions mentioned appear to be often mediated by similar brain networks and in many cases cannot be separated.

4 Evidence from Clinical Studies Another source of evidence regarding neural representation of timing comes from clinical observations of patients suffering from different neurodegenerative diseases. Such data have attracted the attention of many neuroscientists and have also been a focus for TIP studies. These clinical approaches comprised predominantly focal lesion studies and conﬁrmed the evidence derived from neuroimaging studies for the involvement of different brain areas in TIP (see above). The results of these studies support the idea that similar brain regions are involved in both motor and cognitively mediated timing. Accordingly, cerebellum may be considered as a candidate locus for both sub- and suprasecond timing. Patients with cerebellar lesions display disordered performance for both motor tasks (tapping) and time judgement tasks (temporal discrimination) for both time ranges (Ivry and Keele 1989; Ivry and Richardson

194

E. Szelag et al.

2002). Timing deﬁcits are often reported in patients with cerebellar lesions in processing of both short (hundreds of milliseconds) and long (seconds) intervals. Moreover, the involvement of basal ganglia in time discrimination as well as in planing and execution of motor acts has been reported in several studies. Interaction between an internal clock and dopaminergic neurons localized in basal ganglia seems to be important to understand at the cellular level TIP from seconds to minutes (Meck 1996). A large number of studies on patients with Parkinson’s disease have reported deﬁcits in motor and perceptual timing for intervals within the seconds-tominutes range (Malapani et al. 2002; Pastor et al. 1992). These timing deﬁcits were ameliorated by a speciﬁc dopaminergic treatment, whereas non-medicated Parkinson’s disease patients underestimated time durations in verbal estimation tasks and overestimated the same durations in reproduction tasks, with greater magnitude of overestimation observed at longer intervals (Pastor et al. 1992). These results provide evidence for slower running of the hypothetical internal clock in Parkinson’s disease patients than in healthy controls. To sum up, recent studies have suggested that Parkinson’s disease patients did make inaccurate and variable duration judgments in the seconds-to-minutes range. With respect to involvement of basal ganglia in timing, it seems interesting to note that damage in the right supralenticular white matter, presumably consisting of frontostriatal pathways, was also associated with deﬁcits in time judgements on the suprasecond time level (Rubia et al. 1997). Studies of patients suffering from aphasia following left hemispheric brain damage have also provided an important contribution to our understanding of TIP (Efron 1963; Swisher and Hirsh 1972; Tallal and Newcombe 1978). This type of evidence dates from early observations by Efron and Swisher and Hirsh that aphasic patients displayed parallel deﬁcits in both speech comprehension and rapidly changing temporal information. Such observations were conﬁrmed in several further reports, including our own studies (Szelag et al. 1997; Szelag and P¨oppel 2000; von Steinb¨uchel et al. 1999). We showed, however, important dissociation between speciﬁc language deﬁcits (in phonemic hearing versus effortful non-ﬂuent speech) and temporal deﬁcits (at the subsecond versus the suprasecond level). Speciﬁc left-hemisphere lesions selectively damaged temporal mechanisms operating within either the millisecond or the multisecond time frames. The former range was assessed with an auditory TO threshold (see earlier for explanations), whereas the latter one was assessed with a subjective accentuation paradigm which corresponded to the upper limit of the temporal integration mechanism, operating in a time window of a few seconds (Szelag 1997; P¨oppel 1997, 2004). In patients with left-hemisphere postcentral lesions, suffering from Wernicke’s aphasia (characterized by deﬁcits in phonemic hearing and comprehension) the deﬁcient TO thresholds were reported. In contrast, patient with Broca’s aphasica were unaffected on this processing level. Patients with Broca’s aphasia (characterized by effortful, non-ﬂuent speech) displayed parallel deﬁcits in the suprasecond processing range, which implied that some highly structured syntactic abilities are located in the anterior language area (Broca’s area).

Neural Representation of Time and Timing Processes

195

To conclude, we argue that TIP in aphasics may be selectively affected in either the subsecond or the suprasecond time range, depending on the localization of the lesion and the observed disﬂuency pattern (Szelag et al. 1997). We also postulate that a disruption of timing mechanisms leads to phonological and/or syntactical disorders, commonly observed in aphasic patients. A growing body of evidence supports an association between language and timing. Language deﬁcits of various origins in children and adults have been associated with timing disorders in both time ranges considered. Support for such a conclusion came from studies conducted by Paula Tallal’s group on language-learning-impaired children (Tallal et al. 1996, 1998; Tallal and Newcombe 1978), children and adults with dyslexia (Farmer and Klein 1995), cochlear implant users, aphasics or autistic children studied in our laboratory (Szelag et al. 2004a–c).

5 Evidence from Psychophysical Studies A large body of behavioural evidence regarding discrete information processing has accumulated on both sub- and suprasecond timing. Detailed analysis of these data is beyond the scope of this chapter, and therefore our discussion is limited to the results of our studies on TO threshold which provide convincing evidence for the involvement of cognitive factors in subsecond timing (Szymaszek et al. 2006, 2008). Comparing threshold values in 86 subjects across their life span from 20 to 69 years of age, we found that the subjects’ performance in the TO task generally declined with age. Along with everyday observations, experimental results and theories, our study lends support to the hypothesis of decreased information processing in the elderly (caused, e.g., by slowing down of the brain processing speed or hypothetical internal timing mechanism). However, a novel result of our study showed a clear association between cognitive competences and subsecond timing, where two 1-ms clicks were presented in a rapid sequence. We argue that age-related deterioration in timing is strongly related to deterioration in cognitive functioning, i.e. intellectual abilities and attentional resources. In particular, subjects characterized by a higher level on these two cognitive dimensions displayed better performance in the timing task. Thus, our study emphasized the contribution of mental competencies to subsecond timing.

6 Psychological Time Models: A Brief Retrospective Since the 1960s, several psychological models of TIP have been proposed (Block and Zakay 1996). Let us brieﬂy consider these theories to look closer at the psychology of time.

196

E. Szelag et al.

Two general classes of psychological time models have been postulated: timing without a timer and timing with a timer. In the former class, time experience is a ‘high-dimensional network state’ and constitutes some by-product of information processing. In this model, subjects construct psychological time from processed and stored information. In contrast, in time-with-a-timer models an internal clock consisting of a biological pacemaker is hypothesized to underlie experienced and remembered duration. Such a pacemaker produces a regular series of pulses, the rate of which is modulated by an organism-speciﬁc arousal. Behavioural adjustments to time are modiﬁed by a number of both physiological and cognitive factors. None of these psychological time models related timing to particular brain structures. They rather accounted for the internal operations performed by a subject during temporal experience. They provided a theoretical perspective to cognitive or physiological processes, like attention, reference memory or a general arousal level, serving as a reference in attempts at integrating data and concepts from experimental studies. It is important to stress that the possible existence of different mechanisms and processes suggested by the abovementioned theoretical approaches has been reﬂected in recent neuroimaging as well as clinical data, as discussed above. Experimental veriﬁcation of these theories suggests that the temporal structure may have emerged either from the broad activation of neural networks (as predicted in timingwithout-a-timer models), or that activation of particular brain areas may underlie experienced time (as predicted in timing-with-a-timer models). Activation during TIP may be localized either in particular brain structures (e.g. cerebellum, basal ganglia, Wernicke’s area), or in a broader assembly of neural areas which are synchronized during TIP (electrophysiological data). In light of these considerations, the application of knowledge from the theoretical perspective to neuroimaging data seems very promising. The ﬁndings presented support the notion that temporal brain machinery is not a simple phenomenon but constitutes rather complex logistic-related functions and that some crucial mechanisms or elements of TIP really exist. Several experimental studies and theoretical discussions over the last 50 years have supported the existence of different mechanisms underlying sub- and suprasecond timing. They have focused on either investigation of centralized time areas in the brain related to the internal clock, or distribution of temporal processing throughout the brain, suggested by a state-dependent network.

7 The Future of Timing Studies: Where We Are Going The ﬁnal question that should be considered in this chapter is the following: How should researchers link their interdisciplinary concept of TIP with the requirements of neurorehabilitation to develop modern training devices for improvement of temporal processing within a brain?

Neural Representation of Time and Timing Processes

197

Such training tools could make it possible to transfer of improvements from the time domain to the cognitive domain, because these two areas are closely linked. Nowadays, this search for modern therapeutic tools has mainly focused on language-disordered populations, i.e. children with dysphasia or dyslexia (Farmer and Klein 1995; Merzenich et al. 1996; Tallal et al. 1996). The clinical success of the Fast ForWord program designed by Paula Tallal’s group has been documented in several studies and clinical practice. Current research has focused on applying interventions based on TIP improvements to cognitive deﬁcits. For example, our laboratory, in cooperation with researchers from Munich University, has developed a training program which signiﬁcantly improves both timing and language (auditory comprehension) capacities in aphasics. It has been indicated in the literature that other aspects of human cognition may also be ameliorated by temporal training, e.g. attention, capacity for new learning, motor planning and coordination and man–machine interactions. Acknowledgements Our work is supported by the Polish Ministry for Science and Education Grants No. PBZ-MIN/001/P05/06, 1082/P01/2006/31 and NN402434633 and the Fellowship ‘START’ for A.S. from the Foundation for Polish Science. We wish to thank Ernst P¨oppel for his inspiration to perform these studies.

References Bas¸ar-Eroglu C, Str¨uber D, Sch¨urmann M, Stadler M, Bas¸ar E (1996) Gamma-band responses in the brain: a short review of psychophysiological correlates and functional signiﬁcance. Int J Psychophysiol 24:101–112 Block RA, Zakay D (1996) Models of psychological time revisited. In Helfrich H (ed) Time and mind. Hogrefe & Huber, G¨ottingen, pp 171–195 Efron R (1963) Temporal perception, aphasia and d´ej´a vu. Brain 86:403–424 Elbert T, Ulrich R, Rockstroh B, Lutzenberger W (1991) The processing of temporal intervals reﬂected by CNV-like brain potentials. Psychophysiology 28:648–655 Farmer ME, Klein RM (1995) The evidence for a temporal processing deﬁcit linked to dyslexia: a review. Psychon Bull Rev 2:460–493 Fink M, Churan J, Wittmann M (2005) Assessment of auditory temporal order thresholds – a comparison of different measurement procedures and the inﬂuence of age and gender. Restor Neurol Neurosci 23:1–16 Fink M, Ulbrich P, Churan J, Wittmann M (2006) Stimulus-dependent processing of temporal order. Behav Process 71:344–352 Fraisse P (1984) Perception and estimation of time. Annu Rev Psychol 35:1–36 Hirsh IJ, Sherrick CE (1961) Perceived order in different sense modalities. J Exp Psychol 62: 423–432 Ivry RB, Keele SW (1989) Timing functions of the cerebellum. J Cogn Neurosci 1:136–152 Ivry RB, Richardson TC (2002) Temporal control and coordination: the multiple timer model. Brain Cognit 48:117–132 Joliot M, Ribary U, Llinas R (1994) Human oscillatory brain activity near 40 Hz coexists with cognitive temporal binding. Proc Natl Acad Sci U S A 91:11748–11751 Lewandowska M, Bekisz M, Szymaszek A, Wrobel A, Szelag E (2008) Towards electrophysiological correlates of auditory perception of temporal order. Neurosci Lett. 437:139–143

198

E. Szelag et al.

Lewis PA, Miall RC (2003a) Brain activation patterns during measurement of sub-and suprasecond intervals. Neuropsyychologia 41:1583–1592 Lewis PA, Miall RC (2003b) Distinct system for automatic and cognitively controlled time measurement: evidence from neuroimaging. Curr Opin Neurobiol 13:250–255 Lewis PA, Miall RC (2006) A right hemispheric prefrontal system for cognitive time measurement. Behav Process 71:226–234 Malapani C, Deweer B, Gibbon J (2002) Separating storage from retrieval dysfunction of temporal memory in Parkinson’s disease. J Cogn Neurosci 14:311–322 Matsuzaka Y, Aizawa H, Tanji J (1992) A motor area rostral to the supplementary motor area (presupplementary motor area) in the monkey: neuronal activity during a learned motor task. J Neurophysiol 68:653–662 Meck WH (1996) Neuropharmacology of timing and time perception. Cogn Brain Res 3:227–242 Merzenich MM, Jenkins WM, Johnston P, Schreiner C, Miller SL, Tallal P (1996) Temporal processing deﬁcits of language-learning impaired children ameliorated by training. Science 271:81–84 Pastor MA, Artieda J, Jahanshahi M, Obeso JA (1992) Time estimation and reproduction is abnormal in Parkinson’s disease. Brain 115:211–225 P¨oppel E (1971) Oscillations as possible basis for time perception. Stud Gen 24:85–107 P¨oppel E (1994) Temporal mechanisms in perception. Int Rev Neurobiol 37:185–202 P¨oppel E (1997) A hierarchical model of temporal perception. Trends Cogn Sci 1:56–61 P¨oppel E (2004) Lost in time: a historical frame, elementary processing units and the 3-s window. Acta Neurobiol Exp 64:295–301 Rammsayer T (1999) Neuropharmacological evidence for different timing mechanisms in humans. Q J Exp Psychol B Comp Physiol Psychol 52:273–286 Rubia K, Smith A (2004) The neural correlates of cognitive time management: a review. Acta Neurobiol Exp 64:329–340 Rubia K, Schuri U, Cramon DY, P¨oppel E (1997) Time estimation as a neuronal network property: a lesion study. Neuroreport 8:1273–1276 Swisher L, Hirsh IJ (1972) Brain damage and the ordering of two temporally successive stimuli. Neuropsychologia 10:137–152 Szelag E (1997) Temporal integration of the brain as studied with the metronome paradigm. In: Atmanspacher H, Ruhnau E (eds) Time, temporality, now. Springer, Berlin Heidelberg New York, pp 121–132 Szelag E, P¨oppel E (2000) Temporal perception: a key to understanding language. Behav Brain Sci 23:52 Szelag E, von Steinb¨uchel N, P¨oppel E (1997) Temporal processing disorders in patients with Broca’s aphasia. Neurosci Lett 10:33–36 Szelag E, Kowalska J, Gałkowski T, P¨oppel E (2004a) Temporal processing deﬁcits in high functioning children with autism. Br J Psychol 95:269–282 Szelag E, Kolodziejczyk I, Kanabus M, Szuchnik J, Senderski A (2004b) Deﬁcits of nonverbal auditory perception in postlingually deaf humans using cochlear implants. Neurosci Lett 355: 49–52 Szelag E, Kanabus M, Kolodziejczyk I, Kowalska J, Szuchnik J (2004c) Individual differences in temporal information processing in humans. Acta Neurobiol Exp 64:349–366 Szymaszek A, Szelag E, Sliwowska M (2006) Auditory perception of temporal order in humans: the effect of age, gender listener practice and stimulus presentation mode. Neurosci Lett 403:190–194 Szymaszek A, Sereda M, P¨oppel E, Szelag E (2008) Individual differences in the perception of temporal order: the effect of age and cognition. Cogn Neuropsychol (in preparation) Tallal P, Newcombe F (1978) Impairment of auditory perception and language comprehension in dysphasia. Brain Lang 5:13–24 Tallal P, Miller SL, Bedi G, Byma G, Wang X, Nagarajan SS, Schreiner C, Jenkins WM, Merzenich MM (1996) Language comprehension in language-learning impaired children improved with acoustically modiﬁed speech. Science 271:81–84

Neural Representation of Time and Timing Processes

199

Tallal P, Merzenich MM, Miller S, Jenkins W (1998) Language learning impairments: integrating basic science, technology, and remediation. Exp Brain Res 123:210–219 von Steinb¨uchel N, Wittmann M, Strasburger H, Szelag E (1999) Auditory temporal order judgement is impaired in patients with cortical lesions in posterior regions of the left hemisphere. Neurosci Lett 264:168–171 Wittmann M (1999) Time perception and temporal processing levels of the brain. Chronobiol Int 16:17–32 Wittmann M, Fink M (2004) Time and language-critical remarks on diagnosis and training methods of temporal-order judgement. Acta Neurobiol Exp 64:341–348

Fractionating the System of Deductive Reasoning Vinod Goel

Abstract In this chapter I suggest that the best way to advance our understanding of the neural basis of reasoning is to put equal emphasis on cognitive theory and neuropsychological data and ideas. This approach has led us to view reasoning in terms of a fractionated system that is dynamically conﬁgured in response to task and environmental cues. Three systems that have emerged from our research of the past few years are reviewed: (1) systems for dealing with familiar and unfamiliar material (2) systems for dealing with conﬂict and belief-bias; (3) systems for dealing with certain and uncertain information. These systems are discussed in the context of several dual mechanism theories. It is argued that the data is consistent with Newell and Simon’s account of formal and heuristic processes but not other more radical accounts of dual processes. I conclude by offering a speculative proposal as to how the systems might interact in the course of evaluating logical arguments.

1 Introduction Reasoning is the cognitive activity of drawing inferences from given information. All reasoning involves the claim that one or more propositions (the premises) provide some grounds for accepting another proposition (the conclusion). A subset of arguments are called deductive arguments. Such arguments can be evaluated for validity, a relationship between premises and conclusion involving the claim that the premises provide absolute grounds for accepting the conclusion (i.e., if the premises are true, then the conclusion must be true). A key feature of deduction is that conclusions are contained within the premises and are logically independent of the content of the propositions. As such, deductive reasoning is a good candidate for a

V. Goel Department of Psychology, York University, 4700 Keele St., Toronto, ON, Canada M3J 1P3 [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

203

204

V. Goel

self-contained cognitive module. In fact, it is probably the best and only candidate among higher-level reasoning/thinking processes for a cognitive module. Two theories of reasoning dominate the cognitive literature and provide different characterizations of the nature of this “module.” They differ with respect to the competence knowledge they draw upon, the mental representations they postulate, and the mechanisms they invoke. Mental logic theories (Braine 1978; Henle 1962; Rips 1994) postulate that reasoners have an underlying competence knowledge of the inferential role of the closed-form, or logical, terms of the language (e.g., “all,” “some,” “none,” “and,” etc.). The internal representation of arguments preserves the structural properties of the propositional strings in which the premises are stated. A mechanism of inference is applied to these representations to draw conclusions from premises. Essentially, the claim is that deductive reasoning is a rule-governed process deﬁned over syntactic strings. By contrast, mental model theory (Johnson-Laird 1983; Johnson-Laird and Byrne 1991) postulates that reasoners have an underlying competence knowledge of the meaning of the closed-form, or logical, terms of the language (e.g., “all,” “some,” “none,” “and,” etc.)1 and use this knowledge to construct and search for alternative scenarios.2 The internal representation of arguments preserves the structural properties of the world (e.g., spatial relations) that the propositional string are about rather than the structural properties of the propositional strings themselves. The basic claim is that deductive reasoning is a process requiring spatial manipulation and search. When studies investigating the neural basis of reasoning began a decade ago, a natural, inevitable starting point was to search for a “reasoning module” in the context of one of these two theories (Goel et al. 1997, 1998; Osherson et al. 1998). A system built upon visuospatial processes would be consistent with mental model theory, while a system built upon linguistic/syntactic processes would be consistent with mental logic theory (Goel 2005). After a decade of neuroimaging and patient studies of logical reasoning the data do not support this picture. I will return to this issue later in the chapter. Our approach, informed as much by neuropsychology as by cognitive theory, has been somewhat different. We have looked for double dissociations/breakdowns (i.e., causal joints) in the neural machinery underlying reasoning by examining behavioral data from neurological patients with focal lesions, and neuroimaging data from normal, healthy controls. In terms of the former, we are unaware of a single report of neurological patients with a selective reasoning deﬁcit. In terms of the latter, we ﬁnd some commonality and some variability in neural networks engaged by reasoning tasks across various studies (Goel 2007; Table 1). In the context of this experience, we ﬁnd very little evidence for a “logic module” in the brain. Furthermore, examining the data, in the absence of a commitment to a speciﬁc cognitive theory of reasoning suggests that human reasoning is underwritten by a fractionated 1

Whether there is any substantive difference between ‘knowing the inferential role’ and ‘knowing the meaning’ of the closed-form terms, and thus the two theories is a moot point, debated in the literature. 2 See Newell (1980) for a discussion of the relationship between search and inference.

Visual, nonlinguistic

Visual, nonlinguistic

fMRI

fMRI

fMRI

fMRI

PET

PET

fMRI

Goel et al. (2004a)

Fangmeier et al. (2006) Transitivity (implicit) Acuna et al. (2002)

Heckers et al. (2004)

Categorical syllogisms Goel et al. (1998)

Osherson et al. (1998)

Goel et al. (2000)

Visual, linguistic Visual, linguistic

Visual, linguistic

Visual, nonlinguistic

Auditory, linguistic Visual, linguistic

fMRI

Knauff et al. (2003)

Visual, linguistic Visual, linguistic

Stimuli modalitya

fMRI

PET

Scanning method

Goel and Dolan (2001)

Transitivity (explicit) Goel et al. (1998)

Studies (organized by tasks)

18, 19

18

18, 19

17, 18, 19

18

18, 19

19

19

Occipital lobes RH LH

7, 39, 40 40

7

7, 40

7

7, 40

7

40

39, 40

40

7

7

7, 40

Parietal lobes RH LH

21/22

37, Hi

21, 22, Hi

21

21, 22

37, 21

21, 38 21, 22, Hi

37

Temporal lobes RH LH

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Basal ganglia RH LH RH

24, 32

24

32

24, 32

LH

Cingulate

45

44, 45

45, 46, 47 6

6, 8, 9, 46 6, 47

46, 47 6, 9, 46, 11 6, 9

45, 46 6, 9

LH

(continued)

PSMA, 6

6, 8, 9, 46

6

11, 47

6

6

RH

Frontal lobes

Table 1 Summary of particulars of 19 neuroimaging studies of deductive reasoning and reported regions of activation corresponding most closely to the main effect of reasoning. (Reproduced from Goel 2007.)

Fractionating the System of Deductive Reasoning 205

fMRI

Goel and Dolan (2004)

fMRI

PET

fMRI

Canessa et al. (2005)

Mixed stimuli Goel et al. (1997)

Knauff et al. (2002)

Visual, linguistic Auditory, linguistic

Visual, linguistic

Visual, nonlinguistic Visual, linguistic

Visual, linguistic Visual, linguistic

Visual, linguistic

Visual, linguistic

Stimuli modalitya

19

19

18

18

18

17, 18

19

19

18

17

19

18, 19

17, 18

Occipital lobes RH LH

7, 40

7, 39, 40

39, 40

7

7, 14

7, 39, 40

40

7

37

Parietal lobes RH LH

21, 22

21, 37, 39

21, 22

37

21, 22, 38 39

Temporal lobes RH LH

Yes

Yes

Yes

Yes

Yes

Yes

Basal ganglia RH LH

32

32

24

32

32

31

32

LH

Cingulate RH

Numbers denote Brodmann areas. Cerebellum activations are not noted. Blank cells indicate absence of activation in the region RH right hemisphere; LH left hemisphere; fMRI functional MRI; Hi hippocampus; PSMA pre-sensory-motor area a Refers to the form and manner of presentation of the stimuli b Brodmann areas not provided by authors

PET

Parsons et al. (2001)

Prado and Noveck fMRI (2007) Conditionals (complex) Houde et al. (2000)b PET

fMRI

fMRI

Goel and Dolan (2003)

Conditionals (simple) Noveck et al. (2004)

Scanning method

Studies (organized by tasks)

Table 1 (continued)

6, 9

6, 8, 9, 10, 46

10, 44, 9

6, 45, 46

6

6

RH

6, 9

6, 8, 9, 46

6, 47 9, 46

6, 44, 45

6, 44

LH

Frontal lobes

206 V. Goel

Fractionating the System of Deductive Reasoning

207

system that is dynamically conﬁgured in response to speciﬁc task and environmental cues. Three of these lines of fractionation are reviewed here. Existing cognitive theories are accessed in light of these data and an alternative explanatory framework is provided.

2 Systems for Dealing with Familiar and Unfamiliar Material As mentioned in the previous section, deductive arguments are valid as a function of their logical form. The content of the argument is irrelevant for the determination of validity. Despite this logical truism, one of the most robust, and problematic ﬁndings in the psychology of reasoning is that content has a signiﬁcant effect on the reasoning process (compare arguments 1–3 with 4–9 in Table 2). To explore this issue, we have carried out a series of studies, using syllogisms and transitive inferences, whereby we have held logical form constant and systematically manipulated content of arguments. These studies indicate that two distinct systems are involved in reasoning about familiar and unfamiliar material. More speciﬁcally, a left lateralized frontal-temporal conceptual/language system (Fig. 1a) processes familiar, conceptually coherent material, while a bilateral parietal visuospatial system, with some dorsal frontal involvement (Fig. 1b), processes unfamiliar, nonconceptual or conceptually incoherent material. The involvement of the left frontal-temporal system in reasoning about familiar or meaningful content has also been demonstrated in neurological patients with focal unilateral lesions to prefrontal cortex (parietal lobes intact), using the Wason card selection task (Goel et al. 2004b). These patients performed as well as normal controls in the arbitrary version of the task, but unlike the normal controls they failed to beneﬁt from the presentation of familiar content in the meaningful version of the

Reasoning with familiar material

a

Reasoning with unfamiliar material

b

Fig. 1 (a) Reasoning about familiar material (all apples are red fruit; all red fruit is nutritious; all apples are nutritious) activates a left frontal (BA 47) temporal (BA 21/22) system. (b) Reasoning about unfamiliar material (all A are B; all B are C; all A are C) activates bilateral parietal lobes (BA 7, 40) and dorsal prefrontal cortex (BA 6). (Reproduced from Goel et al. 2000.)

208

V. Goel

task. In fact, the latter result was driven by the exceptionally poor performance of patients with left frontal lobe lesions. Patients with lesions to right prefrontal cortex performed for as well as normal controls. There is even some evidence to suggest that the response of the frontal-temporal system to familiar situations may be content-speciﬁc to some degree (in keeping with some content speciﬁcity in the organization of temporal lobes) (McCarthy and Warrington 1990). For example, while middle temporal lobe regions are activated when reasoning about categorical statements such as “All dogs are pets,” in the case of making transitive inferences about familiar spatial environments, reasoning is mediated by posterior hippocampus and parahippocampal gyrus, the same structures that underwrite spatial memory and navigation tasks (Goel et al. 2004a). Perhaps the most robust example of content speciﬁcity in the organization of the heuristic system is the “theory of mind” reasoning system identiﬁed by a number of studies (Fletcher et al. 1995; Goel et al. 1995).

3 Systems for Dealing with Conﬂict and Belief-Bias A robust consequence of the content effect is that subjects perform better in reasoning tasks when the logical conclusion is consistent with their beliefs about the world (arguments 4–6 in Table 2) than when it is inconsistent with their beliefs (arguments 7–9 in Table 2) (Evans et al. 1983; Wilkins 1928). In the former case, subjects’ beliefs facilitate the task, while in the latter case they inhibit it. Within inhibitory belief trials the prepotent response is the incorrect response associated with belief-bias. Incorrect responses in such trials indicate that subjects failed to detect the conﬂict between their beliefs and the logical inference and/or to inhibit the prepotent response associated with the belief-bias. These belief-biased responses activate ventromedial prefrontal cortex (BA 11, 32), highlighting its role in nonlogTable 2 Three-term transitive arguments sorted into nine categories (see the text) Determinate arguments Valid Invalid

Indeterminate arguments Invalid

No meaningful City A is north of city B City A is north of city B City A is north of city B content City B is north of city C City B is north of city C City A is north of city C (content has no City A is north of city C City C is north of city A City B is north of city C effect on task) (1) (2) (3) Congruent London is north of Paris (content Paris is north of Cairo facilitates task) London is north of Cairo (4) Incongurent London is north of Paris (content Cairo is north of London inhibits task) Cairo is north of Paris (7)

London is north of Paris Paris is north of Cairo Cairo is north of London (5) London is north of Paris Cairo is north of London Paris is north of Cairo (8)

London is north of Paris London is north of Cairo Cairo is north of Paris (6) London is north of Paris London is north of Cairo Paris is north of London (9)

Fractionating the System of Deductive Reasoning

209

Fig. 2 The right lateral/dorsal lateral prefrontal cortex (BA 45, 46) is activated during conﬂict detection. For example, in the argument “all apples are red fruit; all red fruit is poisonous; all apples are poisonous” the correct logical answer is “valid”/“true” but the conclusion is inconsistent with our world knowledge, resulting in a belief-logic conﬂict. (Reproduced from Goel and Dolan 2003.)

ical, belief-based responses. The correct response indicates that subjects detected the conﬂict between their beliefs and the logical inference, inhibited the prepotent response associated with the belief-bias, and engaged the formal reasoning mechanism. The detection of this conﬂict requires engagement of right lateral/dorsal lateral prefrontal cortex (BA 45, 46) (Goel et al. 2000; Goel and Dolan 2003; Prado and Noveck 2007; Fig. 2). This conﬂict detection role of right lateral/dorsal prefrontal cortex is a generalized phenomenon that has been documented in a wide range of paradigms in the cognitive neuroscience literature (Caramazza et al. 1976; Fink et al. 1999; Stavy et al. 2006). One particularly poignant demonstration of this system using lesion data was carried out by Caramazza et al. (1976) using simple two-term reasoning problems such as the following: “Mike is taller than George. Who is taller?” They reported that patients with left-hemisphere lesions were impaired in all forms of the problem but – consistent with imaging data (Goel et al. 2000; Goel and Dolan 2003) – patients with right-hemisphere lesions were only impaired when the form of the question was incongruent with the premise (e.g., who is shorter?).

4 Systems for Dealing with Certain and Uncertain Information Cognitive theories of reasoning do not typically postulate different mechanisms for reasoning with complete and incomplete information. Consider arguments 1–3 in Table 2. All major cognitive theories of reasoning assume that the same cognitive system would deal with each of these inferences; however, patient and neuroimaging data suggest that different neural systems underwrite these inferences.

210

V. Goel

Fig. 3 Lesion overlay maps showing location of lesions in patients tested on reasoning with determinate and indeterminate argument forms. Patients with lesions to right prefrontal cortex (PFC) were selectively impaired in reasoning about indeterminate forms (Mary is taller than Mike; Mary is taller than George; Mike is taller than George). Patients with lesions to left prefrontal cortex showed an overall impairment of reasoning. (Reproduced from Goel et al. 2006.)

Goel et al. (2006) tested neurological patients with focal unilateral frontal lobe lesions (Fig. 3) doing a transitive inference task while systematically manipulating completeness of information regarding the status of the conclusion (i.e., determinate trials and indeterminate trials; see arguments 3, 6, 9 in Table 2). The results demonstrated a double dissociation such that patients with left prefrontal cortex lesions were selectively impaired in trials with complete information (i.e., determinate trials),

Fractionating the System of Deductive Reasoning

211

while patients with right prefrontal cortex lesions were selectively impaired in trials with incomplete information (i.e., indeterminate trials). At the cortical level, this strongly indicates hemispheric asymmetry in systems dealing with reasoning about determinate and indeterminate situations. At the cognitive level, it suggests that different mechanisms may be involved.

5 Implications for Cognitive Theories of Reasoning These types of data are difﬁcult to accommodate within “unitary” theories such as mental model theory and mental logic theory, unless one (1) adopts a position whereby one of the identiﬁed networks constitutes “real” reasoning and all the others are not really reasoning; (2) is prepared to cherry-pick the results (Knauff et al. 2003); or (3) is prepared to question the competence of those who performed existing studies (Monti et al. 2007). An intuitively palatable alternative to these theories – to explain at least part of the results – is provided by dual mechanism accounts. At a very crude level, dual mechanism theories make a distinction between formal, deliberate, rule-based processes and implicit, unschooled, automatic processes. However, dual mechanism theories come in various ﬂavors that differ in the exact nature and properties of these two systems. The theories differentially emphasize explicit and implicit processes (Evans and Over 1996), conscious and preconscious processes (Stanovich and West 2000), formal and heuristic processes (Newell and Simon 1972), and associative and rule-based processes (Goel 1995; Sloman 1996). The relationship among these various proposals has yet to be fully clariﬁed. In previous writings I have suggested that the dissociation between systems dealing with familiar or meaningful information that we have beliefs about and unfamiliar or nonmeaningful information that we have no beliefs about is broadly consistent with the abovementioned set of ideas from dual mechanism theories (Goel 2005). However, as these theories are developing, some discrepancies are emerging between their predictions and neuropsychological data. Furthermore, the development/clariﬁcation of the theories is exposing critical features that in some cases do violence to our commonsense notions of rationality. Here I will discuss the dual mechanism ideas as developed by Evans and Over (Evans 2003; Evans and Over 1996) and further expanded by Stanovich (2004). The two systems postulated by these researchers have come to be widely known as system 1 and system 2. System 1 constitutes a collection of processes whose application is fast, automatic (we have little or no conscious control over them), and mandatory, once triggered by relevant stimuli (i.e., the trigger is causally sufﬁcient for the response). They generally have an evolutionary origin but some automatization through practice is allowed for. The classic example of a system 1 mechanism is the reﬂex arc. Like the reﬂex arc all system 1 processes provide for a causal link between trigger and response, belong to the old part of the brain, and are driven by mechanisms that drive other automatic behaviors across the evolutionary spectrum

212

V. Goel

like foraging and mating (Stanovich 2004). This hypothesis has a less and more extreme version. In the “moderate” account we still have a formal reasoning system that is augmented by these evolutionarily useful modules (Over 2002). In the more extreme version (massive modularity) there is no formal system, just a collection of numerous modules evolved to solve speciﬁc evolutionary problems (Duchaine et al. 2001). It is the former view that I am discussing here. A discussion of the latter is beyond the scope of this review. I want to suggest that this particular division and subsequent characterization of system 1 is inconsistent with both traditional views of rationality and the neuropsychological data. In terms of the neuropsychological data, the neural correlates of system 1 (insofar as belief-bias is part of the system) are underwritten by high-level language/conceptual systems, arguably unique to humans. They are not “reﬂex arc” type mechanisms belonging to the “old part of the brain,” as proposed by the above accounts. Conceptually, there are two problems. First, just because a processes is innate, automatic, and mandatory does not imply that it is part of the “old brain” system. Second, the description of reasoning processes as “innate, automatic, and mandatory” is an unfortunate mischaracterization of the phenomenon. The system 1 mechanisms described above cannot even be candidates for rational systems, as this term is generally understood. In the Western literature, characterizations of rationality have normally included the following features: • Rationality is about means to an end. • In rational behavior, there exists a “gap” between stimulus and response; the antecedent condition is never sufﬁcient for action. • Rationality is ascribed to individual behavior. • A rational choice is not simply a selection, it is a selection for a reason (Bermudez 2002). I take these to be widely accepted, unproblematic constraints on rationality. But the above characterization of system 1 is inconsistent with the existence of a “gap” between stimulus and response. Insofar as system 1 processes are such that the environmental trigger is causally sufﬁcient for the response, all behavior mediated by these processes is removed from the realm of rationality. An excellent example of such behavior is the eye blink reﬂex arc. If I suddenly snap my ﬁngers close to your eyes, you will blink. If I prewarn you of my intention prior to snapping my ﬁngers close to your eyes, you will still blink. Even if I prewarn you, and assure you that I will not touch your eyes (and you trust me and believe me), you will still blink when I snap my ﬁngers. Even if I prewarn you, assure you, and offer you large monetary rewards for not blinking, you will still blink when I snap my ﬁngers close to your eyes. The snapping of my ﬁngers in the proximity of your eyes is causally sufﬁcient for you to blink. Compare this with an example from the belief-bias phenomenon. Suppose a subject evaluates the following (valid) argument as invalid: “Apples are red fruit; all red fruit is poisonous; therefore all apples are poisonous.” There are several important dissimilarities between such a response and a reﬂex like an eye blink. First,

Fractionating the System of Deductive Reasoning

213

subjects can give sensible reasons for their response, for example, “I was focusing on x instead of y.” Second, when the correct answer is pointed out to them they can acknowledge that they have made a mistake and apply the analytic system next time around and generate the normative response. The phenomenology of the belief-bias effect is very different from that of an eye blink. Reasoning fallacies are simply not automatic and mandatory in the same sense as reﬂex arcs. To think otherwise is to invite conceptual confusion. I suspect that these researchers are being misled by their behavioral individuation of systems: “if it is slow and deliberate it belongs to one category; if it is fast and ‘automatic’ it belongs to the other category.” Behavioral categories are superﬁcial and largely uninteresting for scientiﬁc purposes. For example, one can make a category of “all things that move fast” and another category of “all things that move slowly.” In the ﬁrst category one might put things such as cars, planes, comets, and electromagnetic waves, while in the second category one might include bikes, bears, ﬁsh, rocks rolling down hills, etc. Notice the problems here. First, it is unclear how fast or slowly one has to move for membership into the respective category. Do cars move fast enough to be in the “fast” category. They move fast when compared with bikes and bears, but not very fast when compared with electromagnetic waves. Which category do birds belong to? Second, while these categories may be of interest for some purpose, they are of little interest in terms of understanding locomotion because members of the category do not share underlying causal principles of locomotion. Scientiﬁcally interesting categories individuate along causal lines. In light of these concerns, I would say that this particular development of dual mechanism theory is not a candidate to explain the neuropsychological data, and in fact is conceptually confused. An alternative formulation of dual processing theory stems from Simon’s (1983) notion of bounded rationality and the incorporation of this idea into Newell and Simon’s (1972) models of human problem solving. The key idea was the introduction of the notion of the problem space, a computational modeling space shaped by the constraints imposed by the structures of a time and memory bound serial information processing system and the task environment. The built-in strategies for searching this problem space include such content-free universal methods as means–ends analysis, breadth ﬁrst search, depth ﬁrst search, etc. But the universal applicability of these methods comes at the cost of enormous computational resources. But given that the cognitive agent is a time and memory bound serial processor, it would often not be able to respond in real time, if it had to rely on formal, context-independent processes. So the ﬁrst line of defense for such a system is the deployment of task-speciﬁc knowledge to circumvent formal search procedures. Consider the following example. I arrive at the airport in Paris and need to make a telephone call before catching my connecting ﬂight in an hour. I notice that the public telephones require a special calling card. The airport is a multistory building with shops on several ﬂoors. If I know nothing about France I could start on the top ﬂoor at one end of the building, enter a store, and ask for a telephone card. If I ﬁnd one, I can terminate my search and make my phone call. If I do not ﬁnd one, I can proceed to the next store, until I have visited every store on the ﬂoor (or found a telephone card). I could then go down to the next ﬂoor and continue

214

V. Goel

in the same fashion. Following this breadth ﬁrst (British Museum) search strategy, I will systematically visit each store and ﬁnd a telephone card if one of them sells it. The search will terminate when I have found the telephone card, or visited the last store. This may take several hours, and I may miss my connecting ﬂight. If, however, I am knowledgeable about France, I may have a speciﬁc piece of knowledge that may help me circumvent this search: Telephone cards are sold by the tabac shop. If I know this, I merely have to search the directory of shops, ﬁnd the tabac shop, and go directly there, circumventing the search procedure. Notice this knowledge is very powerful, but very situation speciﬁc. It will not help me ﬁnd a pair of socks in Paris or make a telephone call in Delhi. On this account, heuristics are situation-speciﬁc, learned and consciously applied procedures. If present, they are given priority. If they are not available, we fall back upon more general/universal (but computationally intensive) strategies. There are some important differences between the Newell and Simon account and the previous account. The most important have to do with ontological commitments. Newell and Simon are not necessarily talking about two distinct systems/modules with different evolutionary history, but simply two different strategies for processing information. In this sense it is a much weaker account, but it is more consistent with our intuitions about rationality and the neuropsychological data. The dual processing approaches, whatever their particular features, only account for one of several sets of dissociations that we are ﬁnding in the neuropsychological data on reasoning. Therefore, we need to start talking about multiple reasoning systems rather than simply dual reasoning systems. More generally, in terms of viewing reasoning as a dynamically conﬁgured fractionated system I would like to propose the following type of view. We explain neuropsychological data in terms of an interplay between Gazzaniga’s “left hemisphere interpreter” (Gazzaniga 2000) and right prefrontal cortex systems for conﬂict detection and uncertainty maintenance. The function of this interpreter is to make sense of the environment by completing patterns by ﬁlling in the gaps in the available information. I do not think the system is speciﬁc to particular types of patterns. It does not care whether the patterns are logical, causal, social, statistical, etc. It simply abhors uncertainty and will complete any pattern, often prematurely, to the detriment of the organism. The roles of the conﬂict detection and uncertainty maintenance systems are, respectively, to detect conﬂicts in patterns and actively maintain representations of indeterminate/ambiguous situations and bring them to the attention of the interpreter. While there is considerable evidence for the existence of these systems, their time courses of processing and interaction are largely unknown. One speculative account of how processing of arguments might proceed through these systems is presented in Fig. 4. Consider the nine possible types of three-term transitive arguments reproduced in Table 2. Arguments 1–3 that subjects can have no beliefs about are relegated to the formal/universal methods processing system. This system is continually monitored by a conﬂict detector (right dorsolateral prefrontal cortex) and uncertainty maintenance system (right ventrolateral prefrontal cortex). In the case of argument 2 an inconsistency will be detected between premises and the conclusion and an “invalid”

Fractionating the System of Deductive Reasoning

215 Conflict Detector

invalid

R DLPFC

Argument

Formal/Universal System General Pattern Completer

valid

Parietal

L PFC Conflict Detector

Uncertainty Maintenance

R DLPFC

R VLPFC Heuristic System

invalid

invalid valid

L FL/TL

Fig. 4 A speculative account of how the systems identiﬁed by the neuropsychological data may interact in the processing of logical arguments (see the text)

determination made. In the case of arguments 1 and 3 there is no conﬂict. Further pattern completion should validate the consistency of argument 1, resulting in a “valid” response. In the case of argument 3 the uncertainty maintenance system will highlight the uncertainty inherent in the premises and inhibit the left hemisphere interpreter from making unwarranted assumptions, eventually allowing an “invalid” response to be generated. Arguments 4–9, containing propositions that subjects have beliefs about, are initially passed onto the left frontal-temporal system for heuristic processing. However, if a conﬂict is detected between the believability of the conclusion and the logical response (arguments 7–9) the processing is rerouted to, or at least shared with, the formal pattern matcher in the parietal system. In the formal system these arguments are dealt with in a similar manner as arguments 1–3, except for the following important differences: (1) the conﬂict detection system has to continually monitor for belief-logic conﬂict while also monitoring for logical inconsistency; and (2) the fact that subjects have beliefs about the content will also make the task of the uncertainty maintenance system much more difﬁcult. Often it will fail to inhibit the left hemisphere interpreter. Both these situations place greater demands on the cognitive system, resulting in longer reaction times and lower accuracy scores in these types of trials. Arguments 4–6 are passed to the left frontal-temporal heuristic/conceptual system and are largely (though not necessarily exclusively) processed by this system.

216

V. Goel

The believability of the conclusion response is the same as that of the logical response, facilitating the conﬂict detection in argument 5 and pattern completion in argument 4. Even the “invalid” response in argument 6 is facilitated, but for the wrong reason. As above, the unbelievability of the conclusion makes it difﬁcult for the uncertainty maintenance system to maintain uncertainty of the conclusion, but in this case failure facilitates the correct response.

6 Conclusion and Summary Considerable progress has been made over the past decade in our understanding of the neural basis of logical reasoning. In broad terms, these data are telling us that the brain is organized in ways not anticipated by cognitive theories of reasoning. We should be receptive to this possibility. In particular, we need to confront the possibility that there may be no unitary reasoning system in the brain. Rather, the evidence points to a fractionated system that is dynamically conﬁgured in response to certain task and environmental cues. We have reviewed three lines of fractionation of the system of reasoning, discussed their implications for theories of reasoning, and speculated on how they may interact during the processing of various types of logical arguments. There is of course no claim that the systems are exhaustive. The main point is that dissociation data provide important information regarding the causal joints of the cognitive system. Sensitivity to these data will move us beyond the sterility of mental models versus mental logic debate and further the development of cognitive theories of reasoning.

References Acuna BD, Eliassen JC, Donoghue JP, Sanes JN (2002) Frontal and parietal lobe activation during transitive inference in humans. Cereb Cortex 12(12):1312–1321 Bermudez JL (2002) Rationality and psychological explanation without language. In: Bermudez JL, Millar A (eds) Reason and nature: essays in the theory of rationality. Oxford University Press, New York, pp 233–264 Braine MDS (1978) On the relation between the natural logic of reasoning and standard logic. Psychol Rev 85(1):1–21 Canessa N, Gorini A, Cappa SF, Piattelli-Palmarini M, Danna M, Fazio F, Perani D (2005) The effect of social content on deductive reasoning: an fMRI study. Hum Brain Mapp 26(1):30–43 Caramazza A, Gordon J, Zurif EB, DeLuca D (1976) Right-hemispheric damage and verbal problem solving behavior. Brain Lang 3(1):41–46 Duchaine B, Cosmides L, Tooby J (2001) Evolutionary psychology and the brain. Curr Opin Neurobiol 11(2):225–230 Evans JS (2003) In two minds: dual-process accounts of reasoning. Trends Cogn Sci 7(10): 454–459 Evans JSBT, Over DE (1996) Rationality and reasoning. Psychology, New York Evans JSBT, Barston J, Pollard P (1983) On the conﬂict between logic and belief in syllogistic reasoning. Mem Cognit 11:295–306

Fractionating the System of Deductive Reasoning

217

Fangmeier T, Knauff M, Ruff CC, Sloutsky V (2006) FMRI evidence for a three-stage model of deductive reasoning. J Cogn Neurosci 18(3):320–334 Fink GR, Marshall JC, Halligan PW, Frith CD, Driver J, Frackowiak RS, Dolan RJ (1999) The neural consequences of conﬂict between intention and the senses. Brain 122(3):497–512 Fletcher PC, Happe F, Frith U, Baker SC, Dolan RJ, Frackowiak RSJ, Frith CD (1995) Other minds in the brain: a functional imaging study of “theory of mind” in story comprehension. Cognition 57:109–128 Gazzaniga MS (2000) Cerebral specialization and interhemispheric communication: does the corpus callosum enable the human condition? Brain 123(7):1293–1326 Goel V (1995) Sketches of thought. MIT Press, Cambridge Goel V (2005) Cognitive neuroscience of deductive reasoning. In: Holyoak K, Morrison R (eds) Cambridge handbook of thinking and reasoning. Cambridge University Press, Cambridge, pp 475–492 Goel V (2007) Anatomy of deductive reasoning. Trends Cogn Sci 11(10):435–441 Goel V, Dolan RJ (2001) Functional neuroanatomy of three-term relational reasoning. Neuropsychologia 39(9):901–909 Goel V, Dolan RJ (2003) Explaining modulation of reasoning by belief. Cognition 87(1):B11–B22 Goel V, Dolan RJ (2004) Differential involvement of left prefrontal cortex in inductive and deductive reasoning. Cognition 93(3):B109–B121 Goel V, Grafman J, Sadato N, Hallet M (1995) Modelling other minds. Neuroreport 6(13): 1741–1746 Goel V, Gold B, Kapur S, Houle S (1997) The seats of reason: a localization study of deductive and inductive reasoning using PET (O15) blood ﬂow technique. Neuroreport 8(5):1305–1310 Goel V, Gold B, Kapur S, Houle S (1998) Neuroanatomical correlates of human reasoning. J Cogn Neurosci 10(3):293–302 Goel V, Buchel C, Frith C, Dolan RJ (2000) Dissociation of mechanisms underlying syllogistic reasoning. Neuroimage 12(5):504–514 Goel V, Makale M, Grafman J (2004a) The hippocampal system mediates logical reasoning about familiar spatial environments. J Cogn Neurosci 16(4):654–664 Goel V, Shuren J, Sheesley L, Grafman J (2004b) Asymmetrical involvement of frontal lobes in social reasoning. Brain 127(4):783–790 Goel V, Tierney M, Sheesley L, Bartolo A, Vartanian O, Grafman J (2006) Hemispheric specialization in human prefrontal cortex for resolving certain and uncertain inferences. Cereb Cortex 17:2245–2250 Heckers S, Zalesak M, Weiss AP, Ditman T, Titone D (2004) Hippocampal activation during transitive inference in humans. Hippocampus 14(2):153–162 Henle M (1962) On the relation between logic and thinking. Psychol Rev 69(4):366–378 Houde O, Zago L, Mellet E, Moutier S, Pineau A, Mazoyer B, Tzourio-Mazoyer N (2000) Shifting from the perceptual brain to the logical brain: the neural impact of cognitive inhibition training. J Cogn Neurosci 12(5):721–728 Johnson-Laird PN (1983) Mental models: towards a cognitive science of language, inference, and consciousness. Harvard University Press, Cambridge Johnson-Laird PN, Byrne RMJ (1991) Deduction. Erlbaum, Hillsdale Knauff M, Mulack T, Kassubek J, Salih HR, Greenlee MW (2002) Spatial imagery in deductive reasoning: a functional MRI study. Brain Res Cogn Brain Res 13(2):203–212 Knauff M, Fangmeier T, Ruff CC, Johnson-Laird PN (2003) Reasoning, models, and images: behavioral measures and cortical activity. J Cogn Neurosci 15(4):559–573 McCarthy RA, Warrington EK (1990) Cognitive neuropsychology: a clinical introduction. Academic, New York Monti MM, Osherson DN, Martinez MJ, Parsons LM (2007) Functional neuroanatomy of deductive inference: a language-independent distributed network. Neuroimage 37(3):1005–1016 Newell A (1980) Reasoning, problem solving, and decision processes: the problem space as a fundamental category. In: Nickerson RS (ed) Attention and performance VIII. Erlbaum, Hillsdale Newell A, Simon HA (1972) Human problem solving. Prentice Hall, Englewood Cliffs

218

V. Goel

Noveck IA, Goel V, Smith KW (2004) The neural basis of conditional reasoning with arbitrary content. Cortex 40(4–5):613–622 Osherson D, Perani D, Cappa S, Schnur T, Grassi F, Fazio F (1998) Distinct brain loci in deductive versus probabilistic reasoning. Neuropsychologia 36(4):369–376 Over D (2002) The rationality of evolutionary psychology. In: Bermudez JL, Millar A (eds) Reason and nature: essays in the theory of rationality. Oxford University Press, New York, pp 187–207 Parsons LM, Osherson D (2001) New evidence for distinct right and left brain systems for deductive versus probabilistic reasoning. Cereb Cortex 11(10):954–965 Prado J, Noveck IA (2007) Overcoming perceptual features in logical reasoning: a parametric functional magnetic resonance imaging study. J Cogn Neurosci 19(4):642–657 Rips LJ (1994) The psychology of proof: deductive reasoning in human thinking. MIT Press, Cambridge Simon HA (1983) Reason in human affairs. Stanford University Press, Stanford Sloman SA (1996) The empirical case for two systems of reasoning. Psychol Bull 119(1):3–22 Stanovich K (2004) The robot’s rebellion: ﬁnding meaning in the age of Darwin. University of Chicago Press, Chicago Stanovich KE, West RF (2000) Individual differences in reasoning: implications for the rationality debate. Behav Brain Sci 22:645–665 Stavy R, Goel V, Critchley H, Dolan R (2006) Intuitive interference in quantitative reasoning. Brain Res 1073–1074:383–388 Wilkins MC (1928). The effect of changed material on the ability to do formal syllogistic reasoning. Arch Psychol 16(102):5–83

Human Thought and the Lateral Prefrontal Cortex Kalina Christoff

Abstract Human thought is a remarkable evolutionary achievement and one of our species’ deﬁning abilities. It has been closely linked to the prefrontal cortex (PFC) by converging evidence from a wide range of disciplines, from phylogenetic and ontogenetic development, to patient studies and single-cell recordings, to modern neuroimaging. Here I review work on the links between the lateral PFC and two different forms of thought: goal-directed and spontaneous. A special emphasis is placed on the anterior (rostral or frontopolar) lateral PFC, which supports some of the most complex and uniquely human forms of thought, such as reasoning about multiple relations and introspective cognition, and also becomes recruited during the kind of unconstrained thought processes that occurs during rest. I outline an organizational view of the lateral prefrontal cortex which recognizes different lateral prefrontal subregions as functionally distinct and arranged in a rostrocaudal gradient of complexity in processing and representational abstraction, with higher abstraction in thought corresponding to more anterior lateral PFC regions. Furthermore, it appears that the functions of lateral PFC extend beyond goal-directed thinking, to include more spontaneous, free-ﬂowing mental cognition. Unlike goal-directed thought, however, spontaneously occurring thought appears to draw most heavily upon resources outside the PFC, including lateral and medial temporal lobe regions. The nature of interactions between lateral PFC and the temporal lobe remains an important topic for future research that promises to help elucidate some of the most intriguing aspects of human thought, including its spontaneous generation.

1 Introduction The lateral prefrontal cortex is known to be essential for human thought and complex cognitive processing. Numerous patient studies have shown that the prefrontal cortex is essential for the complex cognitive processes underlying reasoning and K. Christoff Department of Psychology, University of British Columbia, Cognitive Neuroscience of Thought Lab, 2136 West Mall, Vancouver, BC, Canada, V6T 1Z4 [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, 219 c Springer-Verlag Berlin Heidelberg 2009

220

K. Christoff

problem solving (Luria 1966; Milner 1963, 1964). Neuroimaging studies have supported this view by demonstrating prefrontal cortex activation across many reasoning tasks (Berman et al. 1995; Baker et al. 1996; Nagahama et al. 1996; Owen et al. 1996; Goel et al. 1997, 1998, 2000; Prabhakaran et al. 1997, 2000; Rao et al. 1997; Goldberg et al. 1998; Osherson et al. 1998; Ragland et al. 1998; Dagher et al. 1999; Goel and Dolan 2000; Wharton et al. 2000). Although the link between human thought and lateral prefrontal cortex function has been clearly established, this link has been formulated mostly at the general level, and a breakdown of the lateral prefrontal cortex into separate functionally distinct regions has proved to be somewhat more difﬁcult. Nonetheless, one approach to a more precise functional deﬁnition has been to regard the prefrontal cortex as a heterogeneous region, comprising specialized subregions with different functional roles. Indeed, cytoarchitectonic studies at the beginning of the twentieth century (Brodmann 1908; Campbell 1905; Elliott Smith 1907; Vogt 1906), as well as recent neuroanatomic (Pandya and Barnes 1987), neurophysiological (di Pellegrino and Wise 1991; Rosenkilde 1979), and neurocircuitry (Alexander et al. 1986; Barbas and Pandya 1991; Pandya and Barnes 1987) studies have suggested that the prefrontal cortex should be subdivided into several structurally and functionally different subregions. A number of regional speciﬁcations have been proposed, from Broca’s original localization of the inferior prefrontal cortex as an area essential for the production of speech, to recently proposed functional specializations of speciﬁc prefrontal subregions such as the ventromedial (Damasio et al. 1996), orbitofrontal (Rolls 1996), and dorsolateral (Goldman-Rakic 1987; Petrides 1991). Some analyses have focused on contrasting one subregion relative to another (D’Esposito et al. 1998; Owen et al. 1996; Petrides 1994), in order to clarify the type of processing for which each subregion is specialized. Some of the most recent ﬁndings from functional neuroimaging have focused precisely on clarifying the functional organization of the lateral prefrontal cortex and linking its different subregions to speciﬁc thought processes. Several relevant studies and analyses are described in this chapter, which converge to indicate a rostrolateral (or anterior-to-posterior) organization within the lateral prefrontal cortex, with complex, abstract thought processes represented in the anteriormost regions, and concrete, externally oriented thought processes distributed more posteriorly in the prefrontal cortex. This organization applies for multiple domains of higher cognitive functions, including reasoning, working memory, and episodic memory retrieval, and suggest a speciﬁc organization of thought in the human brain. Finally, implications for human thought and cognition are discussed, with emphasis on those processes and prefrontal cortex subregions that bear uniquely human capacities such as higher-order relational reasoning and introspective thought.

2 Higher-Order Relational Reasoning Higher-order relational reasoning is a speciﬁc kind of mental computation that develops slowly in humans and that developed so late in primate evolution as to be unique to humans. Such higher-order relational reasoning involves the processes

Human Thought and the Lateral Prefrontal Cortex

221

of relational integration, or considering multiple relations simultaneously. Children under 5 years of age can solve 0- and 1-relational problems, but fail to solve 2-relational problems that demand integration of multiple relations, even when matched for working memory load to the 1-relational problems (Halford 1984). Nonhuman primates can solve 1-relational problems, but cannot solve problems that require processing multiple relations simultaneously (reviewed in Tomosello and Call 1997). Robin and Holyoak 1995 proposed that failures in relational integration may be attributable to the relatively slow frontal lobe maturation in humans (as indexed by myelination and other markers of cortical development) and to the great expansion of frontal cortex in human evolution. Such ontogenetic and phylogenetic developmental evidence supports the hypothesis that the prefrontal cortex has a selective role in the process of relational integration. A number of neuroimaging studies (Christoff et al. 2001; Kroger et al. 2002; Bunge et al. 2004) have investigated the speciﬁc contribution of prefrontal cortex to higher-order relational reasoning, and the process of relational integration in particular. Here we focus on one of these studies (Christoff et al. 2001), which illustrates and summarizes the main ﬁndings. To investigate the process of higherorder relational reasoning, we used functional magnetic resonance imaging (fMRI) to examine brain activation in healthy volunteers during 0-relational, 1-relational, and 2-relational problems adapted from the Raven’s Progressive Matrices test. The relational complexity of each problem was deﬁned as the number of relations that had to be considered simultaneously in order to solve the problem. Verbal protocol and eye-movement analyses have shown that Raven’s Progressive Matrices problems are solved using a sequential, reiterative strategy for inducing and encoding the rules or relationships of change within each problem Carpenter et al. 1990. Thus, the method we employed for deﬁning the number of relations in each problem had been shown to be valid both psychologically Carpenter et al. 1990 and neuropsychologically Waltz et al. 1999. Our goal was to test whether prefrontal cortex has a selective role in the process of relational integration in healthy young volunteers, and to determine whether speciﬁc prefrontal cortex subregions mediate this role. We hypothesized that increases in prefrontal cortex activation would be speciﬁc to the 2-relational problems and would be observed independently of increases in duration of processing. We used an event-related procedure that allowed randomized presentation of different problem types, so that participants were unable to anticipate what type of problem would be presented next. In addition, the duration spent working on each problem was recorded and used in modeling each event-related response. In this way, we were able to identify regions activated by novel processes separately from regions activated by longer engagement of processes common to solving all problem types. Problems had the general form of the Raven’s Progressive Matrices Raven 1938. Each problem consisted of a 3 × 3 matrix of ﬁgures, with the bottom-right ﬁgure missing (Fig. 1). After considering the relationship among the given matrix ﬁgures, participants had to infer the missing ﬁgure and select it among the four alternatives presented on the right side of the matrix.

222

K. Christoff

Fig. 1 Examples of problem types used in the experiment

Three different types of problems were created: 0-relational, 1-relational and 2-relational. The 0-relational problems (Fig. 1a) involved no relationship of change and required no relational processing in order to be solved. The 1-relational problems (Fig. 1b) involved a change in either the horizontal or the vertical dimension and, therefore, required processing of a single relation. Finally, the 2-relational problems (Fig. 1c) involved two relations of change, in both the horizontal and the vertical direction. Inferring the correct answer required considering the converging change along both dimensions. Thus, the 2-relational problems required that two relations be integrated, or considered simultaneously. Behavioral results showed that participants were highly accurate, with 93.8% overall accuracy. Increasing relational complexity resulted in reduced accuracy and slower latency of response. The activation maxima from the primary analysis using the response-timeconvolved hemodynamic response function are illustrated in Fig. 2. No voxels surviving the p < 0.001(Z > 3.09) threshold at the voxel level were revealed in the

Human Thought and the Lateral Prefrontal Cortex

223

RT-convolved HRF

1rel > 0rel

z = −4 mm

z = 12 mm

z = 28 mm

z = 44 mm

T value 6 5 4 3 2 1 0

2rel > 1rel

Fig. 2 Results from voxelwise analysis. RT response time; HRF hemodynamic response function

1-relational and 0-relational comparison. Two small clusters in the premotor cortex approached signiﬁcance when the voxel threshold was relaxed to p < 0.01, but there were no prefrontal cortex differences even at this level. The 2-relational and 1-relational comparison yielded activation in the prefrontal cortex bilaterally. There were two left-lateralized cortical regions of activation. The ﬁrst was a cluster in the left posterior and premotor prefrontal cortex, extending over the inferior prefrontal and precentral gyri, including Brodmann areas (BA) 44 and 6. The second cluster was located in the left anterior prefrontal cortex, extending over the rostrolateral prefrontal cortex, including the lateral portion of BA10. The activation maxima in both of these left prefrontal cortex regions survived correction for multiple comparison (correction was performed using a mask of all prefrontal cortex voxels showing a signiﬁcant main effect of reasoning). The right-lateralized regions of activation for this comparison included the dorsolateral prefrontal cortex, including BA9 and BA46, with maxima of activation in the middle frontal gyrus. At a lower threshold level (p < 0.01), the two clusters merged into a single large right prefrontal cortex cluster of activation, extending in anterior direction to include the rostrolateral prefrontal cortex (lateral BA10). In addition to prefrontal cortex activations, the 2-relational and 1-relational comparison yielded robust activation in the head of the caudate nucleus bilaterally. No other suprathreshold activations were observed for this comparison. The results revealed a consistent pattern: prefrontal cortex activation was speciﬁc to the comparisons between 2- and 1-relational problems, and was not observed in the comparisons between 1- and 0-relational problems. In left rostrolateral prefrontal cortex , this pattern continued to hold even after equating for response time and accuracy among problem types. These results converge with the patient ﬁndings of Waltz et al. 1999 and support the hypothesis that prefrontal cortex is selectively involved during the process of relational integration. The results are also

224

K. Christoff

consistent with the view that prefrontal cortex is preferentially engaged during relationally complex processes Robin and Holyoak 1995 and conﬁrm that, as proposed by Halford et al. (1998), the relational complexity of a task can be successfully used to predict prefrontal cortex recruitment. Rostrolateral prefrontal cortex activations have been observed during highly complex tasks across a wide range of domains. On the basis of a review of studies reporting activation in this region (Christoff and Gabrieli 2000), we have proposed that it may be selectively involved in active processing, such as manipulation or evaluation, performed upon self-generated information. The present ﬁndings bear interesting implications relating to this hypothesis and are consistent with it: The relational information associated with a problem from the Raven’s Progressive Matrices test is not given in the problem, but has to be inferred, or self-generated, on the basis of given information (the individual object features such as shape, texture, and size). The observed link between rostrolateral prefrontal cortex activation and the process of relational integration, therefore, could be due to the associated process of manipulating the self-generated information about the change among objects.

3 Rostrocaudal Organization Within the Lateral Prefrontal Cortex Most considerations of regional specialization within the prefrontal cortex have concentrated on the posterior prefrontal cortex, including the dorsolateral, ventral, medial, and orbitofrontal regions. There has been far less consideration of the anteriormost part of the prefrontal cortex, usually referred to as the frontopolar (or rostrolateral) prefrontal cortex. With remarkable frequency, however, functional neuroimaging studies have detected frontopolar cortex activation when people perform complex cognitive tasks. Activation in this region has been reported for many reasoning tasks, such as the Tower of London task (Baker et al. 1996), the Wisconsin Card Sorting Test (Berman et al. 1995; Goldberg et al. 1998; Nagahama et al. 1996), inductive and probabilistic reasoning tasks (Goel et al. 1997; Osherson et al. 1998), and the Raven’s Progressive Matrices test (Prabhakaran et al. 1997). Frontopolar activations are also common when people perform memory tasks involving episodic retrieval (for reviews, see Cabeza et al. 1997; Nolde et al. 1998; Nyberg et al. 1996). Many of these studies report both dorsolateral and frontopolar activations, but few offer suggestions as to what different psychological operations are mediated by these anatomically distinct frontal regions (exceptions are Baker et al. 1996; Koechlin et al. 1999). Here, we review the results from neuroimaging studies in the domains of reasoning and episodic memory retrieval, and examine the evidence for a functional distinction within the prefrontal cortex in a rostrocaudal (anterior–posterior) direction. We present analyses of the distribution of stereotaxic coordinates of activation foci reported by neuroimaging studies in the domains of reasoning and episodic

Human Thought and the Lateral Prefrontal Cortex

225

memory retrieval. These analyses were conducted in order to gain evidence as to what psychological processes may be mediated by the frontopolar cortex and, where possible, to reveal differences between task conditions associated with frontopolar and dorsolateral activations.

3.1 Reasoning: Distribution Analysis The studies used problem solving tasks such as the Tower of London task, the Wisconsin Card Sorting Test, inductive (or probabilistic) reasoning, and the Raven’s Progressive Matrices test. The stereotaxic coordinates of the local maxima of activation reported by these studies are displayed on the rendering of a standardized brain in Fig. 3. Frontopolar activations were observed as consistently as dorsolateral activations for reasoning tasks. Frontopolar activation was reported in eight of ten studies and dorsolateral activation was reported in seven of the ten studies. The studies that observed dorsolateral activation during reasoning also observed frontopolar activation, with two exceptions in which only frontopolar activation was observed (Goel et al. 1997; Osherson et al. 1998). These two studies, however, used rather demanding

SUPERIOR

FRONTAL

Tower of London

Ravens' Matrices

Card Sorting

Inductive Reasoning

RIGHT

Fig. 3 Activation foci reported during reasoning

LEFT

226

K. Christoff

baseline conditions that required complex sentence comprehension and semantic decisions. These demanding baselines may have involved dorsolateral activation to the same extent as the reasoning tasks.

3.2 Mental Processes Related to Frontopolar Activation: Evidence from Reasoning Studies The review of reasoning studies above shows that both dorsolateral and frontopolar areas are frequently activated during reasoning, but it does not distinguish between these two areas. However, a number of researchers have suggested that particular reasoning processes may be of greater signiﬁcance in recruiting frontopolar activation than others. Baker et al. (1996) proposed that the frontopolar cortex is involved in sequence selection and evaluation. Baker et al. used the Tower of London task and compared one condition consisting of “easy” problems that required only two or three moves for an optimal solution and another condition consisting of “difﬁcult” problems that required four or ﬁve minimum moves. Even though both types of problems involved planning and evaluation of a sequence of moves, the “difﬁcult” problems required considering a longer sequence of moves than the “easy” problems. The difﬁcult problems were found to produce signiﬁcantly greater right frontopolar activation than the easy problems, although bilateral increases in dorsolateral activation were also observed. On the basis of evidence from previous studies, Baker et al. attributed the increases in dorsolateral activation to the increased working memory requirements and proposed that the frontopolar activation was reﬂecting the increased need for sequence selection and evaluation. There is also evidence to suggest the frontopolar cortex may be involved in feedback evaluation. Feedback evaluation is a crucial component of the Wisconsin Card Sorting Task and four of the ﬁve studies using a card sorting task reported activation in this region. Tasks other than the Wisconsin Card Sorting Test have also provided support for the idea that the frontopolar cortex is involved in feedback evaluation. Elliott et al. (1997) presented participants with a guessing task based on the formal structure of the Tower of London task and varied whether or not participants received feedback after each response. There was frontopolar activation for the feedback condition relative to the no-feedback condition. No dorsolateral activation was found for this comparison. Another mental process associated with frontopolar activation is hypothesis generation and evaluation – an important aspect of both card sorting and inductive reasoning tasks. During card sorting, participants have to generate hypotheses as to what is the correct sorting rule, and evaluate these hypotheses in light of the feedback. Inductive reasoning, on the other hand, is in itself considered a form of hypothesis generation and testing (Goel et al. 1997). It has been studied in both the verbal and the visuospatial domain. Neuroimaging studies of verbal inductive reasoning have

Human Thought and the Lateral Prefrontal Cortex

227

operationalized induction as the process of evaluating an argument’s plausibility, given certain premises. In this case, a hypothesis about the argument’s plausibility would have to be generated and evaluated in light of the information contained in the premises. Inductive reasoning has also been studied through nonverbal tasks, tasks such as the Raven’s Progressive Matrices, where the same processes of hypothesis generation and evaluation need to be carried out in the visuospatial instead of the verbal domain. Frontopolar activation was observed during performance of the Raven’s Progressive Matrices task across two types of problems (Prabhakaran et al. 1997). This further supports the possibility that this cortical region may be involved when hypotheses are being generated or evaluated. In summary, there are several mental operations that seem to be associated with frontopolar activation, namely, sequence selection and evaluation, feedback evaluation, and hypothesis generation and evaluation. These mental operations seem to share a common feature that can be described as evaluation of a self-generated response or plan for action. This can be a self-generated sequence of moves or a plan for it in the Tower of London task; a self-generated response according to a sorting category in the Wisconsin Card Sorting Test, or a self-generated hypothesis as to the plausibility of an argument or the item which should follow next in the sequence in the case of inductive reasoning and the Raven’s Progressive Matrices test. It is possible therefore that the frontopolar cortex is speciﬁcally involved in self-generated evaluation, a process that is critical when nonroutine cognitive strategies have to be generated and selected in the context of novel tasks or activities. If self-referential evaluation and introspective thought in general are processes that characterize the role of the frontopolar cortex, this cortical region should play an important role not only in reasoning, but also in other cognitive domains requiring introspectively based decisions.

3.3 Frontopolar Activations in Tasks Other Than Reasoning Apart from reasoning, activation of the frontopolar cortex has been observed in a number of functional imaging studies employing various cognitive paradigms, such as self-ordered tasks (Owen et al. 1996; Petrides et al. 1993), semantic monitoring tasks (MacLeod et al. 1998), self-relevant tasks (Stone et al. 1998), cognitive branching tasks (Koechlin et al. 1999), and working memory tasks (Cohen et al. 1994; Grasby et al. 1993; Jonides et al. 1997; Rypma et al. 1999; Smith et al. 1996). As a more detailed examination reveals, it is likely that these tasks, under the conditions in which they were presented, involved some form of self-referential evaluation. Self-ordered tasks usually involve a sequence of responses, where each response can be executed only after the previously executed responses are taken into consideration. Petrides et al. (1993) used a task that required participants to say aloud in random order the numbers one through ten without repeating themselves; this produced activation in left frontopolar cortex (x, y, z = −35, 42, 22; BA10/46).

228

K. Christoff

Owen et al. (1996) used another self-ordered task that produced activation in the right frontopolar cortex (x, y, z = 34, 51, 6; BA10). In this case participants touched eight or 12 circles in a random sequence until one of them changed color. In each trial, participants had to avoid the circles which had changed their color in previous trials. Both tasks seem to require that participants evaluate each response in light of the previous responses that they themselves have previously executed. Semantic monitoring tasks, on the other hand, should not necessarily require selfreferential evaluation. However, the speciﬁc task used by MacLeod et al. (1998) required not only classifying individual words according to a prespeciﬁed semantic category, but also monitoring the frequency of words that belonged to this category and making an estimate of this frequency for each block. When this conjoint semantic monitoring and target frequency estimation condition was compared with a passive word viewing control task, right frontopolar cortex was found to be activated (x, y, z = 25, 61, 6; BA10). In three of the ﬁve conditions employed by MacLeod et al., participants had to give a gross estimate (i.e., a percentage) of the target frequency, which makes the task bear a striking resemblance to the previously described inductive reasoning tasks where a hypothesis about the probability of a statement has to be formulated and evaluated. Indeed, relative frequency is considered to be the principal source of information about probability (Gigerenzer and Murray 1987). Therefore, it is possible that the generation and evaluation of a target frequency estimate, rather than semantic classiﬁcation, was the mental operation responsible for frontopolar cortex activation. Yet another process, that of cognitive branching, was interpreted as responsible for the bilateral frontopolar activation (x, y, z = 36, 66, 21; BA10; and x, y, z = −36, 57, 9; BA10) reported by Koechlin et al. (1999). Cognitive branching refers to the process of allocating attentional resources when attention has to be alternated between two concurrently ongoing activities. The cognitive branching condition employed by Koechlin et al. consisted of presenting participants with alternating blocks of uppercase and lowercase letters. As uppercase and lowercase blocks alternated, participants had to alternate between two different sets of goals, making a judgment for each letter presented. During a lowercase block, the judgment for the ﬁrst letter in the block was whether or not it was “t,” and for subsequent letters in the block whether the current letter and the one presented prior to it were in immediate succession in the word “tablet.” On the other hand, during an uppercase block, the judgment for the ﬁrst letter was whether or not it was the same as the last letter in the previously presented uppercase block. For subsequently presented letters in uppercase blocks, the judgment was identical to the one for lowercase blocks – whether or not the two letters were in immediate succession in the word “tablet.” Thus, in order to perform the task, participants had to keep in mind the ﬁrst set of goals while acting on the second, after which they had to shift to keeping in mind the second set, while acting on the ﬁrst, and so forth. It is difﬁcult to precisely analyze the component processes in such a demanding situation, but it is possible that in a sequence where two sets of goals alternate in a regular fashion such as the one used in this study, keeping track of which set of goals is currently to be followed involves

Human Thought and the Lateral Prefrontal Cortex

229

considering one’s own responses to the block of items immediately preceding the current block. Therefore, it is plausible that the task employed by Koechlin et al. involved internally oriented processes. Another domain in which tasks have sometimes elicited frontopolar activation is working memory. A task that has been used widely in the working memory literature is the n-back task, during which participants are presented with a series of items, each appearing one at a time, followed by the next item in the series. The task is to press a button when the item that is being presented at the moment is the same as the one presented a certain number (n) of items earlier. For instance, in a 1-back task, a button has to be pressed if the item is the same as the one presented immediately prior to it, whereas in a 2-back task, it has to match the item presented before the last previously presented item, and so on. Using the n-back task, Jonides et al. (1997) found right frontopolar activation for a 2-back condition (x, y, z = −42, 50, 22; BA10) and bilateral frontopolar activation in a 3-back condition (x, y, z = −39, 44, 18; BA10; and x, y, z = 37, 48, 18; BA10). Using the same n-back task in a 2-back condition, Cohen et al. (1994) observed bilateral frontopolar activation for verbal items such as letters (x, y, z = −29, 38, 20; BA10/46; and x, y, z = 31, 42, 22; BA10/46), as well as spatial items such as locations (x, y, z = −26, 49, 11; BA10; and x, y, z = 34, 53, 10; BA10). Similarly, using a 3-back condition, Smith et al. (1996) found left-lateralized frontopolar activation for both letter items (x, y, z = −37, 55, 2; BA10) and location items (x, y, z = −33, 44, 20; BA10). Using a different working memory task, Grasby et al. (1993) found bilateral frontopolar activation in a supraspan compared with a subspan condition (x, y, z = −34, 46, 0; BA10; and x, y, z = 24, 52, 0; BA10). The supraspan condition required the free recall of a list of 15 words immediately after they had been presented, whereas the subspan condition involved free recall of lists of ﬁve words. Yet another working memory task, based on a simultaneous item-recognition task developed by Sternberg (1966), was used in a study by Rypma et al. (1999) and bilateral frontopolar activation was reported for a six-letter condition in comparison with a one-letter condition (x, y, z = −27, 53, 1; BA10; and x, y, z = 25, 53, 1; BA10). In general, working memory conditions that activate frontopolar cortex involve maintenance of working memory load approaching or exceeding the average shortterm memory span: Tasks such as 2- or 3-back, or keeping in mind 15 words or six unrelated letters, are usually considered to be around or above span limit. One possible explanation is that as the number of maintained items approaches or exceeds this limit, there appears a need to strategically organize the process of maintenance. The observed frontopolar activations could be related to this additional process of maintenance organization, which may involve processing of internally generated information (such as particular groupings of the items into chunks). Another possible explanation is that prefrontal cortex activation, including activation of frontopolar cortex, is observed whenever there is increase in task difﬁculty. The issue of task difﬁculty and its relation to frontopolar activation will be discussed in greater detail later in this chapter.

230

K. Christoff

3.4 Rostrocaudal Prefrontal Cortex Differences: A Hierarchical Distinction As mentioned earlier, there is evidence that the dorsolateral prefrontal cortex is speciﬁcally involved during the processes of manipulation and monitoring of information within working memory. It represents the second stage of processing in the two-stage model of working memory proposed by Petrides (1994), Petrides et al. (1995) and Owen et al. (1996) and conﬁrmed by more recent neuroimaging ﬁndings (D’Esposito et al. 1998). However, many tasks involving manipulation and monitoring have been shown to produce activation not only in the dorsolateral, but also in the frontopolar cortex. This suggests that the frontopolar cortex may also be involved during these processes. It is, however, possible that as suggested previously these two areas of the cortex are involved in different types of manipulation and monitoring. The type of monitoring and manipulation within working memory associated with dorsolateral prefrontal cortex activation has, in the majority of working memory studies, involved monitoring and manipulation of externally generated information, such as letters or locations. On the other hand, the evidence that the frontopolar cortex is speciﬁcally involved in the process of evaluation of self-generated responses or plans for action suggests that the frontopolar cortex may be needed in addition to the dorsolateral cortex, when the task requires monitoring and manipulation of information that has been internally generated. In this sense, the frontopolar region can be viewed as subserving an additional, third level of executive processing within the human prefrontal cortex (Fig. 4). Therefore, we hypothesize a rostrocaudal distinction within the prefrontal cortex, involving the dorsolateral and frontopolar prefrontal regions, and distinguishing between active processing performed upon information that has been externally generated and processing performed upon information that has been self-generated (Fig. 4, stages II and III). This distinction is hierarchical in both anatomical and functional terms: The dorsolateral prefrontal cortex may be sufﬁcient for the processing of externally generated information, but both frontopolar cortex and dorsolateral cortex are needed when self-generated information is being processed.

3.5 Episodic Retrieval: Distribution Analysis In an attempt to examine the generality of rostrocaudal prefrontal cortex differences, we next turn to functional neuroimaging results reported in episodic memory retrieval. Episodic retrieval studies have revealed consistent activations in both frontopolar and dorsolateral prefrontal cortex, but they have failed to discover what distinguishes activations in these two prefrontal regions (Wagner et al. 1998; Henson et al. 1999). Nonetheless, the frontopolar and the dorsolateral cortex may subserve distinct functions during episodic retrieval and there is some evidence as to what

Human Thought and the Lateral Prefrontal Cortex

231

sfg 9

10

9/46 46

mfg

45 if 47/12 g

Frontopolar PFC (BA 10)

Dorsolateral PFC (BA 9, 46, 9/46) Ventrolateral PFC (BA 45, 47/12)

III. Monitoring and manipulation of internally generated information II. Monitoring and manipulation of externally generated information I. Maintenance of one or few items

Fig. 4 Hierarchical organization of prefrontal function: a three-stage model. BA Brodmann area; PFC prefrontal cortex

these functions may be: Activation in the dorsolateral prefrontal cortex has been found to be speciﬁcally associated with monitoring processes during episodic retrieval (Henson et al. 1999; Shallice et al. 1994); the role of the frontopolar cortex has been somewhat more controversial but, importantly, one of the leading hypotheses has been that it is associated with a postretrieval evaluation of the self-generated products of the retrieval process (Johnson et al. 1996; Shallice et al. 1994). The studies in the following review were classiﬁed according to the type of episodic retrieval procedure they used. Episodic retrieval tests can be divided into two groups according to the degree to which the memory judgment requires evaluation of self-generated information. We expected that, on the average, tests in which participants have to evaluate self-generated information would be more likely to have resulted in frontopolar activation than when no such introspectively directed evaluation was required. Tests classiﬁed as requiring minimal evaluation of self-generated information included simple episodic recognition procedures, either forced-choice or yes–no recognition. Forced-choice recognition involves indicating which one of two simultaneously presented items (one old and one new) has been previously presented during the acquisition phase; yes–no recognition involves sequential categorization

232

K. Christoff

of individual test items as either old or new. In both cases, successful performance in recognition tests is based on evaluating externally generated information – whether an item has been presented previously or not. It should be noted, however, that even though evaluating self-generated information was not required by the tasks in this group, the possibility that it can occasionally occur cannot be excluded. Indeed, this possibility is especially marked in the sequential yes–no recognition procedure, which may involve setting a criterion for evaluating memory characteristics evoked by test items, which would in its own turn require retrieval and evaluation of previous self-generated responses (Nolde et al. 1998). However, regardless of the possibility that it can occasionally occur, the evaluation of self-generated information is minimally required by these test procedures. On the other hand, tests classiﬁed as requiring more evaluation of self-generated information included cued recall, free recall, and complex recognition procedures such as counting of oddball items within a block. None of these tests can be performed by evaluating externally generated information. In all cases, people must evaluate the information they have generated (or retrieved) themselves in response to some sort of cue or instruction. Cued recall typically involves not only the generation, but also the evaluation of self-generated answers (Nolde et al. 1998). During cued recall, participants are presented with cues to help them remember speciﬁc items learned during acquisition. According to the type of cued recall used, these cues can be word stems, word fragments, or one of a pair of associated words (word associates). Participants need to recall the acquisition word with which the stem or fragment can be completed, or which was associated with the word presented as a cue. Typically, each of the cues can be associated with not only studied but also nonstudied words, and therefore participants may generate more than one solution before attributing one to the acquisition task (Nolde et al. 1998). Thus, cued recall appears to require evaluation of information generated by the participants themselves. Similarly, free recall tasks are generally thought to require self-initiated cueing and selection among possible candidate responses, and would also be likely to involve evaluation of self-generated information. Counting the number of oddball items within a block, on the other hand, even though formally classiﬁed as a recognition procedure, presents participants with the dual task of having to categorize each item and keep track of the number of items in the oddball category. This type of procedure most likely requires some complex cognitive processes similar to those previously discussed for cognitive branching and probability estimation, and is therefore likely to involve evaluation of self-generated information. The maxima reported by studies included in each of the two groups are plotted onto a rendering of a standardized brain in Fig. 5. While only half of the recognition studies (ﬁve out of ten) reported frontopolar activation, nearly all of the other studies (13 out of 15) reported signiﬁcant activation foci in this region of the cortex. Seven of the ten recognition studies and 11 of the other 15 studies revealed dorsolateral activation. The results suggest a hierarchical distinction between the dorsolateral and the frontopolar cortex: The dorsolateral cortex appears to be involved in both types of evaluative processes, whereas the frontopolar cortex is activated much more consistently by tasks requiring evaluation of self-generated information.

Human Thought and the Lateral Prefrontal Cortex

233

Fig. 5 Activation foci reported during episodic memory retrieval

The review of reasoning, episodic retrieval, and other studies presented here provides evidence that the frontopolar cortex may be speciﬁcally involved in complex cognitive activities that require evaluation of self-generated information. Frontopolar cortex is nearly ubiquitously activated by complex reasoning tasks, and the few exceptions may be interpreted in terms of speciﬁc experimental details. All complex reasoning tasks demand the evaluation of self-generated information. The review of episodic retrieval studies suggested that retrieval tests which require only evaluation of externally generated information are not likely to activate the frontopolar cortex, whereas tasks that pose an additional requirement for evaluation of self-generated information almost invariably produce activation in this region. This provides further evidence for a hierarchical distinction between the frontopolar region and the more posteriorly located dorsolateral prefrontal cortex. Both types of evaluation are likely to produce dorsolateral prefrontal cortex activation. Therefore, the proposed rostrocaudal prefrontal cortex distinction should not be taken to imply a double dissociation. Instead, it should be taken to suggest a hierarchical organization in which the frontopolar cortex is necessary, although not sufﬁcient, in order for the most complex stages of processing to be carried out (Fig. 4). Thus, not only the frontopolar cortex but also the dorsolateral cortex (and probably other posterior regions) would be needed for the evaluation of self-generated information. Such hierarchical models of information processing are common in sensory and motor systems. Here, we propose such a hierarchical model for prefrontal mediation of thought.

234

K. Christoff

3.6 Lesion Studies and Deﬁcits in Self-Referential Evaluation Patients with frontal-lobe lesions are often impaired in the tasks discussed previously involving self-referential evaluation. Frontal-lobe lesion patients exhibit deﬁcits in all of the previously reviewed reasoning tasks, including the Tower of London task (Norman and Shallice 1986; Owen et al. 1990; Shallice 1982), the Wisconsin Card Sorting Test (Milner 1963, 1964; Robinson et al. 1980), and inductive inference tasks such as judging frequencies (Milner et al. 1985; Smith and Milner 1988) and cognitive estimation (Shallice and Evans 1978; Vilkki and Holst 1991). Importantly, these studies have included frontal lesions that in many cases involved the frontopolar area. Other ﬁndings suggesting impaired self-evaluation after frontal-lobe damage abound, including the lack of behavioral restraint frequently observed after frontal-lobe damage (Miller 1985), increased impulsivity (Miller 1992), utilization behavior (Shallice et al. 1989), and inability to monitor the effectiveness of self-generated plans (Luria 1973). All of these different impairments can be attributed to a general deﬁcit in evaluating self-generated information. In addition to reasoning, frontal-lobe abnormality seems to be associated with a speciﬁc pattern of memory dysfunction. Importantly, there is impairment in memory performance on free recall tests (della Rocchetta 1986; Janowsky et al. 1989; Jetter et al. 1986), although when the same material is tested in recognition procedures performance is relatively preserved (Stuss et al. 1994; for a review, see Wheeler et al. 1995). Free recall was classiﬁed in the previous section as likely to require evaluation of self-generated retrieval strategies and retrieval outcomes. Consistent with this, the deﬁcit on free recall in patients with frontal-lobe lesions is generally interpreted as impairment in the subjective organization that aids recall (Gershberg and Shimamura 1995). One instantiation of this free recall impairment is the frequently observed confabulation and faulty retrieval of remote memories in patients with frontal-lobe damage (Baddeley and Wilson 1986; Moscovitch 1989; Stuss et al. 1978; Stuss and Benson 1986). Confabulation is also interpreted as a deﬁcit in retrieval strategy and the evaluation of the search outcome (Baddeley and Wilson 1986, 1988; Burgess and Shallice 1996; Moscovitch 1989; Squire and Cohen 1982). In addition, the performance of patients with lesions in the prefrontal region is also impaired in the previously discussed self-ordered working memory tasks (Milner et al. 1985; Petrides and Milner 1982), for which the ability to generate and evaluate self-generated strategies is critical. In addition, patients with frontal-lobe lesions exhibit a speciﬁc and limited deﬁcit in some particular episodic memory tests such as source memory (Janowsky et al. 1989) and recency judgments (Milner et al. 1991) where, similarly to free recall, self-generated memories must be evaluated, manipulated, or transformed. Patients with frontal-lobe lesions also exhibit a propensity to make false alarms or, in other words, to endorse foil or baseline items as having been seen before (Schacter et al. 1996). This may be related to the previously mentioned processes of criterion setting and could be interpreted as a failure to evaluate and adjust the self-generated criterion used to distinguish between studied items and other items that bear similarity to the studied items.

Human Thought and the Lateral Prefrontal Cortex

235

Unfortunately, in the majority of clinical cases there is a lack of detailed knowledge about the lesions, and even when their extent is known, it rarely follows precisely functional or architectonic borders. This, in addition to the need to average across patients with sometimes very different lesion locations, can greatly reduce the ability to make subregional inferences. However, at least one lesion study has provided evidence consistent with the rostrocaudal prefrontal distinction proposed here. Vilkki and Holst (1991) used a digit symbol test to assess the ability to achieve a self-selected goal deﬁned as the number of symbols the patients estimated would be achieved in 1 min or less. Patients with anterior prefrontal lesions were found to be more impaired than patients with posterior prefrontal lesions in estimating achievable goals – a process relying extensively on self-referential evaluation.

3.7 Relation Between Neuroimaging and Patient Findings The following parallel can be drawn between the brain lesion and functional neuroimaging results discussed so far: Patients with lesions in the prefrontal region appear to be impaired in tasks, such as free recall, that are usually associated with frontopolar activation (in addition to dorsolateral activation). On the other hand, they exhibit only mild or no impairment in tasks such as recognition, that tend to produce dorsolateral (but no frontopolar) activation in healthy controls. This may at ﬁrst appear somewhat surprising because it seems as if the dorsolateral cortex is not necessary for performance in neuropsychology studies of brain-lesioned patients, even though it is consistently recruited by the same tasks in neuroimaging studies of healthy people. In principle, there is a possibility that at least some of the activation observed in neuroimaging studies is epiphenomenal and that damage to a consistently activated region may have little or no effect on performance. There are, however, at least two other possible explanations, consistent with a hierarchical rostrocaudal prefrontal organization. First, for statistical reasons, a group of mixed prefrontal lesions is likely to include more patients with lesions in either one of these two regions than patients with lesions in only one of them – the dorsolateral region. For tasks requiring both regions, lesions extending over either of the two regions would produce deﬁcits, whereas for tasks requiring only dorsolateral cortex, only a subset of these lesions – the ones extending over the dorsolateral region – would produce deﬁcits. Therefore, on average, the group would show a stronger deﬁcit for tasks requiring both regions than for tasks requiring only dorsolateral cortex. This alone, however, may not be sufﬁcient to account for the normal performance of prefrontal groups in tasks that involve dorsolateral activation. In view of the hierarchical relationship between the two regions, it is conceivable that task processes usually associated speciﬁcally with dorsolateral prefrontal cortex activation may also be subserved by the frontopolar cortex, which, although normally not recruited, may take over some of the functions of the dorsolateral prefrontal cortex when the latter is lesioned. In such a case, only the patients with lesions extending over both the dorsolateral cortex and the frontopolar cortex would be impaired, which would

236

K. Christoff

greatly reduce the probability that the group on the average will exhibit a deﬁcit. However, whether the frontopolar cortex in patients with dorsolateral lesions can indeed assume functions typically subserved by the dorsolateral cortex remains to be determined by future neuroimaging studies of patient populations. In any case, it is striking that the deﬁcits seen in patients with frontal lesions are more closely linked to frontopolar than dorsolateral activations in healthy people.

3.8 Conclusions The review of reasoning and episodic retrieval studies presented here suggests that the frontopolar cortex is a functionally distinct prefrontal region that may be selectively involved in active processing, such as evaluation, monitoring, or manipulation, performed upon internally generated information. It is proposed that there may be a hierarchical distinction in a rostrocaudal direction between the frontopolar and the dorsolateral prefrontal regions of the cortex. Dorsolateral cortex may be sufﬁcient for the evaluation or manipulation of externally generated information, whereas frontopolar cortex is additionally required when evaluation and manipulation of internally generated information needs to be performed. Such a hierarchical distinction is consistent with, and can be viewed as an extension of, the two-stage model of processing within the lateral prefrontal cortex, which has been previously proposed in the literature. This latter model has viewed the ventrolateral and dorsolateral regions as subserving two different stages of executive processing within the lateral prefrontal cortex, with the dorsolateral region being involved at the second stage, where monitoring and manipulation of information held in working memory is required. The rostrocaudal prefrontal cortex distinction proposed here goes one step further and suggests that the frontopolar region in the human prefrontal cortex can be seen as subserving a third stage of executive processing, involving evaluation of information that has been generated at the previous stage of executive processing. Thus, the ventrolateral, dorsolateral, and frontopolar regions can be seen as forming a three-stage hierarchical system within the prefrontal cortex.

4 Evaluating Self-Generated Information and Introspective Thought To test the hypothesis that the frontopolar cortex is involved in the evaluation of self-generated information, we performed an event-related fMRI experiment, using a simple matching task designed to contrast directly the processing of internally and externally generated information (Fig. 6). The internal (Fig. 6, regions A, B) and external (Fig. 6, regions C, D) task conditions were similar in terms of overall demands, but differed in the critical requirement for processing self-generated information. During the sample phase, two objects were presented in the top part of

Human Thought and the Lateral Prefrontal Cortex

237

Fig. 6 Behavioral task. Stimuli were objects of six possible geometric shapes, ﬁlled with one of six possible textures. Each trial started with a ﬁxation cross, followed by an instruction cue. During the sample phase, a target set of two objects was presented at the upper part of the screen for 2 s. This was followed by a 6 s delay phase, during which the target set either remained on the screen (no-load trials) or was removed (load trials). During the test phase, a probe set of one or two objects was presented at the bottom half of the screen and subjects had to match it to the target set according to the instructions. On internally generated information trials (A, B) subjects had to infer the dimension of change between the top two objects (shape or texture) and decide whether the bottom two objects also change along this dimension. On externally generated information trials (C, D) subjects had to decide whether the bottom object matched any of the top objects along a speciﬁed dimension (shape or texture). On no-load trials (A, C) all objects were available on the screen during the decision, while on load trials (B, D), only the bottom set of objects was present and matching had to be performed from memory. Subjects responded with a “yes” or “no”, by pressing one of two buttons on a hand-held button box. The probe remained on the screen until the subject’s response, but no longer than 2 s. During the 8 s baseline period at the end of each trial, an arrow appeared at the center of the screen every 2 s, pointing randomly to the right or to the left. Subjects had to respond within 500 ms by pressing a key corresponding to the arrow’s direction

the screen. In the internal condition, subjects had to infer the dimension of change between the objects (shape or texture), whereas in external trials, they had to encode the objects in terms of their perceptual features. During the delay phase, the sample objects either remained on the screen (no-load trials) or were removed from the screen (load trials). In the latter case, subjects had to retain the relevant information in working memory. During the test phase one or two match stimuli were presented. In internal trials, subjects had to infer the dimension of change between the bottomtwo objects and decide whether it matched the previously inferred dimension of change between the top objects. In external trials, subjects had to decide whether the bottom object matched any of the top objects along a speciﬁed dimension (shape or texture). Thus, the test phase of each trial required evaluating either externally generated information about objects’ features, or internally generated information about the dimension of change between objects’ features. The contrast between the test phases of internal and external trials was designed to identify brain regions preferentially involved in evaluating internally generated information. Half of the trials (Fig. 6, regions A, C) posed no maintenance requirement, while the other half (Fig. 6, regions B, D) required maintenance of relevant information about the sample set in working memory. The purpose of this load manipulation was twofold: ﬁrst, to examine the processes of generation and maintenance of

1400

100

1200

95

1000

90

% Correct

K. Christoff

RT (ms)

238

External 85

800

Load trials

Internal

No-load trials

Fig. 7 Behavioral performance. Bars show mean reaction times across conditions and symbols show mean accuracy. Error bars indicate standard error of the mean. See Results and Discussion sections for details

self-generated information (occurring during the sample and delay phase of load trials, respectively), and second, to assess the contrast between evaluating different types of information in the absence and presence of concurrent maintenance requirements (occurring during the test phase of no-load and load trials, respectively). Behavioral results showed that subjects maintained a high level of performance throughout the experiment (Fig. 7). The mean accuracy was 96.34% (standard error 0.73%) and did not differ signiﬁcantly across conditions. Responses occurred on the average 1,014 ms (standard error 45.96 ms) after the onset of the test stimulus, and were 114 ms slower during internal trials than during external trials (F1,11 = 34.8, p < 0.001) and 62 ms slower during no-load trials than during loadtrials (F1,11 = 24.5, p < 0.005). A whole-brain voxel-based analysis contrasting the evaluation of internally and externally generated information (Fig. 8b) yielded only three areas of activation: bilateral rostrolateral prefrontal cortex (strongly on the right, weakly on the left) and left primary visual cortex (presumably owing to the different number of objects that had to be visually inspected). Rostrolateral prefrontal cortex activation was located within the predicted region (Fig. 8a), anatomically deﬁned (Christoff et al. 2001) as the region of intersection between middle frontal gyrus and BA10. These ﬁndings were conﬁrmed by an independent region-of-interest analysis of the event-related signal in rostrolateral prefrontal cortex during the test phase (Fig. 8c,d, yellow panels). Rostrolateral prefrontal cortex signal increased during evaluation of internally compared with externally generated information both during no-load (F1,11 = 7.49, p < 0.05) and during load (F1,11 = 5.29, p < 0.05) trials. Furthermore, this differential recruitment was speciﬁc to the process of evaluation, and was not observed during generation (F1,11 = 1.96, p = 0.19) or maintenance (F1,11 = 0, p = 0.98) of internally generated information (Fig. 8c, gray panels) – a result also supported by a signiﬁcant phase by condition interaction (F2,22 = 6.53, p < 0.01). These results indicate that the rostrolateral prefrontal cortex is preferentially recruited during deliberate, evaluative processing performed upon internally generated information, independent of concurrent maintenance requirements.

Human Thought and the Lateral Prefrontal Cortex Load trials

z = −4 mm

z = 0 mm

a

0.2

internally generated information

Evaluation

Maintenance

0

0

2

4

externally generated information

Test

−0.2

z = 0 mm

b

RLPFC

Generation

Delay

Signal change (%)

Evaluation (test phase)

Sample

RLPFC

239

10

20

Time(s)

c

NO-load trials

10

d

Time (s)

20

0

−0.2

10

externally generated information

Time (s)

VLPFC

20

0.2

Evaluation

internally generated information

0

−0.2

Test

−0.2

externally generated information

0.2

internally generated information

Signal change (%)

0

DLPFC Evaluation

Test

internally generated information

Signal change (%)

0.2

Test

Signal change (%)

RLPFC Evaluation

10

externally generated information

Time (s)

20

Fig. 8 Anatomical prediction and fMRI results. (A) The hypothesized (Christoff & Gabrieli, 2000) region of activation, RLPFC, anatomically deﬁned (Christoff et al., 2001) as the intersection between middle frontal gyrus and BA10. (B) Results from a whole-brain group analysis: Regions showing signiﬁcant increase during evaluation of internally generated information relative to the evaluation of externally generated information (load and no-load trials combined). Activations are overlaid on an average anatomical image and are thresholded at p < 0.05 corrected for multiple comparisons. (C, D) Event-related responses in anatomically deﬁned regions of interest. Plots show the median across subjects (±s.e.) in percent signal change. (C) Time-course observed in right RLPFC during the sample, delay, and test phases of load trials. (D) Time-course observed in right RLPFC, right DLPFC, and right VLPFC during the test phase of no-load trials

To test further the regional speciﬁcity of rostrolateral prefrontal cortex recruitment, the event-related signals in anatomically deﬁned dorsolateral prefrontal cortex and ventrolateral prefrontal cortex regions were examined during the evaluation of internally and externally generated information (Fig. 8d, blue and green panels). Only no-load trials were included to avoid effects of concurrent memory load, to which these two regions have been shown to be sensitive (D’Esposito et al. 1998; Fuster 1980; Goldman-Rakic 1987; Owen 1997; Petrides 1994). There was no difference in event-related signal between evaluation of internally and externally generated information in either dorsolateral prefrontal cortex (F1,11 = 0.14, p = 0.71) or ventrolateral prefrontal cortex (F1,11 = 0.15, p = 0.70). Furthermore, the regional speciﬁcity of effect in rostrolateral prefrontal cortex (Fig. 8d) was supported by a signiﬁcant region by condition interaction (F2,22 = 4.61, p < 0.05). Although the two posterior lateral prefrontal cortex subregions did not respond differentially between the internal and external conditions, both subregions were activated during both conditions, as revealed by the presence of a signiﬁcant quadratic trend in dorsolateral prefrontal cortex during internal (T94 = 3.36, p < 0.005) and external (T94 = 3.46, p < 0.001) trials, and in ventrolateral prefrontal cortex during internal (T94 = 4.02, p < 0.001) and external (T94 = 4.15, p < 0.001) trials.

240

K. Christoff

In contrast, rostrolateral prefrontal cortex was recruited only during internal trials (T94 = 4.08, p < 0.001), but not during external trials (T94 = 0.96, p = 0.34). These results provided direct evidence in support of the hypothesis that the rostral region of the human lateral prefrontal cortex is involved in processing self-generated information. The fMRI signal in rostrolateral prefrontal cortex showed a selective increase during processing of internally generated information compared with externally generated information. This increase was speciﬁc to the evaluation phase of each trial and was not observed during the generation or maintenance of selfgenerated information. In contrast to rostrolateral prefrontal cortex , the more posterior dorsolateral prefrontal cortex and ventrolateral prefrontal cortex regions did not differ in activation between the internal and external conditions, but were nevertheless recruited during both conditions. These results are consistent with the hierarchical model of lateral prefrontal cortex organization discussed earlier (Christoff and Gabrieli 2000), according to which dorsolateral prefrontal cortex and ventrolateral prefrontal cortex are involved when externally generated information is evaluated, whereas rostrolateral prefrontal cortex becomes additionally recruited when internally generated information needs to be evaluated. Such explicit processing of self-generated information may exemplify some of the highest orders of transformation in which the prefrontal cortex engages during the perception–action cycle (Benson 1993; Fuster 1980; Mesulam 1998; Stuss and Benson 1986; Wise et al. 1996). It may also be one of the mental processes that distinguish humans from other primate species. There are profound disparities among different primate species in their natural ability to process internally generated information. This is demonstrated by differences in performance in tasks requiring judgments analogous to that required during the test phase of the internal condition of the task employed here (Fig. 6, region A). Such tasks are often referred to as “relational matching-to-sample” procedures (Premack 1983) and can be distinguished from the traditionally employed “identity matching-to-sample” procedure, in that they require the animal to match abstract information about the relationship between a pair of objects (e.g., “same” or “different”) to the relationship between another pair of objects, irrespective of object identities. Only humans and chimpanzees with a history of language (Premack 1983) or token training (Thompson et al. 1997) can perform tasks requiring such judgments, while monkeys fail even after extensive training Thompson and Oden 2000. Furthermore, humans spontaneously develop this ability as early as 5 years of age (Halford 1984), while chimpanzees demonstrate it only in adulthood and only after extensive symbol training. This evolution in ability is paralleled by a twofold increase in the relative size of BA10 from chimpanzees to humans (Semendeferi et al. 2001) – an increase that appears to be selective to this region and occurs even though the relative size of the frontal lobe remains the same between the two species (Semendeferi et al. 1997, 2002). Although further anatomical and cytoarchitectonic studies are needed in order to establish with greater detail and certainty the changes BA10 has undergone in the course of primate evolution, this combination of behavioral and neuroanatomical evidence is consistent with the view that BA10 may play a critical role in mental operations that have emerged at the latest stages of evolutionary development.

Human Thought and the Lateral Prefrontal Cortex

241

Finally, the present study demonstrates involvement of lateral BA10 during the evaluation of self-generated cognitive information, whereas other functional neuroimaging studies have shown that medial BA10 is activated during judgments of self-generated emotional states (Damasio 2000; Gusnard et al. 2001; Lane et al. 1997; Zysset et al. 2002). This suggests that the entire region may be involved in the explicit processing of internally generated information, with lateral BA10 recruited during cognitively oriented tasks and medial BA10 recruited during emotionally oriented tasks. This ability to become aware of and explicitly process internal mental states – cognitive as well as emotional – may epitomize human mental abilities and may contribute to the enhanced complexity of thought, action, and social interactions observed in humans.

5 Spontaneous Thought Processes The study of higher cognitive function has focused almost exclusively upon mental processes occurring during complex, demanding cognitive tasks. The ﬂow of inner mental events, however, continues even when no tasks are present, forming a “stream of thought” in William James’s (1890) classic phrase. Such inner, spontaneous thought processes have been difﬁcult to observe and characterize using traditional experimental methods. Nevertheless, several lines of research from the behavioral literature and a number of functional neuroimaging observations provide relevant implications regarding their cognitive and neural basis. Here, we summarize behavioral and neuroimaging evidence indicating that spontaneous, taskunrelated cognitive processes share common cognitive and neural mechanisms with purposeful, task-related thought processes. This evidence, including a review of the relevant literature and ﬁndings from a new fMRI study, suggests that spontaneous thought is based upon higher-order cognitive processes and brain regions, among which long-term memory processes supported by temporal lobe structures may play a particularly prominent role. A long-established tradition of behavioral research has aimed at studying spontaneous thought processes. Such processes have been referred to as “daydreaming” (Singer 1966; Giambra 1979), “mind-wandering” (Antrobus et al. 1970), “stimulus-independent” (Antrobus 1968; Teasdale et al. 1993) or “taskunrelated” thought (Giambra and Grodsky 1989). Despite apparent differences in terminology, however, the target of research across studies has been the same: thought processes that occur spontaneously and bear no relation to the task at hand. These behavioral studies have indicated that spontaneous, task-unrelated thought processes are closely linked to the same cognitive mechanisms that underlie deliberate, task-related thought processes. Neuroimaging evidence offers remarkably similar conclusions. Most of this evidence has been derived from activation ﬁndings in higher cortical region in the absence of demanding tasks. The absence of task in the context of neuroimaging is typically referred to as “rest” and consists of blocks during which subjects are typically asked to simply

242

K. Christoff

remain lying still in the scanner and “do nothing.” Such rest conditions were originally intended to provide a baseline against which target conditions can be assessed. As Seneca remarked, however, in 62 AD, long before the advent of neuroimaging, “The fact that the body is lying down is no reason for supposing that the mind is at peace. Rest is sometimes far from restful.”1 If rest is marked by spontaneous thought processes relying on the same cognitive mechanisms as task-related thought, then an overlap in the pattern of brain activation during rest and cognitive tasks should be expected. Indeed, such an overlap has been observed ever since the beginning of neural explorations of higher cognition. Ingvar (1975) was the ﬁrst to report a “striking similarity” between the relative distribution of blood ﬂow in the “problem solving mode,” on the one hand, and in the “resting mode,” on the other hand. He observed that the relative increase in blood ﬂow over frontal regions associated with a performance of a reasoning test (the Raven’s Progressive Matrices) was matched, and even surpassed, by frontal increases in the absence of a task. Ingvar (1979) termed this resting pattern “hyperfrontal,” and attributed it to the fact that “thought processes in resting consciousness are constantly active.” In the time since these early observations, brain imaging techniques have undergone a reﬁnement in spatial and temporal sensitivity. Despite the lessons from the early neuroimaging ﬁndings, however, the use of a resting baseline for the study of higher cognitive processes has remained a common practice, frequently justiﬁed by the argument that mental processes during rest are likely to be unsystematic and unorganized, so that any corresponding neural activation would be nonlocalized and negligible. On several occasions, however, researchers have presented evidence to the contrary, demonstrating that particular brain regions are systematically activated during rest. One line of evidence suggesting that particular regions are consistently recruited during rest comes from observations of the absence of task-related activation in particular brain regions when a resting baseline is used, indicating that these regions are activated not only during the task, but also during rest. For instance, Kosslyn et al. (1995) observed visual cortex activation during a visual imagery task, but only when it was compared with a listening baseline condition; when a resting baseline was used, imagery activation was obscured, owing to the presence of rest-related activation in visual cortex. A similar pattern of ﬁndings was observed for medial temporal lobe structures (Stark and Squire 2001). In this study, brain structures such as the hippocampus and parahippocampal gyrus were activated not only during taskrelated memory encoding and retrieval, but also during periods of rest (in comparison with several alternative nonresting baseline conditions). The authors argued that “periods of rest are associated with signiﬁcant cognitive activity,” including incidental encoding and retrieval processes that would account for the observed rest-related activation. In addition, ﬁndings of absence of activation in another brain region, the rostrolateral prefrontal cortex, have also been linked to the use of resting baselines 1

Letters from a Stoic, Penguin, p. 111.

Human Thought and the Lateral Prefrontal Cortex

243

(Christoff and Gabrieli 2000). This prefrontal region is consistently activated during higher-order cognitive tasks, such as problem solving (Baker et al. 1996; Christoff et al. 2001) and complex memory retrieval (for a review, see Christoff and Gabrieli 2000). Buckner et al. (1996), however, used such a complex memory retrieval task involving cued recall of paired associates, and observed no rostrolateral prefrontal cortex activation when a resting baseline was used; when a nonresting word repetition baseline was used, however, rostrolateral prefrontal cortex activation was apparent. Similarly, Ragland et al. (1998) observed no rostrolateral prefrontal cortex activation when a problem solving task, the Wisconsin Card Sorting Test, was compared with rest – although such activation has been consistently reported for this task in comparison with nonresting baselines (Berman et al. 1995; Goldberg et al. 1998; Nagahama et al. 1996). Thus, the rostrolateral prefrontal cortex appears to be another brain region consistently recruited during rest – a recruitment that is likely due to evaluative processes directed towards the subjects’ own internal cognitive states during the resting period (Christoff and Gabrieli 2000; Christoff et al. 2003). Instead of focusing on the absence of activation in particular brain regions, some researchers have argued, similarly to Ingvar (1975, 1979), that the overall pattern of brain activation during rest resembles remarkably the pattern of activation associated with particular higher cognitive functions. Andreasen et al. (1995) used an episodic memory task requiring the recall of a speciﬁc event from one’s past experience, and a rest condition (or “random episodic silent thinking,” as they described rest, using an intentionally ironic acronym). Compared with a semantic memory condition consisting of recalling words that start with a speciﬁc letter, both the episodic and the rest condition produced activation in higher cortical regions, including prefrontal and parietal association cortices. Andreasen et al. argued that rest is likely to be associated with “random episodic memory” processes, or a type of free-association, uncensored recollection of past experiences. Indeed, in debrieﬁng interviews with subjects after the study, mental activity during rest was described as “quite vigorous” and consisting of a mixture of freely wandering past recollections, future plans, and other personal thoughts and experiences that appeared to be loosely linked. A related set of ﬁndings was reported by Binder et al. (1999), who observed a largely overlapping set of polymodal cortical regions activated both by a semantic retrieval task and by rest. The semantic task required subjects to listen to the names of animals (e.g., squirrel), and to respond when a named animal is found in the USA and is used by people (e.g., cow). Compared with a perceptual baseline (a tones task during which subjects listened to a sequence of low and high tones and had to respond when it included two high tones), both the semantic and the rest condition produced activation in regions similar to those reported by Andreasen et al. (1995) – prefrontal and parietal association cortices, medial temporal lobe, and cingulate cortices. Binder et al. argued that the observed rest-related pattern of activation reﬂected conceptual processes occurring during the conscious resting state: processes involving semantic knowledge retrieval, representation in awareness, and manipulation of represented knowledge.

244

K. Christoff

This summary of ﬁndings shows that a number of higher cortical regions, including visual areas, medial temporal lobe, and lateral association cortical areas, are consistently recruited during rest as well as during a variety of higher cognitive tasks. Furthermore, this recruitment has been associated with speciﬁc higher cognitive functions, such as visual imagery, memory retrieval, conceptual processing and problem solving – processes that seem to occur both in the presence and in the absence of cognitive tasks.

5.1 Rest as a Condition of Interest Discussions of neural recruitment associated with cognitive processes during rest have concentrated either on a particular brain region (Kosslyn et al. 1995; Christoff and Gabrieli 2000; Stark and Squire 2001), or on a network of activations overlapping between rest and a particular cognitive function, such as episodic (Andreasen et al. 1995) or semantic (Binder et al. 1999) retrieval. Furthermore, although the importance of studying the spontaneously occurring cognitive processes during rest has been emphasized repeatedly, virtually all discussion of these processes has been based on comparisons between rest, on the one hand, and some task posing relatively high cognitive demands, on the other hand. If rest is to be treated as a condition of interest, however, it is necessary to examine the whole-brain pattern of activation resulting from a comparison between rest and a baseline task that poses only minimal cognitive demands and is thus as closely matched to rest as possible. In the next section, we describe an fMRI study (Christoff et al, 2004) in which rest was explicitly treated as a condition of interest, and was compared with a baseline task requiring only minimal cognitive demands, in order to assess the resulting whole-brain pattern of activation.

5.2 Rest Compared with a Task of Minimal Cognitive Demands: An fMRI Study Subjects alternated between performing an arrows task and resting (Fig. 9). Blocks were 16 s long and there were eight blocks of each condition. During the arrows task, an arrow appeared on the screen every 2 s, pointing randomly to right or left. Subjects responded with their right hand, pressing one of two buttons on a hand-held button-box (right button for right arrow and left button for left arrow). They were instructed to respond as quickly as possible after each arrow’s onset, and were told responses would be considered incorrect if they occurred later than 500 ms after the arrow onset. Behavioral performance indicated that subjects followed closely the arrows task instructions. The mean accuracy was 98% (standard error 0.52%; range across subjects 94–100%), and all responses occurred within 500 ms after the arrow’s onset, with a mean response time of 353 ms (standard error 15.92 ms; range across subjects 270–452 ms).

Human Thought and the Lateral Prefrontal Cortex arrows

>

<

<

0

2

4

rest

245

arrows

> <

16

32

rest

< 48

sec

Fig. 9 Behavioural paradigm. Rest was compared to a closely matched, continuously engaging task of minimal cognitive demands (arrows)

Rest > Arrows

T value z = −32mm

z = −28mm

z = −16mm

z = 4mm

z = 20mm

z = 48mm

z = 20mm

z = 48mm

Arrows > Rest

z = −32mm

z = −28mm

z = −16mm

z = 4mm

6 4 2 0

Fig. 10 Activation results. Whole-brain pattern of activation resulting from random-effects group analysis (12 subjects). Activation maps are displayed on axial slices derived from the groupaveraged anatomical image. Results are thresholded at p < 0.005, with an extent threshold of 10 voxels

Comparing rest and the arrows task yielded activations in multiple regions (Fig. 10). Regions of strong activation in right temporopolar cortex and left parahippocampal gyrus were observed, surviving a threshold of p < 0.05 corrected for multiple comparisons throughout the brain. After relaxing the activation threshold to p < 0.001 uncorrected, the temporopolar and parahippocampal activations were found to extend bilaterally. At this threshold level, several additional regions of activation were observed, including left rostrolateral prefrontal cortex, primary and extrastriate visual areas, left insula, and right inferior parietal cortex. Comparing the arrows task and rest yielded a single cluster of activation in the left primary motor cortex (Fig. 10). Activation in this cluster was signiﬁcant at p < 0.05 corrected for multiple comparisons within an a priori deﬁned region of interest comprising the left motor cortex (BA4).

246

K. Christoff

A network of regions related to spontaneous cognitive processes was identiﬁed, by treating rest as a condition of interest and comparing it with a continuously engaging task of minimal cognitive demands. The most robust activation was localized in temporal lobe structures, including lateral anterior and medial temporal regions. Activations were also observed in anterior lateral prefrontal cortex and visual cortex areas. Thus, neural recruitment during rest was statistically robust and consistently localized to speciﬁc brain regions. Furthermore, the strength of observed activations was comparable to that seen during many highly demanding cognitive tasks. The observed activations were localized to brain regions that have been implicated repeatedly in a variety of higher cognitive functions – as well as rest (Andreasen et al. 1995; Binder et al. 1999). However, unlike previous related studies, no cognitive task was present in either condition of the present comparison; therefore, the observed pattern could not have been inﬂuenced by a relative difference in the employment of particular task-related cognitive processes. Furthermore, cognitive demands posed by the two conditions in the present comparison were kept at a minimum; thus, the activation pattern was also unlikely to be inﬂuenced by variations in cognitive demands. The lack of modulation in medial prefrontal cortex – a region frequently activated with reduction in cognitive demands (Shulman et al. 1997; Raichle 1998; Mazoyer et al. 2001) – was, therefore, likely due to the lack of modulation of cognitive demands in the present comparison.

6 Final Conclusions Neuroimaging and patient studies such as those outlined here bring us closer to understanding the complex processes underlying human thought and reasoning. The proposed anterior-to-posterior hierarchical organization of thought within the lateral prefrontal cortex hints at the importance of considering this region in terms of functionally distinct subregions. In particular, the apparent organization of the lateral prefrontal cortex according to different levels of abstraction in thought, with more abstract thought distributed in the anterior direction, has potential implications for cognitive models of cognition and thought as well as for neuropsychological investigations of executive function. Nevertheless, full understanding of human thought will not be achieved until the complete organization and functional connections of the lateral prefrontal cortex are understood, including both local interconnections within the lateral prefrontal cortex, as well as its long-range connections with other cortical regions, including the temporal lobe and the basal ganglia. Future research aimed at further elucidating the functions and interactions of these regions will create a clearer picture of the brain mechanisms involved in human cognition. However, this picture will remain incomplete unless light is shed on spontaneously occurring, undirected thought processes and phenomena, including mind-wandering and insight during problem solving, as well as on goal-directed thought processes. Understanding our own thought processes from a scientiﬁc and cognitive neuroscience perspective is a daunting and challenging task, which may never be fully

Human Thought and the Lateral Prefrontal Cortex

247

achievable. In our striving to achieve understanding, we may change those very thought processes which we are trying to understand. But possibly precisely because of this, achieving such an understanding remains one of the most important goals of contemporary neuroscience. Acknowledgements Preparation of this chapter was supported by a Tula Foundation Young Scientist award and a Canadian Foundation for Innovation (CFI) award to K.C. I am indebted to Rachelle Smith for her exceptional help and thoughtful comments and suggestions in preparing this chapter.

References Alexander GE, DeLong MR et al. (1986) Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu Rev Neurosci 9:357–381 Andreasen NC, O’Leary DS et al. (1995) Remembering the past: two facets of episodic memory explored with positron emission tomography. Am J Psychiatry 152(11):1576–1585 Antrobus JS (1968) Information theory and stimulus-independent thought. Br J Psychol 59: 423–430 Antrobus JS, Singer JL et al. (1970) Mindwandering and cognitive structure. Trans N Y Acad Sci 32(2):242–252 Baddeley A, Wilson B (1986) Amnesia, autobiographical memory, and confabulation. In: Rubin DC (ed) Autobiographical memory. Cambridge University Press, Cambridge, pp 225–252 Baddeley A, Wilson B (1988) Frontal amnesia and the dysexecutive syndrome. Brain Cogn 7(2):212–230 Baker SC, Rogers RD et al. (1996) Neural systems engaged by planning: a PET study of the Tower of London task. Neuropsychologia 34(6):515–526 Barbas H, Pandya DN (1991) Patterns of connections of the prefrontal cortex in the rhesus monkey associated with cortical architecture. In: Levin HS, Eisenberg HM, Benton AL (eds) Frontal lobe function and dysfunction. Oxford University Press, New York, pp 35–58 Benson DF (1993) Prefrontal abilities. Behav Neurol 6:75–81 Berman KF, Ostrem JL et al. (1995) Physiological activation of a cortical network during performance of the Wisconsin Card Sorting Test: a positron emission tomography study. Neuropsychologia 33(8):1027–1046 Binder JR, Frost JA et al. (1999) Conceptual processing during the conscious resting state. A functional MRI study. J Cogn Neurosci 11(1):80–95 Brodmann K (1908) Beitraege zur histologischen Lokalisation der Grosshirnrinde. VI Mitteilung. Die Cortexgliederung des Menschen. J Psychol Neurol 10:213–246 Buckner RL, Raichle ME et al. (1996) Functional anatomic studies of memory retrieval for auditory words and visual pictures. J Neurosci 16:6219–6235 Bunge SA, Burrows B et al. (2004) Prefrontal and hippocampal contributions to visual associative recognition: interactions between cognitive control and episodic retrieval. Brain and cognition, 56(2), 141–152 Burgess PW, Shallice T (1996) Confabulation and the control of recollection. Memory 4(4): 359–411 Cabeza R, Kapur S et al. (1997) Functional neuroanatomy of recall and recognition: a PET study of episodic memory. J Cogn Neurosci 9(2):254–265 Campbell AW (1905) Histological studies on the localization of cerebral function. Cambridge University Press, Cambridge Carpenter PA, Just, MA (1990) What one intelligence test measures: A theoretical account of the processing in the Raven Progressive Matrices Test. Psychological Review, 97(3), 404–431

248

K. Christoff

Christoff K, Gabrieli JDE (2000) The frontopolar cortex and human cognition: evidence for a rostrocaudal hierarchical organization within the human prefrontal cortex. Psychobiology 28(2):168–186 Christoff K, Prabhakaran V et al. (2001) Rostrolateral prefrontal cortex involvement in relational integration during reasoning. Neuroimage 14(5):1136–1149 Christoff K, Ream JM et al. (2003) Evaluating self-generated information: anterior prefrontal contributions to human cognition. Behav Neurosci 117(6):1161–1168 Christoff K, Ream JM et al. (2004) Neural basis of spontaneous thought processes. Cortex 40(4–5):623–630 Cohen JD, Forman SD et al. (1994) Activation of the prefrontal cortex in a nonspatial working memory task with functional MRI. Hum Brain Mapp 1:293–304 D’Esposito M, Aguirre GK et al. (1998) Functional MRI studies of spatial and nonspatial working memory. Cogn Brain Res 7(1):1–13 Dagher A, Owen AM et al. (1999) Mapping the network for planning: a correlational PET activation study with the Tower of London task. Brain 122(10):1973–1987 Damasio AR (2000) Subcortical and cortical brain activity during the feeling of self-generated emotions. Nat Neurosci 3(10):1049–1056 Damasio H, Grabowski TJ et al. (1996) A neural basis for lexical retrieval. Nature 380 499–505 della Rocchetta AI (1986) Classiﬁcation and recall of pictures after unilateral frontal or temporal lobectomy. Cortex 22(2):189–211 di Pellegrino G, Wise SP (1991) A neurophysiological comparison of three distinct regions of the primate frontal lobe. Brain 114:951–978 Elliott R, Frith CD et al. (1997) Differential neural response to positive and negative feedback in planning and guessing tasks. Neuropsychologia 35(10):1395–1404 Elliott Smith G (1907) A new topographical survey of the human cerebral cortex. J Anat Physiol 41:237–254 Fuster JM (1980) The prefrontal cortex. Anatomy, physiology and neuropsychology of the frontal lobe. Raven, New York Gershberg FB, Shimamura AP (1995) Impaired use of organizational strategies in free recall following frontal lobe damage. Neuropsychologia 33(10):1305–1333 Giambra LM (1979) Sex differences in daydreaming and related mental activity from the late teens to the early nineties. Int J Aging Hum Dev 10(1):1–34 Giambra LM, Grodsky A (1989) Task-unrelated images and thoughts while reading. In: Shorr JE, Robin P, Connella JA, Wolpin M (eds) Imagery: current perspectives. Plenum, New York, pp 7–31 Gigerenzer G, Murray DJ (1987) Cognition as intuitive statistics. Erlbaum, Hillsdale Goel V, Dolan RJ (2000) Anatomical segregation of component processes in an inductive inference task. J Cogn Neurosci 12(1):110–119 Goel V, Gold B et al. (1997) The seats of reason? An imaging study of deductive and inductive reasoning. Neuroreport 8:1305–1310 Goel V, Gold B et al. (1998) Neuroanatomical correlates of human reasoning. J Cogn Neurosci 10(3):293–302 Goel V, Buchel C et al. (2000) Dissociation of mechanisms underlying syllogistic reasoning. Neuroimage 12(5):504–514 Goldberg TE, Berman KF et al. (1998) Uncoupling cognitive workload and prefrontal cortical physiology: a PET rCBF study. NeuroImage 7(4):296–303 Goldman-Rakic PS (1987) Circuitry of primate prefrontal cortex and regulation of behavior by representational memory. In: Plum F, Mountcastle VB (eds) Handbook of physiology. Section 1, the nervous system, vol 5. Higher functions of the brain. American Psychological Society, Bethesda, pp 373–417 Grasby PM, Frith CD et al. (1993) Functional mapping of brain areas implicated in auditory-verbal memory function. Brain 116:1–20 Gusnard DA, Akbudak E et al. (2001) Medial prefrontal cortex and self-referential mental activity: relation to a default mode of brain function. Proc Natl Acad Sci U S A 98(7):4259–4264

Human Thought and the Lateral Prefrontal Cortex

249

Halford GS (1984) Can young children integrate premises in transitivity and serial order tasks? Cogn Psychol 16(1):65–93 Halford GS, Wilson WH et al. (1998) Processing capacity deﬁned by relational complexity: implications for comparative, developmental, and cognitive psychology. The Behavioral and brain sciences, 21(6), 803–831; discussion 831–864 Henson RN, Shallice T et al. (1999) Right prefrontal cortex and episodic memory retrieval: a functional MRI test of the monitoring hypothesis. Brain 122:1367–1381 Ingvar DH (1975) Patterns of brain activity revealed by measurements of regional cerebral blood ﬂow. In: Ingvar DH, Lassen NA (eds) Brain work. Academic, New York Ingvar DH (1979) “Hyperfrontal” distribution of the cerebral grey matter ﬂow in resting wakefulness; on the functional anatomy of the conscious state. Acta Neurol Scand 60(1):12–25 James W (1890) Principles of psychology. Holt, New York Janowsky JS, Shimamura AP et al. (1989) Source memory impairment in patients with frontal lobe lesions. Neuropsychologia 27(8):1043–1056 Jetter W, Poser U et al. (1986) A verbal long term memory deﬁcit in frontal lobe damaged patients. Cortex 22:229–242 Johnson MK, Kounios J et al. (1996) Electrophysiological brain activity and memory source monitoring. Neuroreport 7(18):2929–2932 Jonides J, Schumacher EH et al. (1997) Verbal working memory load affects regional brain activation as measured by PET. J Cogn Neurosci 9(4):462–475 Koechlin E, Basso G et al. (1999) The role of the anterior prefrontal cortex in human cognition. Nature 399(6732):148–151 Kosslyn S.M, Thompson WL et al. (1995) Topographical representations of mental images in primary visual cortex. Nature 378(6556):496–498 Kroger JK, Sabb FW et al. (2002) Recruitment of anterior dorsolateral prefrontal cortex in human reasoning: a parametric study of relational complexity. Cerebral Cortex, 12(5), 477–485 Lane RD, Fink GR et al. (1997) Neural activation during selective attention to subjective emotional responses. Neuroreport 8(18):3969–3972 Luria AR (1966) Higher cortical functions in man. London: Tavistock Publications Luria AR (1973) The working brain: an introduction to neuropsychology. Basic Books, New York MacLeod AK, Buckner RL et al. (1998) Right anterior prefrontal cortex activation during semantic monitoring and working memory. Neuroimage 7(1):41–48 Mazoyer B, Zago L et al. (2001) Cortical networks for working memory and executive functions sustain the conscious resting state in man. Brain Res Bull 54(3):287–298 Mesulam MM (1998) From sensation to cognition. Brain 121(6):1013–1052 Miller LA (1985) Cognitive risk-taking after frontal or temporal lobectomy – I. The synthesis of fragmented visual information. Neuropsychologia 23(3):359–369 Miller LA (1992) Impulsivity, risk-taking, and the ability to synthesize fragmented information after frontal lobectomy. Neuropsychologia 30(1):69–79 Milner B (1963) Effects of different brain lesions on card sorting. Archiv Neurol 9:90–100 Milner B (1964) Some effects of frontal lobectomy in man. In: Warren JM, Akert K (eds) The frontal granular cortex and behavior. McGraw-Hill, New York, pp 313–334 Milner B, Petrides M et al. (1985) Frontal lobes and the temporal organization of memory. Hum Neurobiol 4(3):137–142 Milner B, Corsi P et al. (1991) Frontal-lobe contribution to recency judgements. Neuropsychologia 29(6):601–618 Moscovitch M (1989) Confabulation and the frontal systems: strategic versus associative retrieval in neuropsychological theories of memory. In: Roediger HLI, Craik FIM (eds) Varieties of memory and consciousness: essays in honour of Endel Tulving. Erlbaum, Hillsdale, pp 133–160 Nagahama Y, Fukuyama H et al. (1996) Cerebral activation during performance of a card sorting test. Brain 119:1667–1675 Nolde SF, Johnson MK et al. (1998) The role of prefrontal cortex during tests of episodic memory. Trends Cogn Sci 2(10):399–406

250

K. Christoff

Norman DA, Shallice T (1986) Attention to action: willed and automatic control of behavior. In: Davidson RJ, Schwarts GE, Shapiro D (eds) Consciousness and self-regulation. Advances in research and theory, vol 4. Plenum, New York, pp 1–18 Nyberg L, Cabeza R et al. (1996) PET studies of encoding and retrieval: the HERA model. Psychon Bull Rev 3(2):135–148 Osherson D, Perani D et al. (1998) Distinct brain loci in deductive versus probabilistic reasoning. Neuropsychologia 36:369–376 Owen AM (1997) The functional organization of working memory processes within human lateral frontal cortex: the contribution of functional neuroimaging. Eur J Neurosci 9(7):1329–1339 Owen AM, Downes JJ et al. (1990) Planning and spatial working memory following frontal lobe lesions in man. Neuropsychologia 28(10):1021–1034 Owen AM, Evans AC et al. (1996) Evidence for a two-stage model of spatial working memory processing within the lateral frontal cortex: a positron emission tomography study. Cereb Cortex 6(1):31–38 Pandya DN, Barnes CL (1987) Architecture and connections of the frontal lobe. In: Perecman E (ed) The frontal lobes revisited. IRBN, New York, pp 41–72 Petrides M (1991) Functional specialization within the dorsolateral frontal cortex for serial order memory. Proc R Soc Lond Ser B 246:299–306 Petrides M (1994) Frontal lobes and behaviour. Current opinion in neurobiology, 4(2), 207–211 Petrides M, Milner B (1982) Deﬁcits on subject-ordered tasks after frontal and temporal lobe lesions in man. Neuropsychologia 20:601–614 Petrides M, Alivisatos B et al. (1993) Functional activation of the human frontal cortex during the performance of verbal working memory tasks. Proc Natl Acad Sci U S A 90:878–882 Petrides M, Alivasatos B et al. (1995) Functional activation of the human ventrolateral frontal cortex during mnemonic retrieval of verbal information. Proc Natl Acad Sci U S A 92:5803– 5807 Prabhakaran V, Smith JA et al. (1997) Neural substrates of ﬂuid reasoning: an fMRI study of neocortical activation during performance of the Raven’s Progressive Matrices test. Cogn Psychol 33(1):43–63 Prabhakaran V, Narayanan K et al. (2000) Integration of diverse information in working memory within the frontal lobe. Nature neuroscience, 3(1), 85–90 Premack D (1983) The codes of man and beasts. Behav Brain Sci 6(1):125–167 Ragland JD, Gur RC et al. (1998) Frontotemporal cerebral blood ﬂow change during executive and declarative memory tasks in schizophrenia: a positron emission tomography study. Neuropsychology 12(3):399–413 Raichle ME (1998) The neural correlates of consciousness: an analysis of cognitive skill learning. Philos Trans R Soc Lond Ser B 353(1377):1889–1901 Rao SM, Bobholz JA et al. (1997) Functional MRI evidence for subcortical participation in conceptual reasoning skills. Neuroreport 8(8):1987–1993 Raven JC (1938) Standardization of progressive matrices. British Journal of Medical Psychology, 19, 137–150 Robin N, Holyoak KJ (1995) Relational complexity and the functions of prefrontal cortex. In MS Gazzaniga (ed) The cognitive neurosciences (1 ed., pp 987–997). Cambridge, MA: MIT Press Robinson AL, Heaton RK et al. (1980) The utility of the Wisconsin Card Sorting Test in detecting and localizing frontal lobe lesions. J Consult Clin Psychol 48(5):605–614 Rolls ET (1996) The orbitofrontal cortex. Philos Trans R Soc Lond Ser B 351:1433–1443 Rosenkilde CE (1979) Functional heterogeneity of the prefrontal cortex in the monkey: a review. Behav Neural Biol 25(3):301–345 Rypma B, Prabhakaran V et al. (1999) Load-dependent roles of frontal brain regions in the maintenance of working memory. Neuroimage 9(2):216–226 Schacter DL, Curran T et al. (1996) False recognition and the right frontal lobe: a case study. Neuropsychologia 34:793–808

Human Thought and the Lateral Prefrontal Cortex

251

Semendeferi K, Damasio H et al. (1997) The evolution of the frontal lobes: a volumetric analysis based on three-dimensional reconstructions of magnetic resonance scans of human and ape brains. J Hum Evol 32(4):375–388 Semendeferi K, Armstrong E et al. (2001) Prefrontal cortex in humans and apes: a comparative study of area 10. Am J Phys Anthropol 114(3):224–241 Semendeferi K, Lu A et al. (2002) Humans and great apes share a large frontal cortex. Nat Neurosci 5(3):272–276 Seneca LA (1969) Letters from a Stoic (pp. 111). London, England: Penguin Books Shallice T (1982) Speciﬁc impairments of planning. Philos Trans R Soc Lond Ser B 298(1089):199–209 Shallice T, Evans ME (1978) The involvement of the frontal lobes in cognitive estimation. Cortex 14(2):294–303 Shallice T, Burgess PW et al. (1989). The origins of utilization behaviour. Brain 112:1587–1598 Shallice T, Fletcher P et al. (1994) Brain regions associated with acquisition and retrieval of verbal episodic memory. Nature 368:633–635 Shulman GL, Fiez JA et al. (1997) Common blood ﬂow changes across visual tasks: II. Decreases in cerebral cortex. J Cogn Neurosci 9(5):648–663 Singer JL (1966) Daydreaming: an introduction to the experimental study of inner experience. Random House, New York Smith ML, Milner B (1988) Estimation of frequency of occurrence of abstract designs after frontal or temporal lobectomy. Neuropsychologia 26:297–306 Smith EE, Jonides J et al. (1996) Dissociating verbal and spatial working memory using PET. Cereb Cortex 6(1):11–20 Squire LR, Cohen NJ (1982) Remote memory, retrograde amnesia and the neuropsychology of memory. In: Cermak LS (ed) Human memory and amnesia. Erlbaum, Hillsdale, pp 275–303 Stark CE, Squire LR (2001) When zero is not zero: the problem of ambiguous baseline conditions in fMRI. Proc Natl Acad Sci U S A 98(22):12760–12766 Sternberg S (1966) High-speed scanning in human memory. Science 153(3736):652–654 Stone M, Seger CA et al. (1998, June) Studying privileged access with functional MRI. Paper presented at the Neural Correlates of Consciousness Conference, Bremen, Germany Stuss DT, Benson DF (1986) The frontal lobes. Raven, New York Stuss DT, Alexander MP et al. (1978) An extraordinary form of confabulation. Neurology 28(11):1166–1172 Stuss DT, Eskes GA et al. (1994) Experimental neuropsychological studies of frontal lobe functions. In: Boller F, Grafman J (eds) Handbook of neuropsychology, vol 9. Elsevier, Amsterdam, pp 149–185 Teasdale JD, Proctor L et al. (1993) Working memory and stimulus-independent thought: effects of memory load and presentation rate. Eur J Cogn Psychol 5(4):417–433 Tomasello M, Call J (1997) Primate cognition. New York: Oxford University Press Thompson RKR, Oden DL et al. (1997) Language-naive chimpanzees (pan troglodytes) judge relations between relations in a conceptual matching-to-sample task. J Exp Psychol Anim Behav Process 23(1):31–43 Thompson RKR, Oden DL (2000) Categorical perception and conceptual judgments by nonhuman primates: The paleological monkey and the analogical ape. Cognitive Science, 24(3), 363–396 Vilkki J, Holst P (1991) Mental programming after frontal lobe lesions: results on digit symbol performance with self-selected goals. Cortex 27(2):203–211 ¨ Vogt O (1906) Uber strukturelle Hirnzentra, mit besonderer Ber¨ucksichtingung der strukturellen Felder des Cortex pallii. Anat Anz 29:74–114 Waltz JA, Knowlton BJ et al. (1999) A system for relational reasoning in human prefrontal cortex. Psychological Science, 10(2), 119–125 Wagner AD, Desmond JE et al. (1998) Prefrontal cortex and recognition memory. Functional-MRI evidence for context-dependent retrieval processes. Brain 121(10):1985–2002 Wharton CM, Grafman J et al. (2000) Toward neuroanatomical models of analogy: a positron emission tomography study of analogical mapping. Cogn Psychol 40(3):173–197

252

K. Christoff

Wheeler MA, Stuss DT et al. (1995) Frontal lobe damage produces episodic memory impairment. J Int Neuropsychol Soc 1:525–536 Wise SP, Murray EA et al. (1996) The frontal cortex-basal ganlgia system in primates. Crit Rev Neurobiol 10(3–4):317–356 Zysset S, Huber O et al. (2002) The anterior frontomedian cortex and evaluative judgment: an fMRI study. Neuroimage 15(4):983–991

Neural Correlates of Insight Phenomena Jing Luo(¬), Gunther Knoblich, and Chongde Lin

1 Introduction Difﬁcult problems are sometimes solved in a sudden ﬂash of illumination, a phenomenon referred to as “insight.” Recent neuroimaging studies have begun to reveal the neural correlates of the cognitive processes underlying such insight phenomena (Luo and Niki 2003; Jung-Beeman et al. 2004; Luo et al. 2004a, 2006; Mai et al. 2004; Lang et al. 2006). However, researchers have encountered a number of difﬁculties in applying neuroimaging methods to investigate insight. We will outline these difﬁculties, deﬁne general criteria brain-imaging studies of insight should meet, and then discuss in detail to what extent these criteria have been met in recent attempts to unravel the brain bases of insight. One main difﬁculty that is well known from behavioral research is to produce insight phenomena in the laboratory. Even in purely behavioral studies it is hard to be certain whether participants in laboratory settings actually have insights or whether they solve problems in a more stepwise manner. A related problem is the small numbers of problems that are available to study insight (Bowden et al. 2005). In addition, the well-known classical insight problems, such as the nine-dot problem (Scheerer 1963), the two string problem (Maier 1930), and the candle problem (Duncker 1945), greatly vary in their sources of difﬁculty (Kershaw and Ohlsson 2004). This raises the question whether the data obtained in laboratory research can be generalized. Furthermore, studying how people solve single problems normally does not produce very reliable data. For this reason, recent experimental paradigms addressing the brain bases of insight have adopted so-called mini-insight problems (Bowden et al. 2005). These problems can normally be solved in a short period with or without external help and many exemplars can be created (Luo and Niki 2003; Jung-Beeman et al. 2004). J. Luo Chinese Academy of Sciences, Department of Psychology, Da-tun Road 10, Chao-yang District, Beijing 100101, Peoples Republic of China [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

253

254

J. Luo et al.

However, in addition to picking the right tasks, brain-imaging studies of insight pose a number of other (related) problems. This led us to suggest a number of criteria that would characterize the ideal experimental paradigm for studying the neural correlates of insight with functional MRI (fMRI) or EEG (Luo and Knoblich 2007). We will discuss each of these ﬁve criteria in turn: (1) Use of problems that elicit restructuring: Although there is debate about how insight is best characterized, most modern researchers agree with the Gestalt psychologists’ point of view that insight involves a restructuring of the problem situation (Duncker 1945; K¨ohler 1921; Wertheimer 1925, 1959). In order to solve insight problems one needs to detach oneself from one’s prior experience with similar problems and to treat the problem in a novel and efﬁcient way (Davidson 2003; Knoblich et al. 1999; Ohlsson 1992; Weisberg 1995). For this reason, any study that claims to investigate the neural correlates of insight should create mental events that essentially contain the feature of restructuring. (2) Multiple insight events and accurate onset time: The neuroimaging study of insight requires that multiple insight events can be elicited within a limited time period. Most event-related fMRI or event-related-potential (ERP) studies require at least ten to 50 trials in each condition. Although it is possible that efﬁcient single trial analysis methods will be reliably established in the future, for now, multiple insight events are a must. A related requirement is that one needs to be able to precisely time-lock the insight events. In event-related fMRI or ERP studies, the researcher needs to know exactly the temporal onset of the critical mental events. One may wonder whether one can avoid this problem using fMRI studies adopting a block design. However, we believe that these designs are not well suited for the study of insight. The reason is that restructurings are usually short-lived moments of exceptional thinking that would only make up a tiny fraction of all mental processes occurring within a block. Thus, it is likely that brain activations reﬂecting insight will get lost in myriads of other activations when using block designs. (3) Hypothesis testing: The ideal experimental paradigm to study insight should allow one to perform ﬂexible manipulations to test various kinds of research hypotheses. This includes general hypotheses derived from theories as well as hypotheses about the precise function of particular brain areas. For example, during the moment of insight, an old inefﬁcient way of thinking is likely to be replaced by a new and more efﬁcient way of thinking and this replacement implies cognitive conﬂict. Thus, one could predict that brain areas that mediate the processing of cognitive conﬂict (e.g., the anterior cingulate cortex, ACC; Botvinick et al. 2001; Carter et al. 2000; MacDonald et al. 2000) should participate in the restructuring. This hypothesis is based on cognitive models of insight and should hold across different studies of insight, regardless of the particular problem used. However, in addition to testing general hypotheses, it is also important to know the exact function of a given region in restructuring. To determine this function is not as simple as it might seem, because insight is a holistic process in which people achieve multiple breakthroughs in one single thought (Kershaw and Ohlsson 2004). A good experimental paradigm for the

Neural Correlates of Insight Phenomena

255

study of insight would be ﬂexible enough to enable a number of manipulations to test more speciﬁc hypotheses. In the abovementioned example, ACC activation could be related to different functions, such as conﬂict monitoring (realizing the contradictions between different ways of thinking), error detection (realizing that one’s initial thinking was inappropriate), problem success (realizing the crucial step towards the solution), or general attentive control. The above examples illustrate that the ideal experimental paradigm for the study of insight would enable one to conduct precise tests of multiple alternative hypotheses. (4) Deﬁning reference states: An ideal insight paradigm should enable researchers to deﬁne suitable reference states. Brain-imaging analysis relies heavily on the contrast between a target state and a reference state (i.e., the baseline). An ideal reference state should be comparable with the target state in every aspect except the one to be examined. Compared with other domains of brain-imaging research, it is relatively difﬁcult to come up with good reference states in studies of insight problem solving, because insight includes a set of highly integrated processes that are released in one moment. This makes insight somewhat incomparable with other analytical modes of thinking. (5) Triggers for insight: Finally, the ideal insight paradigm should allow one to study internally and externally triggered insights. This refers to the fact that problem solvers can achieve restructurings on their own (internal trigger). Alternatively, restructuring can be triggered by solution hints (external trigger). Although many behavioral and neuroimaging experiments addressing insight problem solving are based on the assumption that solution hints trigger similar processes as internally generated solution attempts (Kershaw and Ohlsson 2004; Ormerod et al. 2002; Weisberg and Alba 1981; Luo and Niki 2003), one cannot be sure whether this assumption really holds. Without doubt, the phenomenon of interest is internally generated insight. Triggering insights externally is just a way of creating paradigms that make scientiﬁc research on insight tractable. However, there is a conﬂict between the requirement of ecological validity that dictates one to investigate internally generated insights and the methodological requirement of accurate onset times for target events in event-related fMRI and ERP studies (see point 2). Accurate onset times are more easily obtained for externally triggered insights than for the internally generated ones. Although we can ask participants to indicate the time of their insight (e.g., with a button press), participants’ reports may be delayed. Thus, researchers may have to come up with some estimate that allows them to go back several hundred milliseconds to anchor the onset time of the internally generated insight and they can never be sure whether the event timing is correct. The above list of criteria illustrates that coming up with an adequate experimental setting to study insight is highly challenging. In fact, we are not aware of any single brain study of insight that would meet all of these criteria. However, we think the ﬁve criteria are useful as a benchmark against which brain studies of insight can be tested. However, each single study will likely have to make compromises. There is no general rule as where to cut back. This will depend on the particu-

256

J. Luo et al.

lar question addressed by a study and the particular methods employed (Luo and Knoblich 2007). Before we discuss how recent studies addressing the brain processes underlying insight meet the abovementioned criterion, we would like to emphasize our belief that insight can only be understood if we conceptualize it as a cluster of different processes working together. In our view, insight is not a single process. Rather, it is a collection of different mental processes. Some of these processes may occur in many different insight tasks (e.g., detecting that there is a conﬂict). Others may be less general and may only be needed for particular types of insight tasks. For instance, perceptual processes may only be involved in tasks where the solution crucially depends on a spatial restructuring of problem elements. Thus, studying insight is different from studying basic cognitive processes. Take episodic memory as an example for such a basic process. We know that a particular set of brain mechanisms, including the key function of associating or binding different mental events, subserve the formation and retrieval of episodic memory, regardless of whether the contents of the memory are words, pictures, voices, or emotional responses. However, for higher cognitive processes such as thinking and reasoning, the situation is more complicated. Recent neuroimaging studies reveal that in seemingly identical syllogistic reasoning tasks, the content-based (concrete) reasoning and the abstract reasoning (which lacked semantic content) activated distinct neural networks (Goel et al. 2000). The diversity of brain processes involved is even larger for insight. Although restructuring is believed to be the main component of insight, previous research suggests that there are different types of restructuring. Restructuring can involve a perceptual reinterpretation of the problem (Ohlsson 1992), directing attention to the critical problem elements (Knoblich et al. 2001, 2005; Grant and Spivey 2003), a recombination of elements that gives the problem a new meaning (Bowden et al. 2005; Davidson et al. 1995), or a change in the goal of problem solving (Ohlsson 1992). Therefore, when looking at different paradigms for insight study, we may not ask if these studies investigate the identical cognitive process or which study is the only correct one that truly addressed the target topic. Rather, we may keep in mind that different paradigms may be related to different types of restructuring and all of them belong to the category of insight, and the correct question we should ask is which type of restructuring is examined in a given study. In this chapter, we will discuss four different experimental paradigms that have been used to address the brain processes underlying insight: The ﬁrst paradigm (Luo and Niki 2003; Mai et al. 2004; Luo et al. 2004a) used ambiguous riddles and puzzles to study insights that result from a reinterpretation of meaning. The second paradigm used items from the Remote Associates Test (RAT) to address how one can create meaningful links between seemingly unrelated items (Jung-Beeman et al. 2004; Kounios et al. 2006). The third paradigm addressed perceptual contributions to insight, in particular, the decomposition of perceptual chunks (Luo et al. 2006). Finally, a fourth paradigm addressed the question of whether insight occurs in repetitive tasks that provide opportunities for effective changes in strategy (Haider and Rose 2007; Lang et al. 2006).

Neural Correlates of Insight Phenomena

257

2 Reinterpretation of Meaning in Riddles and Puzzles One attempt to study the brain bases of insight used riddles and puzzles that can reliably produce insight-like experiences within a relatively short time. One important factor in this approach is to select riddles for which participants understand the exposition of the problem well but for which they cannot produce a solution. During scanning, an external trigger (the solution) is provided to catalyze the riddle-solving process. This allows one to produce insight-like experiences at particular points in time and to record neural activity correlated with these experiences in particular time windows. Of course, this implies the troubling assumption that an external solution hint triggers similar processes as an internally generated insight. Three types of riddles or puzzles have been used in brain studies on insight so far. The ﬁrst type were riddles such as “The thing that can move heavy logs, but cannot move a small nail” (the answer is “river”; Luo and Niki 2003) that require a solution word in order to resolve a seeming contradiction. The second type were ambiguous sentences that require a reinterpretation of dominant word meaning in order to be understood, such as “The haystack was important because the cloth ripped” that referred to the situation of “parachute jumping” (Mai et al. 2004; Luo et al. 2004a). The third type were the so-called cerebral gymnastics puzzles such as “Unfortunately, Smith and his son met a trafﬁc accident; Smith died on the spot and the boy was badly hurt. They brought the boy to the hospital for he needed an immediate operation. However, the surgeon saw the son and said: ‘Sorry, I cannot perform an operation to my own son.’ How could this occur?” (The answer is “The surgeon is boy’s mother.”) (Luo et al. 2004b). Solving these riddles or puzzles crucially involved the process of restructuring because the solutions almost certainly differ from the solver’s initial way of conceptualizing the problem. For instance, in the “river” riddle, to come up with the correct answer “river” one has to ignore object weight that is the focus of the problem description, and restructure the question in a way that allows a reformulation of the problem in terms of density of objects. Similarly, in the “parachute jumping” riddle and the “mother surgeon” puzzle, one has to change one’s initial understanding that “cloth” refers to something wearable and a “surgeon” must be a man. There are huge number of such kinds of riddles and puzzles, so it is not difﬁcult to get sufﬁcient items that are suitable with regard to cognitive components, complexity, length, and other features. However, the riddles and puzzles that create a need for restructuring are usually difﬁcult. People cannot solve many of these on their own without external help in a short period. Some items will take the thinker several seconds to solve, whereas others will take minutes, hours, or even days. So, it is impossible in a neuroimaging study to simply provide participants with the puzzles and to wait for the moment of insight. In order to obtain a sufﬁcient number of events of insight problem solving in a limited time period one needs to provide solution hints to trigger successful problem solving. The fact that insight phenomena investigated by the riddle-solving approach are externally triggered is a disadvantage, but the advantage of this approach it that it produces multiple insight events

258

J. Luo et al.

with a very accurate onset time. Moreover, using different kinds of reference states, we can ﬂexibly use this approach to test various hypotheses on insight. For example, Weisberg (1995) proposed differentiating between superﬁcial and structural changes in a problem representation. A structural change allows new types of solution to be proposed or in some ways constrains the solution that can be proposed, whereas a superﬁcial change has neither of those effects. To examine the neural network involved in the superﬁcial and structural changes, we asked participants to work on a list of “cerebral gymnastics” puzzles (e.g., the abovementioned mother surgeon puzzle; Luo et al., unpublished results). For each participant a set of puzzles was selected so that he/she understood the puzzles very well but could not solve them. Then, during fMRI scanning, we showed each participant the selected puzzles, followed by three kinds of hints: restructuring hints that should result in a deep structural change of problem representation (for the mother surgeon example, the hint presented is “the surgeon has long hair”); unrelated hints that should induce superﬁcial changes in the problem representation but should not lead to restructuring (e.g., “the surgeon has blue eyes”); and repetition hints that restated the original problem description (e.g., “the surgeon was unable to do the operation”). Participants are likely to obtain the correct solution with the help of repetition hints, but the unrelated hints and restructuring hints usually do not trigger a correct solution. In the experimental session, participants were shown three to ﬁve hints after attempting to solve each puzzle on their own. The hints were presented one by one in a randomized sequence, and participants were not given any information in advance on which hint was the real efﬁcient one. This ensured that participants paid equal attention to each hint. The event-related fMRI results showed that the different types of hints led to activation of different neural networks. Most importantly, in the restructuring hints condition we observed activation in bilateral superior frontal gyrus (BA 8/6), medial frontal gyrus (BA 8) extending to cingulate cortex, and bilateral posterior middle temporal gyrus, suggesting that this network is involved in restructuring. In contrast, the more superﬁcial change in the problem representation induced in the unrelated hints condition was associated with activation in anterior parts of bilateral superior and middle temporal gyrus (BA 22/21), together with frontal activation in superior/medial superior frontal gyrus (BA 8) and in left middle frontal gyrus (BA 9). Another example for how the riddle-solving approach can be used to test speciﬁc hypotheses is the investigation on the role of ACC in insight. Although activation of ACC and of medial prefrontal cortex were consistently observed in brain studies of insight (Luo and Niki 2003; Mai et al. 2004; Luo et al. 2004a,b, 2006), its exact function was unclear. Two observations further revealed the dynamic feature of ACC activation during insight. First, ERP study indicated that ACC activation was present as early as 380 ms after the onset of a restructuring cue (Mai et al. 2004). Given that it takes around 2,000 ms for the participants to fully understand the meaning of a solution cue, the problems were still not completely solved when ACC became active. Second, we examined how ACC activation changed across a long session of solving riddles and found ACC activity decreased as the session progressed (Luo et al. 2004a). This suggests that ACC becomes functionally less important when

Neural Correlates of Insight Phenomena

259

problem solvers start to develop general strategies to deal with a particular type of task, even when most tasks require restructuring. On the basis of these observations, together with the prevalent theory that suggested ACC implements an early warning system (Botvinick et al. 2001) and is engaged when top-down control fails to block the automatic processing of information out of the central processing mechanism, we predicted (1) that the activation of ACC was insensitive to the difﬁculty of solution/hint understanding and (2) that the activation of ACC was sensitive to the variation of implicit regularity underlying the structure of puzzles. To check whether or not ACC was responsive to the difﬁculty of solution/hint understanding, we compared the solution cues that were judged by the problem solver as “understandable, but fairly hard” and those judged as “obvious to understand” (Luo et al. 2004a). The results showed that ACC was equally involved in both types of cues. In contrast to ACC, lateral prefrontal cortex was observed to be responsive to the difﬁculty of processing of the solution cue; this area showed higher activation when the solution cue was difﬁcult to process. A further ERP study by Qiu et al. (2006, 2007) compared three kinds of solutions: hints that conﬁrmed participants’ initial correct thinking; hints that led to a successful restructuring that allowed participants to solve insight problems they could not solve on their own; and hints that did not lead to a successful solution of insight problems. The results showed that, relative to the conﬁrming hint, the other two hints both elicited more negative ERP deﬂections between 250 and 400 ms. The dipole analysis localized the generator of the difference waves within ACC. This observation implies that the activation of ACC is unrelated to the ﬁnding of the correct solution. As long as the hint suggests a new solution path the solvers had not thought about so far, ACC activation increases. In a further recent study (Luo et al., unpublished results), we compared four types of solution cues: correct solutions to comprehensible questions (type 1, e.g., “airconditioned” to “The ofﬁce was cool because the windows were closed.”); correct solutions to ambiguous questions (type 2, e.g., “parachute” to “The haystack was important because the cloth ripped.”), fake solutions to comprehensible questions (type 3, e.g., “knife” to “The dirty clothes were cleaned, because the rotation had been done.” – the solution is washing machine); and fake solutions to fake questions (type 4, e.g., “raining” to “The teacher changed a classroom, because the surface is round.”). The results of 16 participants showed that, relative to correct solutions to comprehensible questions (type 1), not only the true solutions to ambiguous questions (type 2), but also the fake solutions to comprehensible questions or fake questions (type 3 or type 4) evoked lateral and medial prefrontal cortex activation (the territory of activation extended into ACC). This result implied that these areas participated in the processing of an unexpected solution, regardless of whether the solution ﬁnally turned out to be reasonable or not. To examine whether or not the activation of ACC was sensitive to the variation of regularity underlying the structure of puzzles, we compared the neural correlates of solving two kinds of puzzles (Luo et al. 2004b). In condition A, the subjects solved a list of puzzles that were constructed by different principles; whereas in condition B, all of the puzzles were constructed by the same principle. Thus, it was

260

J. Luo et al.

possible for the solvers to allocate some task-general strategy to solve the puzzles in condition B. For condition A, this was relatively difﬁcult to achieve. The results showed that, relative to the resting baseline, both conditions evoked comparable activities in the left lateral prefrontal cortex, but that condition A evoked more ACC activity than condition B. This conﬁrms that ACC is sensitive to the deep structure of problems, showing stronger activation when this structure is variable, and does not permit one to develop a top-down strategy.

3 Integrating Meaning in the Remote Associates Test The RAT approach to investigate the brain basis of insight was developed by Jung-Beeman et al. (2004) and Kounios et al. (2006, 2008). In most studies, participants were presented with three words such as french, car, shoe or boot, summer, ground and were required to generate a solution word that can form a compound word or two-word phrase with each of the three words (the solution words are horn and camp in the abovementioned two examples). According to Bowden et al. (2005), although RAT items are not as complex as classic insight problems, they exhibit three properties of insight problems: (1) Solvers are often misdirected in their solution efforts. For example, in the problem pine, crab, sauce the word pine might direct the initial search of memory toward items such as pine tree or pine cone rather than pineapple. (2) Solvers often cannot report how they overcame an impasse (“It just popped into my head.”). (3) Solvers sometimes have an “Aha!” experience when they achieve solutions. Bowden et al. (2005) used the subjective Aha! experience as the deﬁning criterion for whether an insight had occurred. If an item was solved with the Aha! experience (“You may not be sure how you came up with the answer, but are relatively conﬁdent that it is correct without having to check it.”), then this item was classiﬁed as an insight item; all other items were classiﬁed as noninsight items. The items were successfully preselected so that the ratio of trials for successful insight solving and noninsight solving was comparable in most of the participants (56 vs. 41%), and so were the response times (10.25 vs. 11.28 s) (Jung-Beeman et al. 2004). An advantage of the RAT approach is that problems can be solved in a short time without any external help, and that, therefore, it can be used to investigate internally generated insight. As mentioned earlier, however, it is relatively difﬁcult to determine the onset time of an internally generated insight because participants’ reports are usually delayed ones and the researcher has to go back several hundreds milliseconds to anchor the onset time of the internally generated insight. Jung-Beeman et al. (2004) in their fMRI study chose a point about 2 s prior to each button press with which participants indicated they had found the solution as the onset of insight or noninsight events. Their parallel EEG study showed there was a burst of gamma-band activity associated with the insight solutions (but not

Neural Correlates of Insight Phenomena

261

noninsight solutions) beginning approximately 0.3 s before the button-press solution response at the anterior right temporal electrodes. The RAT approach has been used to test the neurological model of insight of Bowden et al. (2005). This model proposes that insight problem solving originates from the integration of problem elements that are nondominant for the individual. The core assumption is that weak semantic activation in the right hemisphere that is not consciously available is crucial for this integration and for obtaining insights in general. Using visual-hemiﬁeld presentation and subliminal priming, Bowden and Jung-Beeman (1998) found that participants showed greater priming effects (i.e., faster responses to solution target words than to unrelated target words) for solution words presented to the right hemisphere through the left visual ﬁeld than to the left hemisphere through the right visual ﬁeld. Consistent with these results, an fMRI study revealed increased activation in the right anterior superior temporal gyrus for insight relative to noninsight solving of RAT, and scalp EEG recordings revealed a sudden burst of high-frequency (gamma-band) neural activity in the same region just before insight (Jung-Beeman et al. 2004).

4 Altering Meaning Through Perceptual Reorganization: Chunk Decomposition Chunk decomposition refers to the decomposition of familiar patterns into their component elements so that they can be regrouped in another meaningful manner. Such a regrouping is required in some insight problems because during problem encoding problem elements become automatically grouped into familiar chunks. For instance, it is easy to decompose the loose perceptual chunk that forms the word “BIT” into its component letters and remove one letter (i.e., the “B”) away to form the word “IT,” whereas it is much more difﬁcult to transform “BIT” to form the word “PIT” by removing the lower part of the letter “B.” Knoblich et al. (1999) proposed that the need to decompose perceptual chunks is an important difﬁculty source characterizing many insight problems. In their study, problem solvers were given a false matchsticks arithmetic statement, written using roman numerals (e.g., I, II, and IV), operations (+ and −) and an equal sign (=) and were required to transform the statement into a true equation by moving only one stick from one position to another. It was easy for the participants to transform the equation VI = VII + I to VII = VI + I, whereas it was difﬁcult for them to transform the equation XI = III + III to VI = III + III, because the chunk tightness of “X” is much more higher than that of “VII.” Unfortunately, the matchsticks arithmetic task is not appropriate for neuroimaging studies, because the task domain does not provide a large enough variety of problems. To overcome this problem, we developed a new chunk decomposition task using Chinese characters as materials (Luo et al. 2006). As a logographic language system, Chinese characters are ideal examples of perceptual chunks (Perfetti et al. 2005; Tan et al. 2001, 2005a,b; Fu et al. 2002; Siok et al. 2004). Chinese characters

262

J. Luo et al.

Fig. 1 Illustration of construction of Chinese character and chunk decomposition task using Chinese characters. TCD tight chunk decomposition, LCD loose chunk decomposition

are composed of radicals, which in turn are composed of strokes (Fig. 1). Strokes are the simplest and most basic components of a Chinese character. Usually, strokes do not carry meaning in themselves. In contrast, radicals convey information about the meaning and pronunciation of the character. The radicals usually consist of several strokes and can be thought of as subchunks of a character. Thus, radicals are meaningful chunks, whereas strokes are not meaningful in isolation. According to the chunk decomposition hypothesis it should be much easier to separate a character by its radicals than to separate a character by its strokes, because particular strokes are tightly embedded in a perceptual chunk. In other words, the decomposition of characters into strokes should require a speciﬁc process that breaks the tight bond among strokes created by the perceptual chunk. Participants were given tasks that always involved two valid characters, one on the left side of the display and the other on the right. They were asked remove a part of the right character and add it to the left character so that two new valid

Neural Correlates of Insight Phenomena

263

characters resulted after the move (Fig. 1). There were two conditions. In the tight chunk decomposition (TCD) condition, the problem could be solved only if participants decomposed the character into separate strokes and moved some of the resulting strokes from the right to the left character. In the loose chunk decomposition (LCD) condition, it was sufﬁcient to decompose the character into separate radicals and to move one of the resulting radicals to the left character. Pilot studies showed that problems requiring the decomposition of a tight chunk were much more difﬁcult than problems requiring the decomposition of a loose chunk. The former were often not solved or took several minutes to solve (frequently accompanying “Aha!” reaction), whereas the latter were usually solved within 2–4 s. The large differences in problem difﬁculty make it generally difﬁcult to address the brain processes related to problem solving. Therefore, we provided a hint to catalyze the puzzle-solving process, after the problem solvers had failed to solve the puzzle by themselves and had got into an impasse state. During the hint stage, the to-be-moved part of the right-side character was highlighted in red (Fig. 1). This method enabled us to produce a large enough number of chunk decomposition trials in the TCD condition. Contrasting the processing of the hint between the TCD condition and the LCD condition (where participants had already solved the problem on their own and the presentation of the hint just conﬁrmed their previous solution), we were able to identify the brain areas contributing to chunk decomposition. Our results showed that the early visual cortex was less active in the TCD condition than in the LCD condition, whereas the higher visual cortex was more active in the TCD condition. These results suggest the following interpretation. The individual features/components contained in a chunk are processed in the early visual cortex (Uchida et al. 1999). During normal chunk perception, the processing of these individual features/components will be automatically grouped to form a holistic chunk. However, chunk decomposition requires that these individual chunk features be rearranged into a different perceptual chunk. Thus, processing of individual features is suppressed as reﬂected by the inhibition in early visual cortex, while the grouping is rearranged as reﬂected by the higher activation in higher visual cortex. A more general implication of these results is that perceptual processes seem to be involved in at least some forms of restructuring. In this sense the results seem to support the Gestalt psychologists’ original claim that restructuring shares similarities with perceptual reinterpretation.

5 Insight and Strategy Change Research by Haider and Rose (2007) suggests that insight cannot only occur when people try to solve a particular difﬁcult problem. They claim that restructuring can also occur when people ﬁnd new strategies to deal with routine problems. In order to address this hypothesis they use standard implicit learning tasks, such as the serial reaction time task (Nissen and Bullemer 1987) or the number reduction task (NRT; Haider and Frensch 2005). Haider and Rose (2007) start with the observation that

264

J. Luo et al.

10–70% of the participants are able to verbally describe the deterministic regularity built into the task when asked to do so in a postexperimental interview. Although researchers in the ﬁeld of implicit learning are not interested in these participants and exclude them from further data analyses, Haider and colleagues point out that the process of spontaneously arising explicit knowledge during an incidental learning situation strongly resembles the process of ﬁnding the solution for an insight problem. Accordingly, they used modiﬁed implicit learning tasks as a paradigm to study insight (Wagner et al. 2004; Lang et al. 2006). In the standard version of the NRT, participants receive a string of six digits one by one on a computer screen. The string always consists of the same three digits, “1,” “4,” and “9” arranged in a different order for each trial (e.g., “9 9 9 1 4 1”). The participants’ task is to compute the ﬁnal response for the entire string. To do so they are instructed to process the digit string pairwise from left to right by applying one of two rules (Fig. 2). The ﬁrst rule states that two identical digits in a pair yield the same digit (same-rule). The second rule states that the result for two nonidentical digits is the remaining third digit (different-rule). Participants are explicitly told these two rules. In the example “9 9 9 1 4 1,” participants ﬁrst receive the ﬁrst two digits of the digit string “9 9.” These two digits are identical, and therefore comply with the same-rule, resulting in “9.” After the response “9” has been entered, the third digit “9” occurs on the screen. Participants compare their response with the new digit “9.” Again the same-rule generates “9.” Then the fourth digit “1” is presented. Participants compare their last response “9” with the fourth digit. According to the different-rule, this comparison yields “4.” The ﬁfth digit “4” again yields “4” (samerule) and the sixth digit “1” yields “9” (different-rule). In sum, the stimulus string “9 9 9 1 4 1” yields the response string “9 9 4 4 9” according to the two rules. Participants are especially instructed to conﬁrm the last result “9” as the ﬁnal result of the entire string by pressing the “Enter” key.

Fig. 2 Demonstration of the number reduction task

Neural Correlates of Insight Phenomena

265

In the experiments of Haider and colleagues people calculate the solutions for a large number of such digit strings and, of course, get faster as they go. In this respect the task is like any other implicit learning task. However, there is one crucial difference. The response string has a regular structure so that for any given stimulus sequence, the fourth response is always identical to the third response, and the ﬁfth response is always identical to the second response. Put differently, participants’ responses 4 and 5 are a mirror image of responses 2 and 3. However, this regularity within response strings is neither communicated to participants, nor are they asked to search for regularity hidden in the task. The results show that participants who have discovered the task regularity are able to speed up substantially. This big improvement results in discontinuities in the learning curve that cannot be explained by the principle of gradual learning. Some participants start to enter responses 4 and 5 in very quick succession because these can be directly derived from responses 3 and 2, ignoring the stimulus. Others skip responses 3, 4, and 5 altogether because responses 2 and 5 are always identical. The sudden change in strategy in this incidental learning situation that goes hand in hand with explicit knowledge about the regularity in the response structure meets the condition of restructuring very well. The reason is the strategy change, which enables the participants to deal with this (tedious) task in an unexpectedly simple way, brought about when the participants’ initial understanding of the nature of the cognitive task is fundamentally changed. In addition, participants achieve the strategy change on their own without any external trigger. Thus, this approach is appropriate for investigating internally generated restructurings. In contrast to the previously discussed paradigms that focus on the moment of insight, the implicit learning approach allows one to more closely look at the genesis of a single “insight” (strategy change). As a consequence this approach is not appropriate to study which areas of the brain are active at the moment of insight. However, it enables one to dynamically track brain activations that prepare a strategy change, that is, the transition from getting more effective without knowing why to explicitly using more effective strategies.

6 Conclusion Although the cognitive processes underlying insight have been studied for many years (Sternberg and Davidson 1995), the study of the brain basis of insight and restructuring has only recently begun. In this chapter, we reviewed four types of experimental approaches that have been used so far. Although the validity of these experimental designs seems not good enough (yet) to fully demystify the wellknown legends of important discoveries (like Archimedes’s solving of the golden crown problem or Kekul´e’s discovery of the ring structure of benzene), these paradigms do address different important aspects of insight. Thus, we are conﬁdent that future brain research will help us to understand why some of our best ideas seem not to result from hard work but seem to come to us out of the blue.

266

J. Luo et al.

Acknowledgements This work was supported by NSFC (30770708) and KSCX2-YW-R-28 to J.L.

References Botvinick MM, Braver TS, Barch DM, Carter CS, Cohen JD (2001) Conﬂict monitoring and cognitive control. Psychol Rev 108:624–652 Bowden EM, Jung-Beeman M (1998) Getting the right idea: semantic activation in the right hemisphere may help solve insight problems. Psychol Sci 6:435–440 Bowden EM, Jung-Beeman M, Fleck J, Kounios J (2005) New approaches to demystifying insight. Trends Cogn Sci 9:322–328 Carter CS, Macdonald AM, Botvinick M, Ross LL, Stenger VA, Noll D, Cohen JD (2000) Parsing executive processes: strategic vs. evaluative functions of the anterior cingulate cortex. Proc Natl Acad Sci U S A 97:1944–1948 Davidson JE (2003) Insights about insightful problem solving. In: Davidson JE, Sternberg RJ (eds) The psychology of problem solving. Cambridge University Press, Cambridge, pp 149–175 Duncker K (1945). On problem-solving. Psychol Monogr 58:1–110 Fu S, Chen Y, Smith S, Iversen S, Matthews PM (2002) Effects of word form on brain processing of written Chinese. Neuroimage 17:1538–1548 Goel V, Buchel C, Frith C, Dolan RJ (2000) Dissociation of mechanisms underlying syllogistic reasoning. Neuroimage 12:504–514 Grant E, Spivey M (2003) Eye movements and problem solving: guiding attention guides thought. Psychol Sci 14:462–466 Haider H, Frensch PA (2005) The generation of conscious awareness in an incidental learning situation. Psychol Res 69:399–411 Haider H, Rose M (2007) How to investigate insight: a proposal. Methods 42:49–57 Jung-Beeman M, Bowden EM, Haberman J, Frymiare JL, Arambel-Liu S, Greenblatt R, Reber PJ, Kounios J (2004). Neural activity observed in people solving verbal problems with insight. PLoS Biol 2:500–510 Kershaw TC, Ohlsson S (2004). Multiple causes of difﬁculty in insight: the case of the nine-dot problem. J Exp Psychol Learn Mem Cogn 30:3–13 Knoblich G, Ohlsson S, Haider H, Rhenius D (1999) Constraint relaxation and chunk decomposition in insight problem solving. J Exp Psychol Learn Mem Cogn 25:1534–1556 Knoblich G, Ohlsson S, Raney GE (2001) An eye movement study of insight problem solving. Mem Cognit 29:1000–1009 ¨ Knoblich G, Ollinger M, Spivey M (2005) Tracking the eyes to obtain insight into insight problem solving. In: Underwood G (ed) Cognitive processes in eye guidance. Oxford University Press, Oxford Kounios J, Frymiare JL, Bowden EM, Fleck JI, Subramaniam K, Parrish TB, Jung-Beeman M (2006) The prepared mind: neural activity prior to problem presentation predicts subsequent solution by sudden insight. Psychol Sci 17:882–890 Kounios J, Fleck JI, Green DL, Payne L, Stevenson JL, Bowden EM, Jung-Beeman M (2008) The origins of insight in resting-state brain activity. Neuropsychologia 46:281–291 K¨ohler W (1921). Intelligenzpr¨ufungen am Menschenaffen. Springer, Berlin Heidelberg New York Lang S, Kanngieser N, Jaskowski P, Haider H, Rose M, Verleger R (2006) Precursors of insight in event-related brain potentials. J Cogn Neurosci 18:2152–2166 Luo J, Knoblich G (2007) Studying insight problem solving with neuroscientiﬁc methods. Methods 42:77–86 Luo J, Niki K (2003) Function of hippocampus in “insight” of problem solving. Hippocampus 13:316–323

Neural Correlates of Insight Phenomena

267

Luo J, Niki K, Phillips S (2004a) Neural correlates of the ‘Aha! reaction’. Neuroreport 15: 2013–2017 Luo J, Niki K, Phillips S (2004b) The function of anterior cingulate cortex (ACC) in insightful puzzle solving: ACC activated less when the structure of the puzzle was known. J Psychol Chin Soc 5:195–213 Luo J, Niki K, Knoblich G (2006). Perceptual contributions to problem solving: chunk decomposition of Chinese characters. Brain Res Bull 70:430–443 MacDonald AW III, Cohen JD, Stenger VA, Carter CS (2000) Dissociating the role of the dorsolateral prefrontal and anterior cingulate cortex in cognitive control. Science 288:1835–1838 Mai XQ, Luo J, Wu JH, Luo YJ (2004) “Aha!” effects in a guessing riddle task: an event-related potential study. Hum Brain Mapp 22:261–270 Maier NRF (1930) Reasoning in humans: I. On direction. J Comp Psychol 10:115–143 Nissen MJ, Bullemer P (1987) Attentional requirements of learning: evidence from performance measures. Cogn Psychol 19:1–32 Ohlsson S (1992) Information-processing explanations of insight and related phenomena. In: Gilhooley KJ (ed) Advances in the psychology of thinking. Harvester-Wheatsheaf, London, pp 1–44 Ormerod TC, MacGregor JN, Chronicle EP (2002) Dynamics and constraints in insight problem solving. J Exp Psychol Learn Mem Cogn 28:791–799 Perfetti CA, Liu Y, Tan LH (2005) The lexical constituency model: some implications of research on Chinese for general theories of reading. Psychol Rev 112:43–59 Qiu J, Li H, Luo Y, Chen A, Zhang F, Zhang J, Yang J, Zhang Q (2006) Brain mechanism of cognitive conﬂict in a guessing Chinese logogriph task. Neuroreport 17:679–682 Qiu J, Zhang Q, Li H, Luo Y, Yin Q, Chen A, Yuan H (2007). The event-related potential effects of cognitive conﬂict in a Chinese character-generation task. Neuroreport 18:881–886 Scheerer M (1963). Problem solving. Sci Am 208:118–128 Siok WT, Perfetti CA, Jin Z, Tan LH (2004) Biological abnormality of impaired reading is constrained by culture. Nature 431:71–76 Sternberg RJ, Davidson JE (1995) The nature of insight. MIT Press, Cambridge Tan LH, Liu HL, Perfetti CA, Spinks JA, Fox PT, Gao JH (2001) The neural system underlying Chinese logograph reading. Neuroimage 13:836–846 Tan LH, Laird AR, Li K, Fox PT (2005a) Neuroanatomical correlates of phonological processing of Chinese characters and alphabetic words: a meta-analysis. Hum Brain Mapp 25:83–91 Tan LH, Spinks JA, Eden GF, Perfetti CA, Siok WT (2005b) Reading depends on writing, in Chinese. Proc Natl Acad Sci U S A 102:8781–8785 Uchida I, Kikyo H, Nakajima K, Konishi S, Sekihara K, Miyashita Y (1999) Activation of lateral extrastriate areas during orthographic processing of Japanese characters studied with fMRI. Neuroimage 9:208–215 Wagner U, Gais S, Haider H, Verleger R, Born J (2004) Sleep inspires insight. Nature 427:352–355 Weisberg RW (1995) Prolegomena to theories of insight in problem solving: a taxonomy of problems. In: Sternberg RJ, Davidson JE (eds) The nature of insight. MIT Press, Cambridge, pp 157–196 Weisberg RW, Alba JW (1981) An examination of the role of “ﬁxation” in the solution of several “insight” problems. J Exp Psychol Gen 110:169–192 Wertheimer M (1925). Drei Abhandlungen zur Gestalttheorie. Philosophischen Akademie, Erlangen Wertheimer M (1959). Productive thinking. Harper, New York

Brain-Based Mechanisms Underlying Causal Reasoning Jonathan Fugelsang(¬) and Kevin N. Dunbar

Abstract Since well before the time of contemporary psychological and neuroscientiﬁc research, philosophers and scientists have been fascinated with the concept of causality. Recent advances of neuroscientiﬁc techniques, speciﬁcally, neuroimaging using functional MRI, have allowed scientists to probe the brain in order to uncover the mechanisms underlying people’s conceptions of causality. In this chapter, we provide an overview of a portion of this recent work, speciﬁcally as it pertains to the nature of how people interpret and reason about causality.

1 Introduction One of the most fundamental attributes of the human mind is its ability to perceive and interpret causal relations apparent in the environment. Indeed, the detection of causal relations is a fundamental ability underlying an individual’s success in the dynamic world in which we live. Since before the time of Aristotle (Fig. 1), philosophers and scientists have attempted to provide an account of how we know that one event causes another. Contemporary theories of causation range from accounts that envisage a common mechanism for understanding causality, to accounts that treat understanding causality, and its resultant representations, as distinct across domains (Sperber et al. 1995). Researchers have recently begun to study the neural underpinnings of several tasks that tap different aspects of causal thinking. Speciﬁcally, using a variety of neuroimaging techniques, researchers have examined the perception of causality

J. Fugelsang Department of Psychology, University of Waterloo, 200 University Avenue West, Waterloo, ON, Canada N2L 3G1 [email protected] E. Kraft et al. (eds.) Neural Correlates of Thinking, c Springer-Verlag Berlin Heidelberg 2009

269

270

J. Fugelsang, K.N. Dunbar

Fig. 1 Aristotle (as depicted in this photograph of a painting by Raphael with his mentor Plato) is often credited as the ﬁrst academic to formally discuss the concept of causality. He deﬁned four distinct types of cause: the material, formal, efﬁcient, and ﬁnal types

(Blakemore et al. 2001; Fonlupt 2003; Fugelsang et al. 2005), learning of causal associations (Turner et al. 2004), theory and data interactions in causal thinking (Fugelsang and Dunbar 2005), the representation of stored causal association in semantic memory (Satpute et al. 2005), and the processing of causal inferences in text (Mason and Just 2004). In this chapter, we will provide a brief review of this literature and provide some insights into the possible mechanisms that underlie causal thought. In addition, we will provide some thoughts regarding the degree to which we think neuroimaging can inform the development of theories of causality. We will group our discussions of the literature in terms of (1) studies examining the perception of causality while viewing dynamic displays, (2) learning and reasoning with statistical associations, and (3) accessing stored representation of causal knowledge in semantic memory.

Brain-Based Mechanisms Underlying Causal Reasoning

271

2 Perceptual Causality Within the physical domain, interactions between moving stimuli, such as collisions, are often reported as involving causal relationships. This can occur even with very simple stimuli such as two moving balls, represented by light patches, on a computer screen. For example, as depicted in the top panel of Fig. 2, if ball A moves toward ball B, stops when it contacts ball B, and B then moves away, the motion of ball B is reported by the majority of observers to have been caused by ball A. If, however, there is a temporal gap (middle panel) or a spatial gap (bottom panel), observers report the relationship as noncausal. This is despite the fact that no inherent causal interaction has occurred, just the simple kinematics as described above. The presence of a small gap or delay (an incontiguity) between the two stimulus movements reduces the likelihood with which stimulus interactions are rated as causal. This collision event has been termed the “launching effect” and is the best-known example of what is called perceptual causality (Michotte 1963). A number of studies have examined the underlying neural processes associated with perceptual causality. The ﬁrst reported study to this effect was reported by Blakemore et al. (2001); see also Fonlupt (2003) for an additional analysis of those data. They contrasted causal events where a blue ball collided with a red ball which subsequently moved, with noncausal events where a blue ball either moved across the screen and passed under a stationary red ball or rolled across the screen and

CAUSAL Event

A

B

A

B

A

B

A

B

A

B

No Temporal Delay or Spatial Gap

NON-CAUSAL Event (Temporal Delay)

A

B

A

B

Temporal Delay

NON-CAUSAL Event (Spatial Gap)

A

B

A

B

Spatial Gap

Fig. 2 Causal, temporal delay, and spatial gap events. The three panels depict the motion of a ball A toward a second ball B, and the subsequent motion of ball B

272

J. Fugelsang, K.N. Dunbar

changed color to red after 1 s. They found signiﬁcant activations in V5, medial, and superior temporal lobes bilaterally, as well as regions in the left superior temporal and intraparietal sulcus. As these regions are strongly implicated in tasks involving complex visual analyses, they argued that the visual system is speciﬁcally designed to recover the causal structure of dynamic visual events in the environment. In a related study, Fugelsang et al. (2005) examined the extent to which causal stimuli differentially recruit neural regions associated with spatial and temporal contiguity when those cues to causality are manipulated. Consistent with Blakemore et al. (2001) and Fonlupt (2003), we found similar activations in the temporal lobes when contrasting the causal stimuli to the stimuli with a spatial gap. When causal stimuli were contrasted with stimuli containing a temporal gap, however, activations were predominantly in the frontal and parietal cortices. Importantly, when causal stimuli were contrasted with both noncausal stimuli (those containing spatial and temporal gaps), activations were predominantly found in the frontal and parietal cortices in the right hemisphere (Fig. 3). The frontal activity found is consistent with the frontal activity found by Fonlupt (2003). The frontal activations may be the product of a variety of processes. Perhaps the most parsimonious explanation is that causal stimuli may recruit additional higherorder executive/attentional resources above and beyond those afforded by the visual system. The preferential recruitment of regions in the prefrontal cortices for causal stimuli suggests that such stimuli may capture visual attention (de Fockert et al. 2004) and result in more attentional resources devoted to such stimuli (Smith and

Mean Signal Change

Inferior Parietal Lobule 0.2 0.16 0.12 0.08 0.04 0 Causal

Non-Causal

Mean Signal Change

Middle frontal Gyrus 0.04 0 −0.04 −0.08 −0.12 −0.16

Causal

Non-Causal

Fig. 3 Signiﬁcant brain activations and region of interest analyses for causal collision events relative to noncausal events in Fugelsang et al. (2005)

Brain-Based Mechanisms Underlying Causal Reasoning

273

Jonides 1998). Indeed, this allocation of attentional resources and subsequent recruitment of prefrontal cortex may be one of the hallmarks of causality. We will see this throughout the studies reviewed in this chapter.

3 Learning and Reasoning with Statistical Associations Many causal relations need to be learned through the association of events and outcomes. For example, an individual may learn that he or she has an allergy to peanuts on the basis of the association of allergic reactions arising when foods are eaten that contain peanut products, and not arising when foods are eaten that do not contain peanut products. Several papers have recently been published that look at various aspects of this associative learning process (Corlett et al. 2004; Fletcher et al. 2001; Turner et al. 2004). One of the key components of associative models is that learning depends on surprise. For example, surprising outcomes are thought to enhance attention to stimuli, and thus promote learning. Fletcher et al. (2001) found support for this hypothesis in that initially novel and surprising stimuli produced maximal activation in dorsolateral prefrontal cortex when participants are learning associations. This heightened activation attenuated through learning, but was re-evoked when surprise violations of the learned association were present. These data are consistent with recent work on theory and data interaction in complex causal reasoning conducted by Fugelsang and Dunbar (2005). We presented participants with a task requiring them to interpret data relative to plausible and implausible causal theories. The plausibility of the causal theories was manipulated by presenting participants with a brief introductory statement that depicted a causal theory that contained either a plausible or implausible causal mechanism. Data were then provided to participants in a trial-by-trial format where they viewed multiple trials of data for each causal theory provided. These data were presented in combinations of a cause (a red pill or a blue pill) and an effect (happiness or neutral outcome) co-occurring. Figure 4 presents a graphical depiction of these four event types. Under some conditions the red pill and happiness covaried strongly, under other conditions the red pill and happiness covaried weakly. Importantly, the trials cumulatively presented data that were either consistent with the initial theory, or inconsistent with the theory. We analyzed the imaging data in two stages. We were ﬁrst interested in looking at the degree to which different regions of the brain would be selectively responsive to reasoning with scenarios that contained plausible as opposed to implausible causal mechanisms. Second, we examined the degree to which the consistency of the relationship between the plausibility of the causal mechanism and the data modulated the recruitment of dissociable neural regions. Considering ﬁrst the effect of mechanism plausibility, like our work on perceptual causality, regions of the right superior frontal gyrus were more activated when subjects were reasoning about candidates that contained plausible causal mechanisms than candidates that contained implausible causal mechanisms. We interpreted these ﬁndings to suggest

274

J. Fugelsang, K.N. Dunbar

Fig. 4 Example stimuli representing the four possible combinations of the candidate cause (red pill versus blue pill) and effect (happiness versus neutral emotion) used by Fugelsang and Dunbar (2005)

that plausible causal mechanisms, like collisions that conform to expectations in the Michotte task, capture attention and thus are subject to execute processing (Curtis and D’Esposito 2003; Smith and Jonides 1999). Concerning our second level of analyses, we found that the consistency between theory and data inﬂuenced the degree to which disparate neural tissue associated with learning or conﬂict monitoring/error detection were recruited. Speciﬁcally, as can be seen in Fig. 5, when data were consistent with the theory (regardless of its plausibility), activations were found in the caudate and parahippocampal gyrus. When the statistical data were inconsistent with a theory, however, the anterior cingulate cortex and precuneus were selectively recruited. A further important ﬁnding emerged when we looked at the effects of data consistency for plausible and implausible theory separately. Speciﬁcally, when participants viewed data that were inconsistent with a plausible theory, further activations on the left prefrontal cortex were also found to occur in concert with the activation in anterior cingulate cortex and precuneus. There are several possible interpretations of these ﬁndings. Our preferred interpretation is that participants likely perceived data as error when they were inconsistent with a plausible causal theory. In addition, the selective dorsolateral prefrontal recruitment in concert with the anterior cingulate cortex in this condition may be the result of the active inhibition of the attentional processes associated with the task. Recently, Goel and Dolan (2003) found preferential recruitment of the dorsolateral prefrontal cortex in a deductive reasoning task when beliefs and

Brain-Based Mechanisms Underlying Causal Reasoning

Theory & Data Consistent

275

Theory & Data Inconsistent

Fig. 5 Unique task-associated brain activations occurring when viewing data inconsistent or consistent with a causal theory

logic were in conﬂict and required the inhibition of a response. Another possible interpretation, one that would be consistent with the ﬁndings of Fletcher et al. (2001), is that data inconsistent with a causal theory are surprising, and thus preferentially recruit attentional resources.

4 Accessing Stored Causal Relations As we discussed in the previous sections, there is considerable evidence that (1) the visual system is specially tuned to extract some types of causal relations present in the environment and (2) other types of causal relationships are likely learned through observation. An important avenue of research has been examining how these stored associations are represented and accessed in semantic memory. Are causal associations, like wind and erosion, stored and processed the same as noncausal associations, like bread and butter? Satpute et al. (2005) examined this issue by presenting participants with causal word pairs (along with additional control stimuli), and requiring them to make either associative or causal judgments on these word pairs. Their basic hypothesis was that assessing the causal nature of relations required additional processing than simply judging mere associations. Speciﬁcally, they argued that judging causality likely requires a process called dynamic role binding (Hummel and Holyoak 2003). Evaluating causal relations may require forming and holding an explicit representation of the speciﬁc events bound to the roles of cause and effect. For example, in the wind/erosion example presented previously, a participant needs to evaluate the speciﬁc cause and effect roles of both items in order to make the causal judgment. They found that causal judgments, in contrast to associative judgments, preferentially recruited regions in the left dorsolateral prefrontal cortex and the precuneus.

276

J. Fugelsang, K.N. Dunbar

These activations are consistent with the idea that assessing causality requires additional neurocognitive attentional resources in order to evaluate the causal roles of the word pair in working memory. Interestingly, there are cases in which causal relatedness results in a reduction of working memory resources, and subsequently a reduction in prefrontal cortex activation. For example, pairs of sentences like “Joey’s big brother punched him again and again” and “The next day his body was covered in bruises” are typically read faster, and recruit less prefrontal cortex than sentences such as “Joey went to a neighbor’s house to play” and “The next day his body was covered in bruises” (Mason and Just 2004). Presumably, causal relations in such sentence pairs help bind the sentences together and thus require less generation of inferences. Here, the presence of a clear causal relationship serves to reduce the number of inferences required on the part of the reader.

5 Single or Multiple Causal Representations The ﬁnal area of research we wish to touch on concerns the investigation of single versus multiple representations of causality. Throughout the different sections of this chapter, we have covered a variety of tasks that all involve causal processing in some form or another. Do all of these disparate tasks invoke the same underlying processes when judging causality? There has been considerable debate in the literature regarding the extent to which judgments of causality are the product of single or multiple underlying processes (Scholl and Tremoulet 2000; Schlottmann 2000). For example, the possibility that some events can be directly perceived as causal (e.g., using the Michotte paradigm as discussed previously) suggests that there may be multiple representations or processes that support judgments of causality. Does perceptual causality represent a unique form of human causal processes that can be dissociated from that based on more associative processes? Take ﬁrst the concept of perceptual causality. The ﬁndings that the perception of causality appears very early in human life (Leslie and Keeble 1987) and is culturally invariant (Morris and Peng 1994) have been taken to suggest that the visual system may be specially tuned to recover physical causal structure from the environment. This can be contrasted with casual inference, which demands the learning of causal associations based on covariation experience. This ability to learn causal associations develops somewhat later in life (Gopnik et al. 2001). A series of experiments lead by (Roser et al. 2005) investigated whether causal perception (using the standard Michotte paradigm) could be dissociated from a task requiring causal inference. An obvious difﬁculty arises with determining the extent to which causal perception and causal inference are subserved by the same or different underlying processes using traditional behavioral measures. To date, the majority of research testing for the existence of unique processes supporting causal perception and inference has come from observers’ subjective reports which are highly subjective and open to postperceptual interpretation in the “normal” brain. This difﬁculty, however, can be overcome by using split-brain patients who have undergone surgery to isolate the two cerebral

Brain-Based Mechanisms Underlying Causal Reasoning

277

hemispheres. By analyzing the degree to which each of the isolated cerebral hemispheres processes causality, we can determine the extent to which perceptual and inferential processes involved in understanding causality can be dissociated. Perhaps the most obvious functional hemispheric asymmetry in humans is that of linguistic versus visual-spatial processing. For example, decades of work with patients and countless functional imaging studies have shown that the right hemisphere possesses an advantage for tasks that require visuospatial processing (Corballis 2003; Corballis et al. 2002), whereas the left hemisphere processes an advantage for linguistic processing (Milner 1962). Taking this as our starting point, we predicted that the right hemisphere would exhibit an advantage for perceptual causality, whereas the left hemisphere would exhibit an advantage for inferential causality. Two patients who underwent callosotomy surgery were presented with causal collision events using the standard Michotte paradigm, and a task requiring causal inference which we adapted from Gopnik et al. (2001). The inference task consisted of a short series of four stimulus interactions wherein the participants simply had to judge which of two “switches” (green or red) caused a “lightbox” to turn on. The data were consistent with our hypotheses in that the ability to draw causal inference and the ability to judge causal relationships during the standard Michotte paradigm were governed by different hemispheres of the divided brain (Fig. 6). Speciﬁcally, the right hemispheres were signiﬁcantly more sensitive to the causal perception task than the inference task, whereas the left hemispheres were signiﬁcantly more sensitive to the inference task than the perception task. These data were taken to support

Causal Perception

Causal Inference 1

Percentage

Percentage

1

0.5

0

0.5

0 Right Hemisphere Left Hemisphere

Right Hemisphere Left Hemisphere

Fig. 6 Causal perception and inference task and the data observed by Roser et al. (2005). The inference tasks consisted of the sequential presentation of four stimulus interactions between a red and green switch and a lightbox. Refer to Roser et al. (2005) for a complete description of the task

278

J. Fugelsang, K.N. Dunbar

the hypothesis that causation is a nonunitary construct, and that causal perception and inference can be processed independently. We have recently extended this work to examine the degree to which dissociable perception and inference mechanisms are invoked in the “normal” brain using functional MRI. To do this, participants were asked to respond to a simple collision similar to those used in our previous work (Fugelsang et al. 2005). Importantly, however, we also manipulated the cover story which participants used to evaluate the collision events. For half the trials participants were told to “. . . imagine that the two circular objects are billiard balls.” For the other half of the trials, participants were told to “. . . imagine that the two circular objects are positively charged particles that repel each other when they come close to being in contact with each other.” It was the intention that this manipulation would change the task from a perceptual to an inferential task in that the requirement to imagine the objects as “positively charged particles” requires one to infer basic characteristics about the objects that may be in conﬂict with the perceptual experience. Preliminary analyses of these data have revealed a pattern consistent with that observed with the experiments with split-brain patients. Speciﬁcally, when making judgments on the stimuli when they were to be thought of as billiard balls, regions in the right superior frontal and inferior parietal cortices were recruited. In contrast, when participants were making judgment on stimuli when they were to be thought of as positively charged particles, homologous regions in the left frontal and parietal cortices were recruited in concert with those in the right hemisphere. These data are taken to support a multidimensional interpretation of causality that involves an interplay between basic perception and/or inference depending on the nature of the task.

6 Conclusions In the preceding sections we have outlined a number of research programs that have recently contributed to our understanding of the nature of human casual thinking. As is evident from the diverse ﬁelds of research, ﬁnding a single region in the brain that uniquely represents causal thinking in humans is likely an unrealistic goal. This may speak more to the diversity of the research areas and the kinds of questions being asked than to the nature of causality. Although great progress has been made, many questions remain to be answered which will beneﬁt from creative functional imaging experiments.

References Blakemore S, Fonlupt P, Pachot-Clouard M, Darmon C, Boyer P, Meltzoff A, Segebarth C, Decety J (2001) How the brain perceives causality: an event-related fMRI study. Neuroreport 12:3741–3746 Corballis PM (2003) Visuospatial processing and the right-hemisphere interpreter. Brain Cognit 53:171–176

Brain-Based Mechanisms Underlying Causal Reasoning

279

Corballis PM, Funnell MG, Gazzaniga MS (2002) Hemispheric asymmetries for simple visual judgments in the split brain. Neuropsychologia 40:401–410 Corlett PR, Aitken MR, Dickinson A, Shanks DR, Honey GD, Honey RA, Robbins TW, Bullmore ET, Fletcher PC (2004) Prediction error during retrospective revaluation of causal associations in humans: fMRI evidence in favor of an associative model of learning. Neuron 44:877–888 Curtis CE, D’Esposito M (2003) Persistent activity in the prefrontal cortex during working memory. Trends Cogn Sci 7:415–423 de Fockert J, Rees G, Frith C, Lavie N (2004) Neural correlates of attentional capture in visual search. J Cogn Neurosci 751–759 Fletcher PC, Anderson JM, Shanks DR, Honey R, Carpenter TA, Donovan T, Papadakis N, Bullmore ET (2001) Responses of human frontal cortex to surprising events are predicted by formal associative learning theory. Nat Neurosci 4:1043–1048 Fonlupt P (2003) Perception and judgment of physical causality involve different brain structures. Cogn Brain Res 17:248–254 Fugelsang J, Dunbar K (2005) Brain-based mechanisms underlying complex causal thinking. Neuropsychologia 43:1204–1213 Fugelsang J, Roser M, Corballis P, Gazzaniga M, Dunbar K (2005) Brain mechanisms underlying perceptual causality. Cogn Brain Res 24:41–47 Goel V, Dolan RJ (2003) Explaining modulation of reasoning by belief. Cognition 87:B11–B22 Gopnik A, Sobel DM, Schulz LE, Glymour C (2001) Causal learning mechanisms in very young children: two-, three-, and four-year-olds infer causal relations from patterns of variation and covariation. Dev Psychol 37:620–629 Hummel JE, Holyoak KJ (2003) A symbolic-connectionist theory of relational inference and generalization. Psychol Rev 110:220–264 Leslie AM, Keeble S (1987) Do six-month old infants perceive causality? Cognition 25:265–288 Mason RA, Just MA (2004) How the brain processes causal inferences in text. Psychol Sci 15:1–7 Michotte A (1963) The perception of causality. Basic Books, New York Milner B (1962) Laterality effects in audition. In Mountcastle VB (ed) Interhemispheric relations and cerebral dominance. Johns Hopkins, Baltimore, pp 177–198 Morris NW, Peng K (1994) Culture and cause: American and Chinese attributions for social and physical events. J Pers Soc Psychol 67:949–971 Roser M, Fugelsang J, Dunbar K, Corballis P, Gazzaniga M (2005) Dissociating processes supporting causal perception and causal inference in the brain. Neuropsychology 19:591–602 Satpute AB, Fenker DB, Waldmann MR, Tabibnia G, Holyoak KJ, Lieberman MD (2005) An fMRI study of causal judgments. Eur J Neurosci 22:1233–1238 Schlottmann A (2000) Is perception of causality modular? Trends Cogn Sci 4:441–442 Scholl BJ, Tremoulet PD (2000) Perceptual causality and animacy. Trends Cogn Sci 4:299–309 Smith EE, Jonides J (1999) Storage and executive processes in the frontal lobes. Science 283: 1657–1661 Sperber D, Premack D, Premack AJ (1995) Causal cognition: a multidisciplinary debate. Oxford University Press, Oxford Turner DC, Aitken MR, Shanks DR, Sahakian BJ, Robbins TW, Schwarzbauer C, Fletcher PC (2004) The role of the lateral frontal cortex in causal associative learning: exploring preventative and super-learning. Cereb Cortex 14:872–880

Index

A Action potential, 71 Adaptation, 126, 127, 134 Addiction, 42–46 Aha-experience, 74 Alcohol addiction, 45 Alpha band, 61, 62 Amphetamine, 42, 44–47 Animal studies, 109, 114 Anterior cingulate cortex (ACC), 254, 255, 258–260 Arterial spin labeling (ASL), 17, 23, 26 Articulatory rehearsal, 112, 115, 117, 119, 120 Articulatory suppression, 111, 112, 117 Associative learning, 267 Attention (al), 51, 52, 69–72, 74, 76, 78, 273–276 active inhibition of, 273–276 Auditory cortex, 59 B Belief bias, 203, 208, 209, 212, 213 Binding, 66, 70–74, 80 problem, 52 Binding-by-neural-synchrony, 70, 71 Blindsight, 165–166 Blood oxygenation level dependent (BOLD), 16, 17, 20, 21, 23–26, 30 contrast models, 20 Brain activation, 141–143, 147, 150, 160 reading, 21 Broca’s area, 5, 110, 112, 116–119, 124, 126, 129, 130, 132, 133 Brodmann area, 3, 165

C Calibration, 24, 26, 27, 30 Callosotomy, 277 Categorization, 65, 66, 69, 76–80, 177, 180 Causality, 9 Causal theories, 273 casual inference, 276 causal mechanisms, 273, 274 Central executive, 110, 118 Cerebral blood ﬂow, 39, 40, 142, 143, 146, 147, 150, 160 Chinese characters, 261, 262 Chunk decomposition, 261–263 Classiﬁcation, 175, 177, 179, 180 Clinical studies and timing, 193–195 “Clustered volume” acquisition, 20 Cognition, 147, 149, 151, 155, 159 cognitive branching, 227, 228, 232 cognitive conﬂict, 254 cognitive process, 141, 142, 150, 153 Coincidence detection, 71 Commonalities, 180 Complex thinking, 120 Components of human working memory, 115 Concepts, 175, 179, 181–183 Conﬂict detection system, 209, 214–216 Conﬂict monitoring, 274 Conjunction, 127, 128, 134 Consciousness, 70–72 Content sensitive system, 207, 208 Convergence, 151–154 Core network, 154–160 Cortex. See Prefrontal cortex Cortical inhibition, 59 Cortical macronetwork, 141, 153, 154, 163

281

282

Index

Cortical neuronal population, 141, 147, 151, 154, 155, 157, 160 CW measurement, 83–85

of human language, 116 of language, 111, 115 Executive processes, 118

D Deception, 31 Deduction, 203 Decision making, 6, 9 Diffusion tensor imaging (DTI), 130, 132 Diffusion weighting, 26 Direct neuronal current imaging, 21 Dissociations, 204, 210, 211, 214, 216 double, 5, 9 Distributed neuronal assemblies, 52 Divergence, 151–154 Domain-speciﬁc distribution of working memory processes, 111, 113, 114 Domain-speciﬁc functional networks, 114 Dopamine, 39, 41–48 Dorsolateral prefrontal cortex. See Prefrontal cortex Dual architecture of human verbal working memory, 111, 119 Dual architecture of verbal working memory, 111–113 Dual mechanism theories, 211, 213 Dynamic functional interactions, 119, 120 Dynamic role binding, 275

F Feedback evaluation, 226, 227 Field potentials, 30 fMRI. See Functional Magnetic resonance imaging Focal brain lesions, 116–117 Formal system, 203, 211–213, 215 Free will, 31 Frequency, 68–69, 71–72, 74–75 Frontopolar prefrontal cortex. See Prefrontal cortex Functional connectivity, 129, 130, 132, 134 Functional contrast, 16–18, 23, 25, 27, 29 Functional interactions, 118–120 Functional magnetic resonance imaging, 4, 7, 8, 15–33, 40–42, 48, 55–57, 141, 143, 144, 147, 150, 153, 161, 254, 255, 258, 260, 261 event-related fMRI, 19 fMR adaptation, 27, 29 fMRI decoding, 21, 29 pre-undershoot in, 20 real-time fMRI, 21 and timing, 190, 192–193 Functional-neuroanatomical models of working memory, 110 Functional NIRS (fNIRS), 83

E Echo planar imaging, 17, 22, 25 EEG. See Electroencephalography Electroencephalography, 4, 8, 54–60 artifacts, 98–100 epilepsy, 95, 100, 101 physiology, 96–97 single-trial coupling, 101, 103 technique, 66, 67 Electrophysiology and timing, 190 Emispheric asymmetry, 277 Emotional processing, 31 Episodic memory, 220, 224, 230, 233, 234, 243 Episodic retrieval, 224, 230–233 Evaluating self-generated information, 232, 234, 236–241 Event-related-potential (ERP), 68–69, 254, 255, 258, 259 Evolution evolutionary-based functional neuroanatomical model of working memory, 114, 116 evolutionary differences, 111

G Gall, F., 37 Gamma activation, 72–74 frequency, 30 oscillations, 53, 60 Gestalt, 52, 178, 181 Global synchronization, 60 H Hardware, 17, 18 Hemodynamic point spread function, 25 Hemodynamic response, 18, 22, 23, 28, 30 Hemoglobin, 16 Heuristic system, 203, 208, 211, 214, 215 Higher order cognitive processes, 219, 241, 242, 244, 246 High ﬁeld, 25, 26 strength, 21 High spatial resolution, 19 High temporal resolution, 20 Hint, 255, 257–259, 263

Index History contrast, 16–18 Human speech, 109, 116 thought, 219–247 Humor, 31 Hypothesis generation, 226, 227 I Induced gamma, 72–73 Inferior longitudinal fasciculus, 132, 133 Inhibition, 274, 275 Inner ear, 119 Inner speech, 119, 120 Insight, 6 problem solving, 9 Integration functional, 4–5, 8 Interference effects, 117 Intermediate frontal sulcus, 112–114 Internally generated information, 229, 236–241 Interneurons, 71 Interpretation, 30 Introspection, 31 Introspective thought, 220, 227, 236–241 Intuition, 6 K Kanizsa, G., 73 Knowledge, 175–184 L Languages, 176, 181–184 Lateral prefrontal cortex. See Prefrontal cortex, 219–247 dorsolateral prefrontal cortex, 223, 230, 231, 233, 235, 239, 240, 273, 274 rostrolateral prefrontal cortex, 223, 224, 238–240, 242, 243, 245 Launching effect, 271 Learning, 31, 73–74 Left hemisphere interpreter, 214, 215 Lesion focal, 5, 9 Localization, 123, 124 functional, 3, 4 M Magnetoencephalography, 4, 7, 51, 54–62 Maintenance processes, 117 Manipulation of information in working memory, 118 Manipulation processes in working memory, 117

283 Matchsticks arithmetic task, 261 MEG. See Magnetoencephalography Memory, 73, 74 Mental imagery, 120 Mental logic, 204, 211, 216 Mental models, 204, 211, 216 Michotte task, 274 Mini-insight problems, 253 Modiﬁed Beer-Lambert law, 83, 84, 89, 90 Modular information encapsulation, 129 Moral judgment, 31 Morphing, 79 Mosso, A., 40 Multimodal working memory system, 114, 115, 120 Multisubject averaging, 17, 18, 25, 28 N Near infrared spectroscopy, 4, 8 Neural implementation of human working memory, 110 Neurocognitive models, 109–111 Neurological model of language, 124 Neuronal assembly, 70 Neuropsychology, 5 Neuroreceptor, 143, 146–148 Neuroscience, 5, 7, 8 cognitive, 7, 8 Nicotine, 46 NIRS. See Near infrared spectroscopy Non-articulatory maintenance of phonological information, 115, 117–120 Non-human primates, 110–117 Number reduction task (NRT), 263, 264 O Object recognition, 70, 76, 79 Ocular dominance column, 19, 26 Operational architectonic framework, 75 Oscillation, 68, 69, 71–76 Oscillatory brain dynamics, 51–55 P Paired pulses of stimuli, 24 Paradigm design, 18 Parallel imaging, 21 Parametric manipulation, 19 Parametric modulation, 30 Pattern generation, 144 Perception, 147, 151, 153, 161, 163–168 perception-action cycle, 240 percepts, 175–180 perceptual causality, 271–273, 276, 277 Personality, 45, 46, 48

284 PET. See Positron emission tomography Phase coherence, 60 Phase-resolved spectroscopy (PRS), 84, 86, 89–91 Pheromone, 166–168 Phonological loop, 110 Phonological storage, 110, 112 Physiologic noise, 28 Positron emission tomography, 4, 7, 8, 39–48, 55–57, 141, 143–148, 150–153, 160–162, 165, 166 Posterior fusiform gyrus, 126–129, 132–134 Postprocessing, 18 Prefrontal cortex, 273–276 dorsolateral, 223, 230, 231, 233, 235, 239, 240, 273, 274 frontopolar, 224–233, 235, 236 hierarchy/hierarchical functions, 230–235 lateral, 219–247 rostrolateral, 223, 224, 238–240, 242, 243, 245 Premotor cortex, 111, 118, 119 Principal sulcus, 114 Problem solving, 6, 9, 255, 256, 261, 263 Processing software, 18, 32 Psychological time models, 195–196 Puzzle, 256–260, 263 R [11 C] Raclopride, 42, 44 Radioligand, 146, 147 Rationality, 212–214, 216 Raven’s progressive matrices, 221, 224, 225, 227, 242 Reasoning analogical, 6 causal, 6, 9 deductive, 5, 9 inductive, 6 relational, 6 Receptor ﬁngerprint, 148–149 Recruited ﬁeld, 154–160 Relational reasoning, 220–224 Remote Associates Test (RAT), 256, 260–261 Representation, 70, 71, 75–78 iconic, 8 lexical, 8 representational change, 75 symbolic, 8 Resting mode, 242 Resting state correlations, 20 Restructuring, 254–259, 263, 265 Riddle, 256–258 Right anterior superior temporal gyrus, 261

Index S Sampling rate, 68 Selective attention, 120 Self-referential evaluation, 227, 234–235 Semantic memory, 270, 275 Sensitization, 47, 48 Signal-to-noise ratio, 69 Single-shot EPI, 25 Social interaction, 31 Source modeling, 56, 58 Spatial and temporal contiguity, 272 Spectral power analysis, 68 Spin-echo imaging, 25 Split-brain, 276, 278 Spontaneous thought, 241–246 Steady-state response, 72 Stress, 46, 47 Striatum, 42–48 Structural equation modeling, 21 Substances, 176–179 Superior longitudinal fasciculus, 130–133 Syllogism, 65, 70 Synchronicity, 8 Synchronization, 52–55, 59–61, 65, 68–76 T Taxonomy, 4, 6, 8, 9 Temporal processing, 187, 191, 192, 196, 197 Temporal resolution, 15, 20, 22–25, 27, 31–32 Temporal trainings, 196 Theory dual mechanism, 6, 8 mental logic, 6 mental model, 6 Theory of mind, 31 Theta oscillations, 68, 73, 74 Thinking, 3, 5, 65–80 neural correlaties, 7–10 Three-stage hierarchical system, 236 Time perception, 9 Time-resolved spectroscopy (TRS), 84–86, 89, 90 Timing, 187–197 Tinnitus, 51, 59–62 Tracer, 144, 146, 147, 162 Tractography, 130–132 Transient evoked response, 72 Transition, 75 Typicality, 78 U Uncertainty maintenance, 214–216 Uncinate fasciculus, 130, 131, 133

Index V Validity, 203, 207 Vascular space occupancy (VASO), 21 Verbalization, 120 Verbal rehearsal, 109, 111, 112, 114–116, 119 Visual attention, 272

285 Visual cortex, 263 Voxel volume, 25 W Wernicke, C., 4 Wernicke’s area, 124, 126, 132 Working memory, 31, 70, 71, 109–120, 276